About


Team

Current

  • Yongwook Choi
  • Agnes Chan

Former

  • Greg Sims
  • Jason Miller
  • Sean Murphy

This work is funded by National Institutes of Health grant number 5R01HG004701-03.


References

  1. Choi Y, Sims GE, Murphy S, Miller JR, Chan AP (2012) Predicting the Functional Effect of Amino Acid Substitutions and Indels. PLoS ONE 7(10): e46688.
  2. Choi Y (2012) A Fast Computation of Pairwise Sequence Alignment Scores Between a Protein and a Set of Single-Locus Variants of Another Protein. In Proceedings of the ACM Conference on Bioinformatics, Computational Biology and Biomedicine (BCB '12). ACM, New York, NY, USA, 414-417.

Overview - PROVEAN (Protein Variation Effect Analyzer) v1.1

  1. PROVEAN Score
  2. PROVEAN Peformance
  3. Comparison with Other Tools for Single Amino Acid Substitutions

The results below are based on PROVEAN v1.1. For the performance of previous version (v1.0), please refer to this page.


1. PROVEAN Score

PROVEAN was developed to predict whether a protein sequence variation affects protein function.

PROVEAN is able to provide predictions for any type of protein sequence variations including the following.

  • Single or multiple amino acid substitutions
  • Single or multiple amino acid insertions
  • Single or multiple amino acid deletions

(a) Overview

An overview of the PROVEAN procedure is shown in Figure 1. Briefly, clustering of BLAST hits is performed by CD-HIT with a parameter of 75% global sequence identity. The top 30 clusters of closely related sequences form the supporting sequence set, which will be used to generate the prediction.

A delta alignment score is computed for each supporting sequence. The scores are then averaged within and across clusters to generate the final PROVEAN score. If the PROVEAN score is equal to or below a predefined threshold (e.g. -2.5), the protein variant is predicted to have a "deleterious" effect. If the PROVEAN score is above the threshold, the variant is predicted to have a "neutral" effect.

Figure 1. Computing the PROVEAN score.

(b) Setting the PROVEAN score threshold

PROVEAN introduces a delta alignment score based on the reference and variant versions of a protein query sequence with respect to sequence homologs collected from the NCBI NR protein database through BLAST. The PROVEAN score distribution of a set of 58K UniProt human protein variants with known functional outcome is shown in Figure 2.

For maximum separation of the deleterious and neutral variants for all 4 classes of human protein variants, the default score threshold is currently set at -2.5 for binary classification (i.e. deleterious vs neutral).

Figure 2. PROVEAN (v1.1) score distribution for 58K deleterious and neutral UniProt human protein variations.

PROVEAN score distribution

(c) How to increase stringency of detection

To increase the specificity of detection (i.e. find deleterious variants with higher confidence), a lower score threshold can also be used (e.g. -4.1). To increase the sensitivity of detection (i.e. find more deleterious variants by including those with lower confidence), a higher score threshold can also be used (e.g. -1.3). A summary of specificity and sensitivity obtained using different score cutoffs is shown in Figure 3.

Figure 3. Sensitivity and specificity at different PROVEAN score cutoffs.

(d) PROVEAN v1.1 compared to the previous version (v1.0)

In order to refine the supporting sequence set, some conditions were added and parameters were changed accordingly in the process of selecting supporting sequences.

  • In order to be used as the supporting sequence, a sequence must be at least 30% of the length of query sequence.
  • While clustering, each sequence in a cluster must be at least 30% of the length of its representative sequence.
  • With 75% global sequence identity, top 30 clusters were selected.

2. PROVEAN Performance

The overall accuracy for binary classification of protein variants (deleterious or neutral, classification property is being deleterious):

  • 79.5% for UniProt human protein variations (download)
  • 77.9% for UniProt non-human protein variations (other mammals, plants, bacteria, etc.) (download)

Table 1. Number of human protein variations and prediction accuracy of PROVEAN v1.1.

Variant types Functional outcome Peformance(%)
Deleterious Neutral Total Sensitivity Specificity Accuracy* Balanced accuracy**
Single amino acid substitutions 20821 36825 57646 78.85 79.55 79.29 79.20
Deletions 652 77 729 96.78 66.23 93.55 81.51
Insertions 110 61 171 91.82 78.69 87.13 85.25
Replacements 79 59 138 93.67 64.41 81.16 79.04
Total 21662 37022 58684 79.51 79.49 79.50 79.50


Table 2. Number of non-human protein variations and prediction accuracy of PROVEAN v1.1.

Variant types Functional outcome Performance (%)
Deleterious Neutral Total Sensitivity Specificity Accuracy* Balanced accuracy**
Single amino acid substitutions 14117 16498 30615 80.33 76.00 78.00 78.17
Deletions 142 227 369 82.39 61.23 69.38 71.81
Insertions 34 137 171 76.47 75.18 75.44 75.83
Replacements 886 1029 1915 86.79 63.46 74.26 75.13
Total 15179 17891 33070 80.72 75.09 77.67 77.90

Note:

  • *Accuracy=(TP + TN)/(TP + TN + FP + FN); affected by potential unequal sizes of deleterious and neutral datasets
  • **Balanced Acurracy=(Sn + Sp)/2; unaffected by potential unequal sizes of deleterious and neutral datasets
  • A score threshold of -2.583 was used.
TP: true positive; TN: true negative; FP: false positive; FN: false negative; Sn: sensitivity; Sp: specificity


3. Comparison with Other Tools for Single Amino Acid Substitutions

(a) Prediction accuracy of different tools

The performance of PROVEAN was compared with other prediction tools: Mutation Assessor, SIFT, PolyPhen-2, and Condel, using default score thresholds suggested by individual tools. Overall, the performance of PROVEAN is comparable to other tools as shown in Tables 3 and 4.

Table 3. Prediction accuracy of PROVEAN, Mutation Assessor, SIFT, PolyPhen-2, and Condel for the UniProt human protein variant datasets.

Prediction tools Score thresholds Human dataset References
Sensitivity Specificity Accuracy* Balanced accuracy** No prediction
PROVEAN -4.100
(default) -2.500
-1.300
56.17
79.76
90.28
90.34
78.63
61.65
78.00
79.04
71.99
73.25
79.19
75.96
0 Choi et al., 2012; web
Mutation Assessor 0.800
1.90
96.54
85.29
40.59
71.02
60.90
76.20
68.57
78.15
317
(0.55%)
Reva et al., 2011; web
SIFT 0.050 85.03 68.95 74.77 76.99 1147
(1.99%)
Kumar et al., 2009; web
PolyPhen-2 0.432 88.68 62.45 72.00 75.56 2279
(3.95%)
Adzhubei et al., 2010; web
Condel web server 0.469 93.84 46.23 64.67 70.04 7194
(12.48%)
González-Pérez and López-Bigas, 2011; web

Table 4. Prediction accuracy of PROVEAN, Mutation Assessor, SIFT, and PolyPhen-2 for the UniProt non-human protein variant datasets.

Prediction tools Score thresholds Non-human dataset
Sensitivity Specificity Accuracy* Balanced accuracy** No prediction
PROVEAN -4.100
(default) -2.500
-1.300
59.76
81.11
89.39
86.67
75.22
61.63
74.26
77.94
74.43
73.22
78.17
75.51
0
Mutation Assessor 0.800
1.90
93.17
81.30
45.13
67.16
67.07
73.62
69.15
74.23
732
(2.39%)
SIFT 0.050 87.47 69.27 77.88 78.37 1542
(5.04%)
PolyPhen-2 0.432 87.77 65.81 76.11 76.79 1499
(4.90%)

Note:

  • The UniProt human and non-human protein variant datasets were used.
  • The comparison was performed only for single amino acid substitutions because other tools do not handle other types of variation such as amino acid insertion or deletion.
  • The "No prediction" column shows the number of variants for which the corresponding tool fails to provide a prediction.
  • PROVEAN v1.1, Mutation Assessor web server v1.0, SIFT v4.0.3, PolyPhen-2 webserver v2.1.0, and Condel web server v1.4 were used.
  • *Accuracy=(TP + TN)/(TP + TN + FP + FN); affected by potential unequal sizes of deleterious and neutral datasets
  • **Balanced Acurracy=(Sn + Sp)/2; unaffected by potential unequal sizes of deleterious and neutral datasets
TP: true positive; TN: true negative; FP: false positive; FN: false negative; Sn: sensitivity; Sp: specificity

(b) Prediction consistency among different tools

The prediction results for the UniProt human protein variant dataset were obtained from multiple tools including PROVEAN, SIFT, and PolyPhen-2, and are summarized in Venn diagrams shown in Figure 4. Overall, the prediction results for 78.5% (15,618/19,898) of disease-associated variants and 46.8% (16,244/34,701) common variants are in agreement and shared by all three tools. In addition, each tool produces correct predictions for distinct subsets of disease or neutral variants. Combining prediction results from multiple tools can increase the chance of identifying functional variants that had been missed by other tools.

Figure 4. Venn diagrams showing predictions from PROVEAN, SIFT, and PolyPhen-2 for the UniProt human protein variant dataset (score thresholds used: PROVEAN, -1.3; SIFT, 0.05; PolyPhen-2, 0.432).

Note:

The number and percentage of correctly predicted variants are shown next to each tool.