The AlphaFold prediction tool in Foldit

We are announcing a brand new Foldit feature that will enable players to use the revolutionary AlphaFold algorithm from DeepMind! The AlphaFold feature is currently available for devprev users, and we expect to release it as a main update in the coming days. The AlphaFold feature is now available for all users in select Foldit puzzles.

AlphaFold v2.0

AlphaFold v2.0 is an algorithm to predict the folded structure of a protein from its sequence, and was developed by the company DeepMind in 2020.

Previously, in the 2018 CASP competition for protein structure prediction, DeepMind had made a splash with their initial version of AlphaFold, outperforming dozens of research groups from around the world. The DeepMind group specializes in a type of algorithm called a neural network, and they showed that this type of algorithm held huge potential for the field of protein structure prediction. We wrote a blog post about the initial AlphaFold algorithm when DeepMind published it in January 2020.

After this initial success, DeepMind completely restructured their algorithm, and at the 2020 CASP competition they amazed the world with an even bigger leap forward. The new AlphaFold v2.0 is able to predict protein structures with astounding accuracy. The 2020 CASP results promised big advances for protein research, and the scientific community has been anxiously waiting for DeepMind to release the details about AlphaFold v2.0.

Figure 1. Rankings for the 2018 and 2020 CASP competitions. The original AlphaFold algorithm was clearly a head above the other competitors. The latest AlphaFold v2.0 blew away the competition.

On July 15, 2020 DeepMind published a new paper about AlphaFold v2.0 and released the algorithm code so that anyone could use it.

AlphaFold for protein design

AlphaFold is especially accurate for predicting natural proteins, where it can draw on the rich information in evolutionary patterns. But we’ve also found it to be very good at predicting the structures of designed proteins—even though these proteins have no evolutionary history. In fact, when we check against solved structures of designed proteins, we find that AlphaFold is usually more accurate than the design model itself!

Figure 2. Comparing the accuracy of AlphaFold predicted models and design models for 22 designed proteins with solved structures. The diagonal represents the line of equality. Points above the diagonal are cases where the AlphaFold prediction is more accurate than the design model.

We’ve also found that AlphaFold may be able to help us pick out designs that will fail lab testing. Whenever AlphaFold predicts a structure, the algorithm also produces a confidence value for the prediction. We see that AlphaFold tends to report a higher prediction confidence for successful protein designs.

Figure 3. The distribution of confidence values for 148 lab-tested Foldit designs. Successful designs (blue) tend to yield AlphaFold predictions with higher confidence than design failures (orange).

In 2019, we tested 148 Foldit designs in the lab and found 56 were successful designs—a total success rate of about 38%. If we had rejected designs with AlphaFold confidence under 80%, then we still would have found 50 successful designs, with a success rate of over 60%!

A new Foldit feature

We are excited to announce a new Foldit feature that will let you get AlphaFold predictions for proteins you design in Foldit.

Certain puzzles will display a new DeepMind AlphaFold button in the Main Menu. This button opens up a dialog with a list of your saved solutions on the right-hand side. To request an AlphaFold prediction for a solution, select the solution and click the Upload for AlphaFold button. This will send your solution to the Foldit server and remotely run the AlphaFold algorithm.

A new solution will appear in the left-hand list and show the message “Pending…” while AlphaFold makes its prediction. It will take at least a few minutes to run, and the wait time may be longer depending on how busy the server is. You will not be able to make a new AlphaFold upload while you have a submission currently pending. You may submit up to 5 concurrent jobs; if you currently have 5 AlphaFold uploads pending, you must wait for one to complete before making another submission. Click the Refresh Solutions button to check if your AlphaFold job is done.

When the AlphaFold algorithm has completed, the left-hand solution will display two values:

Confidence is AlphaFold’s own estimate about the accuracy of its prediction. Figure 3 above suggests that designs with higher confidence are more likely to fold successfully. Players should aim for confidence values of 80% or higher.

Similarity measures how closely the AlphaFold prediction matches your designed structure. If similarity is low, then AlphaFold has predicted that your design sequence will fold into a different shape than your designed structure.

To load the AlphaFold prediction into the Foldit puzzle, select the left-hand AlphaFold solution and click the Load button at the bottom of the dialog. Note that AlphaFold predictions may not score as well as solutions that have been optimized in Foldit. If you decide to work off of the AlphaFold solution, we recommend a quick Wiggle and Shake of the raw AlphaFold model.

The AlphaFold confidence and similarity values will not affect your Foldit score in any way. For the time being, the AlphaFold feature is simply a tool that you can use to get feedback about your solution, and to see how your design sequence is predicted to fold up.

Remote computation

Unlike typical Foldit tools, the AlphaFold algorithm runs remotely on an online server.

Normally, when you run Foldit on your computer, all of the Foldit computations are performed by your computer. If your internet connection fails in the middle of a puzzle, you can still continue to use all of the Foldit tools.

This AlphaFold feature is different, and the actual computations will run on a server hosted at the UW Institute for Protein Design (IPD). So, when you click the Upload for AlphaFold button, your solution is sent to the IPD server, which runs the AlphaFold algorithm and then sends the result back to your computer.

The biggest reason for this is that the AlphaFold algorithm is… big. Even the basic slimmed-down version requires several GB of disk space. If we wanted to distribute the AlphaFold software with Foldit, that would increase the download size of Foldit by 10x.

Another reason is that the AlphaFold algorithm runs much less efficiently on common CPUs than on GPUs, which many players may not have. If you ran AlphaFold on your CPU at home, it might take an hour to get a result back. However, if we use our GPUs at IPD, the actual processing will go much faster. Since most of our recent Science puzzles have had fewer than 100 active players at a time, we think that players can get results faster if we process AlphaFold jobs on our server GPUs.

What’s next?

This is an exciting time for the world of protein research! DeepMind has inspired other research groups, including the IPD, to explore similar kinds of neural network algorithms for protein structure prediction. As more researchers publish their findings and learn from one another, we can probably expect to see even more accurate algorithms in the future.

AlphaFold is already transforming the study of natural proteins, and has provided researchers with confident predictions of important proteins with unknown structures. But in the field of protein design, we are still learning how to make the best use of these advances. We hope that Foldit players will find the AlphaFold predictions helpful for designing creative new proteins!

Please note that the new AlphaFold feature is experimental, and it may change or even disappear in the future. Foldit is sharing the server GPUs with other research projects, and we may need to adjust our usage or develop new strategies for running GPU-heavy computations.

Edit Nov 2, 2021: Predicting native vs. designed proteins

Since we launched the AlphaFold tool, several Foldit players have pointed out a puzzling result in certain AlphaFold predictions:

"I copied a native protein sequence onto my design, but the AlphaFold prediction is completely different from the native structure, or it has an extremely low confidence. I thought AlphaFold was supposed to be good at predicting native proteins. What's going on?"

This is because in Foldit we are using an "abbreviated" version of AlphaFold that is not expected to work well on natural protein sequences.

The official, complete AlphaFold pipeline requires an extra step, scanning a large database for sequences that are similar to your query sequence. These similar sequences should all be evolutionarily related, and AlphaFold is able to extract patterns from this evolutionary data. AlphaFold is extremely good at extracting patterns from this evolutionary data, and this seems to be one of the reasons it performed so well in CASP.

When we use AlphaFold to predict Foldit designs, we skip this extra step because it is slow and because we do not expect to find "evolutionarily related" sequences for our designed proteins. Our internal benchmarking shows that AlphaFold is still good at predicting Foldit designed proteins, even though they don't have evolutionary data. However, skipping this step means that AlphaFold may underperform for natural protein sequences.

In the plot above entitled "AlphaFold confidence of Foldit designs", 
one axis lists the #'s 40 to 100 while the other axis lists the 
#'s 0.00 to 0.08. What variables are shown on each axis? Which 
axis is showing the AlphaFold Confidence? Is it the axis with  
the #'s 40 to 100 on it? Are the areas under these curves roughly 
(0.05 x 40)/2 = 1.00 for the pink curve and (0.09 x 20)/2 = 0.90 
for the blue curve, respectively? I found these numbers by eye, 
approximating each curve as a triangle with area = (height x width)/2. 
Should the area under each curve equal 1.00, as in 100% of the 
population of Foldit designs? Thanks!

Since AlphaFold's structure prediction depends only on the
protein's amino acid sequence, you could save computation 
time at the IPD server by checking for previous submissions 
with the same amino acid sequence. You also might as well
calculate the Foldit score for each raw AlphaFold solution
on the IPD server and list this score in one of the new 
AlphaFold menus. Also, what would AlphaFold do if we sent
it a Foldit structure containing cuts, for example? Would
it treat yellow and blue cuts the same way? 

If we are in a Group, can we share these AlphaFold predictions
with other Group members? Should we consider all raw AlphaFold
predictions as already being Shared with Scientists? Can we 
load an AlphaFold prediction as a Guide so we can see how well 
our own solutions overlap with it? If we use Foldit to improve 
the Foldit score for an AlphaFold prediction, does the new score 
count as a solo or an evo?

Does the similarity score for two structures assume that both 
structures have the same sequence? Would it be possible to pick 
any of our saved solutions as a Guide and then list for each 
of our saved solutions how similar it is to the present Guide? 
Perhaps for each solution it could list several similarity 
scores. One could ignore the amino acid sequences and just 
measure how similar the solution's backbone-only structure is 
to the Guide's. Another could assume the amino acid sequences 
are the same and find how similar the solution's all-atom 
(both backbone and sidechain) structure is to the Guide's.
Yet another could account for how similar the solution's
amino acid sequence is to the Guide's. If Foldit listed all
of these similarity scores together in a user-friendly table, 
it would help us decide how diverse our solutions are without 
having to view and compare each solution's 3d structure to the 
Guide's. Foldit could even use the similarity scores to make
dendrograms (like evolutionary trees) that sort the most 
similar structures into small groups or clusters. Foldit
could also let us make our own dot plots like at:
https://fold.it/portal/node/2000356
but showing Foldit score vs similarity score instead of
Foldit score vs rms value.

Figure 3 is a combined histogram and density plot, with AlphaFold confidence plotted on the x-axis.

The semi-transparent histogram bars show how the confidence values are actually distributed, by counting the number of designs that fall into discrete "bins". The curves of the density plot approximate the histogram as a smooth distribution. Even though the blue and orange groups have different numbers of samples, the plots are normalized so that the orange and blue curves have the same area (the two histograms are also scaled so that each covers an equal area). The y-axis is simply scaled so that the area of each curve is 1.00.

I don't think it is helpful to try and interpret the numbers on the y-axis. This plot is meant to illustrate that the blue and orange distributions are very different, and that AlphaFold confidence is somewhat informative about whether a design is likely to be successful. On the other hand, if the blue and orange distributions had overlapped perfectly with one another, that would indicate that AlphaFold confidence is uninformative about design success.

Yes, the similarity score considers sidechains, so the two compared structures must have the same sequence. An AlphaFold prediction is only valid for the input sequence; I don't think we want to encourage comparisons between an AlphaFold prediction and designs with mismatched sequences. You might be right that we should have more tools to think about sequence similarity in Foldit. Sequence diversity may become more important to Foldit in the future, so we might like to have better ways to visualize and interact with sequence diversity.

More suggestions like this are very welcome! So far we have only implemented the bare minimum for the AlphaFold feature, so that Foldit players could start using it as soon as possible. We hope to improve on it in the future; feedback and suggestions from players will help us make this tool more useful!

We might check for duplicate sequence submissions, but I don't think the Foldit score will be very meaningful for evaluating the raw AlphaFold prediction. We expect that the raw prediction will require a Wiggle and Shake to "relax" the structure; before that relaxation, the Foldit score of the raw prediction is practically meaningless. Cuts will not affect the AlphaFold prediction, but they may persist when you load the AlphaFold prediction, and will still need to be closed manually.

To share AlphaFold predictions with your group, you will have to load the prediction and save it separately as a new Foldit share. Do not consider AlphaFold uploads to be "Shared with Scientists" – please continue to share your favorite solutions using the Upload for Scientists button! We might add a feature to load the AlphaFold prediction as a guide. When you work off of an AlphaFold prediction, score improvements will still count toward your Solo score.

When we compare the two methods on a set of natural protein targets from CASP 14, AlphaFold predictions are more accurate than RoseTTAFold. The official RoseTTAFold paper is unfortunately behind a paywall, but you can see the preprint posted on bioRxiv (see figure 1B).

Note that this advantage in prediction accuracy does not necessarily mean that AlphaFold will be better at weeding out poor designs. In fact, the earlier trRosetta neural network seems to be just as predictive as the AlphaFold confidence, when it comes to weeding out successful designs (compare figure 3 above with figure 3B in the linked trRosetta post). The truly amazing results from AlphaFold are the structures it produces.

In other words: AlphaFold confidence is good (but not exceptional) for weeding out bad designs that fail to fold; however, if a design does fold then AlphaFold is exceptionally good at guessing its structure.

In the latest update, the AlphaFold dialog should auto-update every 30 seconds. We do not yet have a "notification" for AlphaFold completions, but you should not need to click Refresh while the AlphaFold dialog is open.

I plugged my stabilized, non-yet-near-the-target solution for 2029 into AlhpaFold, and let it crunch numbers. It spit out Confidence and Similarity of 89.3% and 86.3%, respectively.

So I loaded the AlphaFold-modified solution, and gave it the obligatory shake and wiggle. AlphaFold added five BUNS and one bad loop, and dropped my core score by 100 points.

FoldIt told me my structure had zero buns, zero bad loops, and a full core.

What is this supposed to tell me? If I am to assume that AlphaFold is better at "getting things right", then my first guess is that FoldIt's scoring algorithms are… shall we say… in need of revision. If FoldIt works better at scoring predictions, then of what benefit is AlphaFold?

Please clarify. Thank you.