S. D. V. Stephens

notes / Genetic Engineering / chapter 1: Lectures

3 Coevolution and Protein Language Models

Comparing sequences (homology) is very powerful: differences tell us what they might do. Co-variants, for example, might be dependent on each other.

Interested in coevolution. Evolution is important (even for bioengineers thinking about the future). Sequences are all related; evolution tells us how.

How do we model this (what?): PSSM (position-specific scoring matrix), MG, AE, Potts/MRF (Markov random field), pLM (protein language model), AlphaFold/RoseTTAFold. They are all super different.

All models are input → output sequences: linear, transformers, etc. (the parameterization in the middle).

  • About 70% is explained by protein structure (covariance).
  • ML: stochastic statistical inferences (they don’t understand the protein physics).
  • Large vs. small models have different uses (grouping, subdividing/averaging, etc.).

Want to model conservation and coevolution. Colors become 1s and 0s, then undergo weights (plus a bias term), and then we get an output.

The GREMLIN model as a network: three coloured sequence positions are each one-hot encoded into a group of input nodes x, every input node connects to every output node through weights w, each output node has a bias b, and the outputs give the predicted one-hot sequence x'.

GREMLIN

GREMLIN: Generative REgularized ModeLs of proteINs. GREMLIN, plmDCA and EVcouplings are the same model: a Potts model, i.e. a Markov random field.

AI will sometimes just use the identity matrix to cheat (so we learn nothing). Solution: remove the original input.

  • Proteins: a matrix.
  • DNA: a matrix.

Contact maps / crystal structure. Represent the protein as a contact map.

Pseudolikelihood and masked language modeling

Sketch of a protein language model: a small matrix feeds into a stack of layers, the middle layer is a pairwise grid whose rows and columns are highlighted, further layers follow to the output, and a long arrow connects the first layer to the last.

Pseudolikelihood vs. masked language modeling:

Mask about 15%.

Masked language modelling: two positions of an input sequence are masked, the sequence passes through the model to a hidden layer and then to an output in which the model rediscovers the two masked positions.

Train the model on a single protein and you get something like a contact map (and GREMLIN).

A protein language model is essentially a decomposition of the model. In practice many, many layers are used.

Orphan proteins: proteins without neighbors. (You can alter the neighborhood radius of an orphan protein to make it no longer an orphan.)

Protein language models learn a library of GREMLIN models, but are capable of doing better due to shared parameters.

Circular permutations

Circular permutations of proteins. Want to find circular permutations.