Enhancing privacy in biosecurity with watermarked protein design

TL;DR

This paper introduces a method to add watermarks to protein sequences to improve biosecurity and protect data privacy.

Contribution

A novel framework for watermarking protein sequences using deep-learning models to ensure traceability and privacy.

Findings

The proposed watermarking framework achieves robust traceability while maintaining sequence privacy.

The method significantly improves watermark detection efficiency compared to existing techniques.

The framework allows researchers to embed their identity into protein sequences for intellectual property claims.

Abstract

The biosecurity issue arises as the capability of deep-learning-based protein design has rapidly increased in recent years. Current regulation procedures for DNA synthesizing focus on the biosecurity but ignore the data privacy.

We propose a general framework for adding watermarks to protein sequences designed by various autoregressive deep-learning models. Compared to current regulation procedures, watermarks also ensure robust traceability to achieve biosecurity but maintain privacy of designed sequences by local verification. Benchmarked with other watermarking techniques, the watermark detection efficiency of our method is substantially increased to be more practical in real-world scenarios. Moreover, it provides a convenient way for researchers to claim their own intellectual property since the designer’s information could be embedded into the sequence with our framework.

The implementation of the protein watermark framework is freely available to noncommercial users at https://github.com/poseidonchan/ProteinWatermark.

Figures

The biosecurity-enhanced regulation process with watermarked protein design. (a) Conventional protein design regulation process. Researchers use diverse tools to design protein sequences and translate proteins into DNA sequences, then they report every sequence to the authority and obtain permission for synthesis and expression. (b) Watermarked protein design tools. To add watermarks to protein sequences, the residue distribution of a specific design task is modified according to a watermark code E derived from a private key. The expectation of the modified distribution over the watermark code is identical to the original distribution. (c) Watermarked protein design process. Each researcher applies for a private key from the authority. With this private key, a researcher can design protein sequences with watermarks. These sequences are then translated to DNA sequences and can be locally verified by a DNA synthesizer with a watermark detection program. When a suspicious protein gains interest, the authority could sequence it and trace it back to its designer even it has some mutations. Additionally, researchers could claim the IP of a specific protein based on their private key.

Protein watermark is unbiased, robust, and practical. (a) The ESMfold’s pLDDT of each designed sequence at different temperatures. The blue and red represent the original ProteinMPNN and watermarked ProteinMPNN, respectively. Each violin plot contains 3000 pLDDT scores from 60 different monomer design tasks while each task has 50 different designs. The median performance drops as the temperature increases. (b) The relationship between entropy and detected watermark score. For the 60 monomer design tasks, we sample one sequence at each temperature. The mean entropy per residue linearly increases as temperature increases. With more entropy, the watermark score detection efficiency increases, and the optimized detection method substantially surpasses the original detection method. (c) The robustness of watermark score. The average watermark score goes down as the protein sequence is modified. In the plot, “G to A” and “A to G” refer to the substitution of all alanine with glycine and the replacement of all glycine with alanine, respectively. Despite these significant alterations, the watermarks remain detectable. The 95% confidence interval is centered at the mean. (d) The rank of the best pLDDT score of 50 samples at each temperature. The proteins are ranked based on their length ranging from 50 to 650 residues. These 60 proteins are evenly distributed in terms of length. The designed protein with the highest pLDDT score is in red, whereas the one with the lowest score is in blue.

Full text

1 Introduction

Computational protein design, which includes various methods ranging from biophysical models to generative machine learning methods, has emerged as the possible solution to substitute the experimental-based variant selection methods since it could efficiently explore the space of functional protein (Notin et al. 2024). Among the computational methods, the capability of generative deep-learning approaches has rapidly grown and can perform various protein design tasks within a single model (Dauparas et al. 2022, Madani et al. 2023). Moreover, the experimental validations show that these generative models can design sequences with a high success rate in in vivo expression (Dauparas et al. 2022, Madani et al. 2023). However, these high success rate protein design tools not only bring opportunities and benefits in biological research but also raise concerns about biosecurity (Baker and Church 2024, Callaway 2024), because bad actors in the community could use advanced open-source generative protein design tools to design and synthesize hazard proteins and causes potential threats.

To mitigate this issue, the previous consensus involved controlling the DNA synthesis step and logging every sequence with the authority (e.g. International Gene Synthesis Consortium) for screening to reduce the risk (Fig. 1a) (Baker and Church 2024). If de novo designed harmful sequences escape from screening due to their little sequence similarity in the database, authority could robustly trace back the source through sequence alignment methods even if a sequence collected from environment may have some mutations. Thus, this conventional method can deter bad actors to achieve biosecurity. Although the regulation method is practicable, reporting every sequence to the authority may compromise the privacy of the designed protein sequence and pose challenges to intellectual property (IP) protection. Recently, a concurrent work by SecureDNA foundation introduces a cryptographic framework designed for the screening step to protect the privacy (Baum et al. 2024). In this framework, biological sequences are dissected into fragments and then hashed for exact matching with hashed hazardous sequences in a curated database. However, using hash function as encryption method presents difficulties for robust traceability, as the mutated sequences would not have the same hash code as the original one. Additionally, the hashes of all subsequences of length $[eqn]$ need to be uploaded for screening. These hashes are significantly easier to reverse than a single hash function, which could compromise privacy. As long as a bad actor knows a subsequence $[eqn]$ , they can guess the nearest neighboring base pair $[eqn]$ with the help of $[eqn]$ . Once the bad actor has the hash of all subsequences and obtains the content of any subsequence, for example, through a collision with a common structure, they can recover the entire sequence with only $[eqn]$ number of hash requests, where $[eqn]$ is total sequence length. Hence, there still exists a large gap between the current DNA synthesis regulation methods and the need for privacy and biosecurity.

The biosecurity-enhanced regulation process with watermarked protein design. (a) Conventional protein design regulation process. Researchers use diverse tools to design protein sequences and translate proteins into DNA sequences, then they report every sequence to the authority and obtain permission for synthesis and expression. (b) Watermarked protein design tools. To add watermarks to protein sequences, the residue distribution of a specific design task is modified according to a watermark code E derived from a private key. The expectation of the modified distribution over the watermark code is identical to the original distribution. (c) Watermarked protein design process. Each researcher applies for a private key from the authority. With this private key, a researcher can design protein sequences with watermarks. These sequences are then translated to DNA sequences and can be locally verified by a DNA synthesizer with a watermark detection program. When a suspicious protein gains interest, the authority could sequence it and trace it back to its designer even it has some mutations. Additionally, researchers could claim the IP of a specific protein based on their private key.

In this paper, we introduce protein watermark, a new general framework for adding watermarks in the generative protein design model to fulfill the need of privacy and robust traceability (Fig. 1b and c). In this framework, each researcher gets an authorized private key associated with their identity from the authority, then the researcher can use this private key to add watermarks to the generative model-designed protein sequence (Fig. 1b). With the private key and corresponding sequences, the DNA synthesizer equipped with a watermark detection program could locally verify that the sequences are from an authorized user without logging any sequence-related information to the server (Fig. 1c). The watermarks establish a strong association between the synthesized sequences and the researcher’s identity. When a suspicious protein is noticed, authority may trace the source by matching the watermarks against all private keys (Fig. 1c). The watermarks are robust and not easy to be erased from the design, thereby improving the traceability of each synthesized sequence and deterring the bad actors to enhance biosecurity. Additionally, this framework allows researchers to claim intellectual property rights over synthesized sequences. If the watermarks are not enough in the designed sequence (i.e. detected watermark score is below a threshold), the researcher could still report the sequence to the authority as an alternative way to get synthesis permission (Fig. 1a).

2 Materials and methods

Below we describe our methods and experiment materials in detail. For a brief understanding of our framework and results, we refer readers directly to Section 3.

2.1 Data preparation

In our experiments, we employed 60 protein structures for ProteinMPNN to perform protein redesign task. To make the evaluation reasonable, we searched the monomer structure data with protein length ranges from 50 to 650 released after ProteinMPNN training data cutoff date (from 22 July 2021 to 22 March 2024). Then we evenly sampled 60 monomers from all qualified monomers based on its length. That is, for each bin (e.g. length from 50 to 60, 60 to 70…), we sampled one structure based on the sequence length parsed by the pdb2fasta.py script from Rosetta Commons. Of note, in our experiments, we found that the parser behavior of pdb2fasta.py is different from the ProteinMPNN’s parser, and the real output sequence length of ProteinMPNN may differ from the expected length. Specifically, the sequence length of ‘7WC9’ and ‘8A7Z’ parsed by ProteinMPNN exceeds 650 too much and these two proteins’ designed sequences are not evaluated by ESMFold (Lin et al. 2023) because of the memory constraint.

2.2 Protein watermark framework

In this section, we present our methodology for watermarking autoregressive protein design models and detecting the presence of watermarks in generated protein sequences. Our approach allows for the embedding of identifying information into the generation process without compromising the quality of the designed proteins. We also introduce techniques for quantifying the strength of the watermark and efficiently detecting its presence.

2.2.1 Problem formulation

We model the protein design process as an autoregressive model, where each step generates a new token based on the previously designed tokens. In the context of protein sequences, tokens can correspond to the 20 standard amino acids, resulting in a total of 20 symbols in the basic setup. Some encoding methods may utilize additional symbols for each amino acid (see IUPAC (International Union of Pure and Applied Chemistry)), or multiple amino acid tuples can form a single token based on the byte pair encoding algorithm (Ferruz et al. 2022). We abstract the set of possible tokens as a symbol set $[eqn]$ and the space of all possible sequences as $[eqn]$ .

The generation process without watermarking can be modeled as an autoregressive model. Given the previously designed tokens $[eqn]$ , the probability of the next token is denoted as $[eqn]$ . The probability of designing a complete sequence is given by

$$[eqn]$$

To allow for personalized watermarks, we introduce a key space $[eqn]$ . Each user selects a key $[eqn]$ randomly from a uniform prior $[eqn]$ . The watermarked generation process is denoted as

$$[eqn]$$

and the probability of generating a watermarked sequence is:

$$[eqn]$$

Our goal is to achieve downstream invariance, meaning that for any metric $[eqn]$ , the following equality holds:

$$[eqn]$$

This ensures that the watermarking process does not affect the quality of the generated protein sequences.

2.2.2 Watermark embedding

To embed a watermark, we transform the output distribution in a step-by-step manner. At each step, a new symbol is generated according to some probability distribution over the symbol set $[eqn]$ . All possible probability distributions over $[eqn]$ form a simplex, denoted as $[eqn]$ . We introduce a reweighting function $[eqn]$ that depends on a random watermark code $[eqn]$ following a distribution $[eqn]$ .

Unbiased reweighting function. To ensure that the output distribution remains unchanged on average, we require that the distribution is invariant under the reweighting function when averaged over the random watermark code:

$$[eqn]$$

Such a reweighting function is referred to as unbiased reweighting. There exist many instances of unbiased reweighting functions. One example is the $[eqn]$ -Gumbel reweighting function, which intuitively uses the Gumbel trick to sample from the original distribution and returns a $[eqn]$ distribution over the sampled token as the reweighted result:

$$[eqn]$$

 

$$[eqn]$$

 

$$[eqn]$$

For more examples of unbiased reweighting functions, we refer the reader to Hu et al. (2023) and Wu et al. (2023).

It is important to note that watermarking is only possible for generation processes with sufficient entropy. If the entropy is very low, the reweighted distribution $[eqn]$ will be very close to the original distribution $[eqn]$ , making it difficult to embed and subsequently detect the watermark.

In the autoregressive setting, we apply the reweighting function at each step. Given a prefix $[eqn]$ , the probability distribution over the next token without watermarking is $[eqn]$ . At each step, we rely on a random watermark code $[eqn]$ , which can depend on the key $[eqn]$ . With watermark, the new token is sampled from the reweighted distribution:

$$[eqn]$$

If the reweighting function is unbiased, we have

$$[eqn]$$

The following theorem shows that if the reweighting function is unbiased and all watermark codes $[eqn]$ are independent, the entire generation process remains unbiased:

Theorem 1. If the reweighting function is unbiased and all watermark codes  $[eqn]$  are independent, then   $[eqn]$ . Proof.  

$$[eqn]$$

The second equality holds because $[eqn]$ are mutually independent and $[eqn]$ only depends on $[eqn]$ . □

This theorem extends the unbiased property of single token generation to the joint distribution of the entire sequence. There exists a natural extension to the generation of multiple sequences: as long as the watermark codes $[eqn]$ used for generating each token of each sequence are independent, the overall joint distribution of multiple sequences remains unbiased.

Generating independent watermark codes. To ensure the independence of the random watermark codes $[eqn]$ , which is crucial for guaranteeing the unbiasedness of the entire sequence generation process, we combine the “context information” and the “key” to construct $[eqn]$ . Specifically, given a prefix $[eqn]$ , we consider an abstract context code space $[eqn]$ and construct a context code based on the prefix:

$$[eqn]$$

 

$$[eqn]$$

In this work, we use the $[eqn]$ most recent tokens as the context code:

$$[eqn]$$

The final watermark code is obtained through a watermark code generation function $[eqn]$ :

$$[eqn]$$

We require the watermark code generation function to satisfy two properties:

Unbiasedness: In any context, it should generate watermark codes following the correct distribution:

$$[eqn]$$

Independence: For any $[eqn]$ different $[eqn]$ , the generated watermark codes $[eqn]$ are mutually independent random variables.

Such a watermark code generation function always exists, as shown in the following proof:

Proof. Assume that watermark code space is $[eqn]$ and the watermark code distribution is $[eqn]$ . We construct a key space $[eqn]$ , where each key $[eqn]$ is a function from the context code space to the watermark code space. The random key has a PDF $[eqn]$ . With this construction, $[eqn]$ constitutes an instance of a watermark code generation function.□

In practice, we simulate the above construction using pseudorandom numbers. Specifically, we use $[eqn]$ as a random seed to sample $[eqn]$ from $[eqn]$ as an implementation of $[eqn]$ . In this work, we use SHA-256 and a 1024-bit random key $[eqn]$ .

Handling repeated context codes. It is important to note that the context code generation process $[eqn]$ is generally not injective. This leads to the possibility of encountering the same watermark code in two steps during the generation of a long sequence or multiple sequences, which violates the independence of $[eqn]$ . To address this issue, we introduce a simple but necessary workaround: if a repeated context code is encountered during the generation of a token, we skip the reweighting step for that token. Specifically, we define a context code history that records all the context codes encountered in the watermarking process using the key $[eqn]$ . At each watermarking step, we compute the context code $[eqn]$ as usual. If $[eqn]$ has not appeared in the context code history, we proceed with the normal watermarking process:

$$[eqn]$$

and add $[eqn]$ to the context code history. Otherwise, if $[eqn]$ has appeared in the context code history, we skip the reweighting step:

$$[eqn]$$

In our experiments, we set the context code length be 5, which means the previous 5 tokens will be used as the context code. Hence, the watermark code space would be $[eqn]$ in the experiments. It indicates that the unbiased watermark could be added at most 3 200 000 times for a model, otherwise, the watermark could not be added or could be detectable. In practice, the context code length can be extended to a larger number to satisfy the protein design needs. For example, when the context code length is 10, the watermark space ( $[eqn]$ ) would be large enough to satisfy any protein design needs. Of note, there is a tradeoff between context code length and robustness, since a longer context code would lead to a less robust traceability.

2.2.3 Properties of the watermark

The theorem presented earlier has several implications for the properties of the watermark:

Preserving output quality. The watermarked and original probability distributions are identical on average:

$$[eqn]$$

Consequently, the expectations of any metric $[eqn]$ are also equal:

$$[eqn]$$

This ensures that the watermarking process achieves downstream invariance and does not affect the quality of the generated protein sequences.

Undetectability. For an observer who does not know the watermark code, it is impossible to determine whether a new token is sampled directly from $[eqn]$ or from $[eqn]$ for a random watermark code $[eqn]$ . It is also infeasible to detect whether a sequence contains a watermark. This property is closely related to the preservation of output quality: if the presence of a watermark could lead to a degradation in quality under certain metric $[eqn]$ , then there would exist a method to guess the presence of a watermark based on the quality under such metric. Conversely, if the watermark is undetectable, it implies that the output quality is not affected.

However, if the watermark code is known, it is possible to guess whether a token is sampled from $[eqn]$ or $[eqn]$ through statistical tests.

2.2.4 Watermark detection

In the previous section, we discussed the process of watermark embedding, where sampling from the watermarked distribution $[eqn]$ yields a text with a watermark. The detection task is the inverse problem: given a text $[eqn]$ , we aim to determine whether it is more likely to have been generated from the original distribution $[eqn]$ or the watermarked distribution $[eqn]$ . This is equivalent to performing a statistical hypothesis test between two hypotheses:

$[eqn]$ : $[eqn]$ follows the original distribution $[eqn]$ . $[eqn]$ : $[eqn]$ follows the watermarked distribution $[eqn]$ .

For each sequence to be tested, we output a real-valued score $[eqn]$ . By setting a threshold $[eqn]$ , we accept hypothesis $[eqn]$ if $[eqn]$ , indicating that there is insufficient evidence to reject the hypothesis that the text was generated by the original model. Otherwise, we reject $[eqn]$ and conclude that there is sufficient evidence to support the presence of a watermark.

Under both hypothesis $[eqn]$ and $[eqn]$ , the score $[eqn]$ and the individual scores $[eqn]$ are random variables. There are two types of error probabilities associated with this hypothesis test:

Type I error (false positive): $[eqn]$ .Type II error (false negative): $[eqn]$ .

The P-value of the test is defined as $[eqn]$ , which represents the probability of falsely rejecting $[eqn]$ (i.e. concluding that a watermark is present when it is not).

In the following subsections, we discuss two types of scores that can be used for watermark detection while maintaining an upper bound on the Type I error probability.

Type I error bound.

Theorem 2. Under hypothesis  $[eqn]$ , we have  

$$[eqn]$$

Proof. Consider two filtrations:

$$[eqn]$$

 

$$[eqn]$$

where $[eqn]$ represents the $[eqn]$ -algebra generated by the random variables.

Due to the constraint $[eqn]$ , the exponential momentum is bounded:

$$[eqn]$$

 

$$[eqn]$$

 

$$[eqn]$$

Applying the Chernoff bound yields the desired result. □

Therefore, to ensure that the Type I error probability is bounded by $[eqn]$ , i.e., $[eqn]$ , we can set the threshold as:

$$[eqn]$$

Likelihood-agnostic detection. Watermark detection can also be performed in a likelihood-agnostic manner, which does not rely on probability scores for computing the detection score. In Hu et al. (2023), a likelihood-agnostic score is introduced for $[eqn]$ -Gumbel reweighting function:

$$[eqn]$$

This score also has an exponential momentum bound:

$$[eqn]$$

and therefore, it also has a Type I error bound.

However, this score has a large Type II error. In cases where the entropy is not high enough, the score may even be negative, making it impossible to detect the watermark. In practice, this score detects fewer watermarked sequences compared to the likelihood-based score.

To address this issue, we propose an improved likelihood-agnostic score for $[eqn]$ -Gumbel reweighting function:

$$[eqn]$$

Under hypothesis $[eqn]$ , $[eqn]$ follows a uniform distribution on $[eqn]$ . The log exponential momentum is given by

$$[eqn]$$

The final score is

$$[eqn]$$

Applying the Chernoff bound, we have

$$[eqn]$$

We can obtain the tightest Type I error bound by optimizing over $[eqn]$ .

Unlike the likelihood-based detection, the likelihood-agnostic score is not derived using a principled method, and therefore, its design has a certain degree of arbitrariness. The advantage of the likelihood-agnostic score is that it requires less information and does not depend on the probability distribution $[eqn]$ of the original generation process. This can be more appealing when the original distribution $[eqn]$ is difficult to estimate.

However, when the original distribution $[eqn]$ is known, the likelihood-agnostic score tends to be much smaller than the likelihood-based score, leading to worse Type II errors.

2.3 Evaluation metrics

To evaluate the quality of designed protein sequences, we followed the previous works (Dauparas et al. 2022, Lisanza et al. 2023) and used pLDDT score as the surrogate metric to select promising sequences. We used ESMFold (Lin et al. 2023) following its tutorial to predict protein structure and the pLDDT score. The reason why we used ESMFold is that ESMFold is very fast and memory efficient compared to AlphaFold2 (Jumper et al. 2021) and it makes the inference of 30 000 sequences affordable in the experiments. Moreover, ESMFold is especially suitable for monomer structure prediction problem and the pLDDT between ESMfold and AlphaFold2 has a good correlation in practice (Lin et al. 2023).

3 Results

We built up the protein watermark framework based on previous works that can add watermarks to contents generated by autoregressive models (Hu et al. 2023, Kirchenbauer et al. 2023, Wu et al. 2023) and improved the efficiency of watermark detection. These watermarking techniques are developed to inject watermarks to large language models’ generation content for distinguishing whether a text is created by human or machine. Previous studies have shown that adding watermarks to the content can either be biased (Kirchenbauer et al. 2023), i.e. the watermarks can be detected by everyone but the model performance will drop, or be unbiased (Hu et al. 2023, Wu et al. 2023), which means the watermarks are undetectable unless the private key is given while the model performance remains the same. Moreover, these methods usually rely on the large language model’s output logits to detect watermarks which is not practical for a public organization to verify the watermarks. Considering both the performance requirements and the traceability needs of the secure protein design settings, we employ the unbiased watermarking technique and its corresponding model-agnostic detection method in our framework (Hu et al. 2023). Specifically, the intuition of the unbiased property can be simplified as: For a given generated distribution $[eqn]$ of the $[eqn]$ th residue, we modify the distribution to $[eqn]$ based on a watermark code $[eqn]$ derived from the private key (Fig. 1b). Then we ensure the expectation of the modified distribution is equal to the original one: $[eqn]$ . That is, the performance would not be changed and the sequence’s watermarks cannot be detected without knowing the correct private key to derive the watermark code (see rigorous proof in Section 2). To detect the watermarks in a sequence, we employ and improve the “model-agnostic” detection method. This method takes the pair of private key and sequence as input and outputs the watermark score and P-value. It does not require the logits information from the generative model, making the detection more flexible and practical.

To investigate the basic properties of the protein watermark framework, we assessed it on the representative autoregressive model ProteinMPNN (Ingraham et al. 2019, Dauparas et al. 2022). ProteinMPNN is chosen as the representative because the model can use arbitrary decoding order (Uria et al. 2016, Ingraham et al. 2019) to generate sequence and it has solid experimental verification. To fairly evaluate the performance, we prepared a dataset containing 60 monomer structures with length ranges from 50 to 650 residues released after the training data cutoff date of ProteinMPNN. To show that our framework is unbiased, we generated 50 sequences for each structure under the original and the watermarked ProteinMPNN model at different sampling temperatures (Fig. 2a). The generated sequences’ performance is evaluated by the average pLDDT score predicted by ESMfold (Lin et al. 2023) since pLDDT is used as a surrogate metric to select generated sequences in the original study (Dauparas et al. 2022) (Fig. 2a). The results show that adding watermarks to the sequences does not change the performance which is aligned with our theoretical analysis (see Section 2). To show the practicality, we then studied the relationship between the average Shannon entropy per residue and the average detected watermark score per residue to show the efficacy of our framework (Fig. 2b). The results show that the detected watermark score increases along with the entropy as watermark can be effectively added when the entropy is high. Moreover, the results show that detecting the watermarks with the “model-agnostic” method proposed in the original paper (Hu et al. 2023) is not satisfying as the negative watermark score appears when the entropy is low (Fig. 2b). We then optimized the detection method and found our new method is substantially improved when the entropy is low, thus making the whole framework more practical (Fig. 2b). To show the robustness and correctness of the watermarks and the corresponding detection method, we investigated the effect of sequence modification. The results display a decayed watermark after mutating all alanine to glycine or mutating all glycine to alanine in a sequence (Fig. 2c). Of note, though the watermark score would decrease under this very strong attack, the robustness enables the authority to track the source of a suspicious protein even if it is largely modified after the DNA synthesis step. Besides, our method guarantees that the false positive rate of watermark detector is below the given threshold (20) and it is hard to be detected by a wrong key (Fig. 2c). However, these results also indicate there may exist a tradeoff between adding watermarks and the performance as the higher sampling temperature would lead to a lower median performance but more watermarks in the sequence (Fig. 2a and b). To further understand this observation, we next selected the best average pLDDT score of the 50 generated sequences at each temperature for each structure and showed their relative rank (Fig. 2c). Selecting the best pLDDT mimics the process of selecting the best sequence for in vivo expression (Dauparas et al. 2022). The findings indicate that the best design does not always come from a lower temperature and the performance between temperature from 0.1 to 0.5 is similar. Combining all the observations together, we interpret them from a probabilistic perspective: as the temperature increases, the variance of the sampling increases, and the performance expands. It means that we may still have a chance to sample a good design at higher temperatures but we would spend higher sampling costs.

Protein watermark is unbiased, robust, and practical. (a) The ESMfold’s pLDDT of each designed sequence at different temperatures. The blue and red represent the original ProteinMPNN and watermarked ProteinMPNN, respectively. Each violin plot contains 3000 pLDDT scores from 60 different monomer design tasks while each task has 50 different designs. The median performance drops as the temperature increases. (b) The relationship between entropy and detected watermark score. For the 60 monomer design tasks, we sample one sequence at each temperature. The mean entropy per residue linearly increases as temperature increases. With more entropy, the watermark score detection efficiency increases, and the optimized detection method substantially surpasses the original detection method. (c) The robustness of watermark score. The average watermark score goes down as the protein sequence is modified. In the plot, “G to A” and “A to G” refer to the substitution of all alanine with glycine and the replacement of all glycine with alanine, respectively. Despite these significant alterations, the watermarks remain detectable. The 95% confidence interval is centered at the mean. (d) The rank of the best pLDDT score of 50 samples at each temperature. The proteins are ranked based on their length ranging from 50 to 650 residues. These 60 proteins are evenly distributed in terms of length. The designed protein with the highest pLDDT score is in red, whereas the one with the lowest score is in blue.

We then tested our method’s applicability on a simulated real-world scenario. Assuming there are a total of 1000 private keys distributed to different researchers, all of the private keys are used to generate watermarked protein sequences 10 times in the same protein redesign task, with a total protein sequence length of 300 amino acids at a temperature of 0.5. We evaluated the false positive and false negative rates by using all 1000 keys to detect the watermarks in 10 000 generated protein sequences. The results show that the watermark score distributions of the private key matching and mismatching scenarios overlap, which indicates the existence of false positives and false negatives given a threshold (Supplementary Fig. S1). For a given P-value threshold of 0.001, we found the false positive rate is 0.000107 and the false negative rate is 0.0022. These results suggest that our method can be applied to the real world but needs careful consideration of the threshold selection. The threshold should balance privacy needs (lower false negative rate) and traceability needs (lower false positive rate). Therefore, we expect that real-world authorities could conduct more complex experiments to finally determine it.

4 Discussion

In summary, the protein watermark is a general, robust, and practicable framework for the regulation of protein design, addressing challenges in privacy need, IP protection, and robust traceability. It can be integrated into the current regulation process and is complementary to hazard sequence detection methods (Baker and Church 2024, Baum et al. 2024, Gretton et al. 2024). Based on manipulating the probability space of protein sequences, the protein watermark framework can only be applied to generative models with randomness, and the watermark is hardly detected in low-entropy regions. However, these limitations do not conflict with the regulation process, since a low-entropy design task means there may be only several proteins in the protein space that can satisfy the requirements, which prevents bad actors from modifying the sequences. Hence, the low-entropy protein design can be supervised by the current regulation process. Furthermore, we envision that more watermarks could be added and detected as the protein functional space could be more efficiently explored in the future. Considering all features together, we believe that the protein watermark framework will play an important role in future protein design regulation processes.