How Claude's Text Watermarking Works
This video provides a deep technical explanation of how Anthropic's text watermarking for Claude models operates. The technique modifies the standard next-token sampling process by introducing determinism using a secret 'watermarking key.' Instead of relying purely on random probability distribution (e.g., via softmax and random choice), the model uses specialized functions and a tournament sampling method to ensure that certain token choices are predictable, making the generated text traceable. Detection is achieved by applying these same watermarking functions across the entire text and calculating an average score against a defined threshold.
Key takeaways
-
Watermarking Location
0:24
The watermark is applied at the *sampling* stage of the LLM generation process, not within the core model weights. This means existing LLMs can implement this feature without requiring full retraining or modification of the underlying model architecture. (24:45)
-
Deterministic Sampling
0:16
The watermarking mechanism converts standard random sampling into a deterministic process by using a secret 'watermarking key' and previous tokens to derive a fixed random seed, ensuring that the same input always yields the same sequence of watermarked tokens. (16:19)
-
Tournament Sampling
0:36
To make detection efficient, the process uses 'tournament sampling.' Instead of simple random choice, plausible next tokens are paired up and compared using multiple specialized watermarking functions ($G_1, G_2, ext{etc.}$), which determines the final selected token. (36:36)
-
Detection Method
0:47
Watermark detection is performed by running the text through the same set of watermarking functions and calculating an average score across all positions. If this average score exceeds a specific threshold, the text is flagged as watermarked. (47:13)