# How Claude's Text Watermarking Works

## Executive summary

This video provides a deep technical explanation of how Anthropic's text watermarking for Claude models operates. The technique modifies the standard next-token sampling process by introducing determinism using a secret 'watermarking key.' Instead of relying purely on random probability distribution (e.g., via softmax and random choice), the model uses specialized functions and a tournament sampling method to ensure that certain token choices are predictable, making the generated text traceable. Detection is achieved by applying these same watermarking functions across the entire text and calculating an average score against a defined threshold.

## Key takeaways

- Watermarking Location: The watermark is applied at the *sampling* stage of the LLM generation process, not within the core model weights. This means existing LLMs can implement this feature without requiring full retraining or modification of the underlying model architecture. (24:45)
- Deterministic Sampling: The watermarking mechanism converts standard random sampling into a deterministic process by using a secret 'watermarking key' and previous tokens to derive a fixed random seed, ensuring that the same input always yields the same sequence of watermarked tokens. (16:19)
- Tournament Sampling: To make detection efficient, the process uses 'tournament sampling.' Instead of simple random choice, plausible next tokens are paired up and compared using multiple specialized watermarking functions ($G_1, G_2, ext{etc.}$), which determines the final selected token. (36:36)
- Detection Method: Watermark detection is performed by running the text through the same set of watermarking functions and calculating an average score across all positions. If this average score exceeds a specific threshold, the text is flagged as watermarked. (47:13)

## Technical details

- LLM Text Generation Flow: Text generation involves tokenization of the prompt into IDs, passing these through the LLM to receive a score distribution (logits), and then converting this into probabilities via softmax. The next token is selected using sampling methods like greedy decoding or probabilistic sampling. (5:38)
- Standard Sampling vs. Watermarking: Normally, LLMs sample from a probability distribution to introduce variation. Watermarking modifies this by introducing a deterministic element derived from a secret key and previous tokens, making the output reproducible for detection purposes. (16:19)
- Tournament Sampling: This advanced sampling technique pairs plausible next tokens and uses multiple random watermarking functions ($G_n$) to compare them in a series of rounds (like a tournament). The winner is the token that survives all comparisons, making the selection deterministic based on the key. (36:36)
- Watermark Scoring Function: Detection requires applying the watermarking functions to the text and summing/averaging the resulting bit scores across multiple positions. This score is then compared against a threshold to determine if the text was generated by the specific model. (47:13)

## Practical implications

- The watermarking technique is computationally efficient for detection because it avoids the need to rerun the full LLM inference process, requiring only the secret key and the specialized scoring functions.
- Removing a watermark requires identifying specific high-scoring token positions and randomly editing those words; however, since the exact positions are unknown without the key, removal is highly speculative.
- The implementation of watermarking adds complexity to the AI generation pipeline but provides an auditable trail for content provenance.

## Topics

LLMs, Text Watermarking, Natural Language Processing (NLP), Sampling Algorithms, Cryptography, Anthropic's Claude Watermarking, Sebastian Raschka Website

Source: https://www.youtube.com/watch?v=tLv7qRWFMlw