arXiv: 2604.22191 · PDF

Authors: Chaoran Chen, Dayu Yuan, Peter Kairouz

Affiliations: Google

Primary category: cs.CR · all: cs.CL, cs.CR

Matched keywords: llm, agent, agentic, inference, fine-tun, post-train

TL;DR

Behavioral Canaries audits RL fine-tuning pipelines for unauthorized use of legally protected retrieved documents by planting document-trigger/stylistic-reward pairs in preference data, detecting misuse via induced behavioral shifts rather than memorization.

Key Ideas

  • Standard memorization- and membership-inference-based audits fail for RL-trained models, since RL shapes behavioral style rather than factual retention.
  • Behavioral Canaries instrument preference data with document triggers paired with feedback rewarding a distinctive stylistic response.
  • If the protected data were used in RLFT, a latent trigger-conditioned preference emerges, which auditors can probe.
  • Establishes behavioral-signal auditing as a general mechanism for RLFT pipelines.

Approach

The framework instruments the preference dataset used for RL fine-tuning: each canary pairs a document trigger with reward feedback that favors a specific stylistic response pattern. If a provider ingests these protected documents during RLFT, the model internalizes a trigger-conditioned stylistic preference. Auditors later query the deployed model with the trigger and statistically test for the canary style, detecting distributional behavioral change rather than verbatim leakage.

Experiments

The abstract only reports empirical evaluation of detection performance on RLFT pipelines using a 1% canary injection rate, without naming specific datasets, base models, or baseline auditing methods — details are thin.

Results

At a 1% canary injection rate, the method achieves a 67% true-positive detection rate at a 10% false-positive rate, with AUROC = 0.756. The abstract frames this as evidence that behavioral signals reliably expose unauthorized document-conditioned RL training, though absolute numbers are modest.

Why It Matters

For agent and LLM-infra practitioners, this provides a tool to enforce terms-of-service and legal constraints on retrieved context in agentic workflows. It gives auditors, data owners, and compliance teams a mechanism to detect policy violations in RLHF/RLFT pipelines where existing memorization-based audits are blind.

Connections to Prior Work

Builds on canary-based auditing and data watermarking in LLM training, membership inference attacks, and verbatim-memorization extraction. Extends ideas from RLHF/DPO preference learning and from stylistic backdoor/trojan literature, reframing them as legitimate auditing primitives rather than attacks.

Open Questions

  • Can providers evade detection via canary filtering, paraphrase normalization, or preference-data cleaning?
  • How does detection scale with model size, RL algorithm (PPO vs DPO vs GRPO), and lower injection rates?
  • What are false-positive risks on benign stylistic drift, and how robust is the signal after further fine-tuning or distillation?
  • Legal standing: is a 67%/10% TPR/FPR signal admissible evidence of ToS violation?

Original abstract

In agentic workflows, LLMs frequently process retrieved contexts that are legally protected from further training. However, auditors currently lack a reliable way to verify if a provider has violated the terms of service by incorporating these data into post-training, especially through Reinforcement Learning (RL). While standard auditing relies on verbatim memorization and membership inference, these methods are ineffective for RL-trained models, as RL primarily influences a model’s behavioral style rather than the retention of specific facts. To bridge this gap, we introduce Behavioral Canaries, a new auditing mechanism for RLFT pipelines. The framework instruments preference data by pairing document triggers with feedback that rewards a distinctive stylistic response, inducing a latent trigger-conditioned preference if such data are used in training. Empirical results show that these behavioral signals enable detection of unauthorized document-conditioned training, achieving a 67% detection rate at a 10% false-positive rate (AUROC = 0.756) at a 1% canary injection rate. More broadly, our results establish behavioral canaries as a new auditing mechanism for RLFT pipelines, enabling auditors to test for training-time influence even when such influence manifests as distributional behavioral change rather than memorization.