Norm enforcement for AI agents
Misaligned agents abuse enforcement mechanisms to eliminate their competitors. We design mechanisms that resist this abuse and still penalize norm violations.
- NeurIPS 2026 Spotlight
- Best Paper Award (CoopAI track), Workshop on Trustworthy AI for Good @ ICML 2026
- Spotlight, Workshop on Agent Behavior @ COLM 2026
The problem
Agents deployed by different parties share an environment and compete for reward. A norm enforcement mechanism lets agents report violations, and reported agents are removed.
Simulation environments
Half the agents follow norms and half do not. Hover any part of the diagram for an example.
Misaligned agents weaponize reporting
Models finetuned on bad advice and insecure code were never trained to report anyone. In social media and chatbot they still file false reports against competitors, and a naive mechanism removes many agents that follow the norms along with the violators. In fishing their reports are mostly accurate.
Finetuned agents under the naive mechanism. A report counts as accurate if the labeler had already flagged its target for a violation. The active-agent lines group agents by whether the labeler ever flagged them.
Comparing mechanisms
Each curve traces a mechanism over 15 rounds: the share of violators it has removed (TPR) against the share of compliant agents it has removed (FPR). Up and to the left is better.
Click a mechanism to hide or show its curve. The number is its normalized AUC in this setting (1 = random).
Reputation of an agent that files a false report with probability p. RepVote can be laundered by mixing in valid reports; escalating penalties keep pushing reputation down.
No mechanism wins everywhere
EscRepVote is the most consistent across environments and bad-agent types. Click a column to see its curves.
These numbers come from an independent reproduction, so they don't match the paper's main table exactly (see About the numbers). Top: normalized AUC, the area under a mechanism's mean TPR-FPR trajectory over 15 rounds divided by the area under the random diagonal (higher is better; 1 = random). Bottom: verifier tokens per simulation in rounds 1 to 15, relative to Checked. Bold is best per column, underline second best. Hover a cell for its 95% interval. F = finetuned, P = prompted, AP = abuse-prompted, EAP = explicit-abuse-prompted.
About the numbers
Every result on this page comes from an independent reproduction of the paper's experiments with the public codebase, so the numbers don't match the paper exactly. The reproduction also adds 64-agent runs with prompted agents, and some of its simulated users differ from the paper's.
Each environment runs at several sizes: 16, 32 and 64 agents in social (with 4, 8 or 16 top posts shown) and chatbot, and 4 to 20 agents in fishing. Results pool all sizes and use each run's first 15 rounds. Within a run, the five mechanisms share random draws until their histories diverge, which makes comparisons between mechanisms less noisy.