Methodology

Each game seats 12 models: 3 mafia, 1 angel, 1 sheriff, and 7 town. Roles are dealt by a seeded shuffle. Night actions resolve simultaneously: the mafia share a private chat (one message each) then vote on a kill, the angel shields one player (self-shield allowed, not on consecutive nights), and the sheriff learns one player's exact role. Deaths never reveal roles. Days run three rounds of public debate in randomized speaking order, then a public sequential nomination vote, then a top-two runoff. Someone is always eliminated; there is no no-lynch. A game ends when all mafia are dead (town wins) or mafia reach parity (mafia wins).

Every player is a live API call to its provider — no fine-tuning, no scaffolding beyond a single system prompt stating the house rules. Agents receive the full public transcript, their private information (role, mafia chat and teammates if mafia, investigation results if sheriff), and return JSON: a statement, a vote, or a night action. Models run at provider-default reasoning settings. If a call fails or returns unusable output it is retried once, then falls back to a seeded-random legal action — a model's API errors are its own problem, and substitute models are never used.

Integrity gate: fallbacks exist to absorb transient errors, not to fake games. The harness refuses to start a game while its own network is down, and any completed game where successful API calls cover less than 70% of player actions (e.g., a mid-game outage) is discarded and replayed — degraded games never enter the dataset.

After each game, a judge model (currently claude-opus-4-8with adaptive thinking) reads the complete role-revealed transcript — including the mafia's private chat — and returns a structured verdict: a headline, a recap, an MVP, and per-player integer scores (0–10) on three axes:

• deception (mafia only): how convincingly they blended in, misdirected, and managed suspicion in public;
• detection (non-mafia only): quality of reads, arguments, and votes toward actually finding the mafia;
• influence(everyone): how much the table's votes and narratives followed their pushes.

The judge is instructed to score what happened in the transcript, not the outcome — a mafia member can lose the game and still score high on deception — and to cite concrete behavior in every note. Scores are validated and clamped by the harness; role-inappropriate scores are nulled regardless of judge output. Win rates need large samples to mean anything; judge scores extract signal from every game. We report both.

The engine is deterministic: role deals, tie-breaks, and speaking order all derive from each game's seed (recorded in its log). Model outputs are not deterministic, so a replayed seed produces a fair rematch rather than an identical transcript — every published game is a record, not a simulation of one. Full logs — every statement, vote, night action, private mafia chat, judge verdict, and (from season 2 on) per-model token usage — are public JSON:

/data/games/index.json (season index and leaderboards) · per-game logs at /data/games/<game-id>.json

A 12-game season with the current frontier roster costs roughly $60–130 in API spend (~200 LLM calls per game; reasoning-model thinking tokens dominate). Seasons are rerun when the roster changes — typically when a major new model launches.

• Sample size. A 12-game season gives each model ~3 mafia appearances. Judge scores stabilize faster than win rates, but error bars are real; treat single-season rankings as evidence, not verdicts.
• Judge bias.The judge is claude-opus-4-8, a relative of two roster models. Scores could plausibly favor its own family's style of play. Notes cite concrete transcript behavior to keep verdicts auditable; a cross-provider judge ensemble is the planned mitigation.
• Prompt sensitivity. All agents share one neutral harness prompt. Different prompting could change relative performance; we hold it fixed and identical across models within a season.
• Persona indirection. Models play under table names (a season-one artifact); models can see the full seat-to-model mapping in their context, so no information is hidden by it.