The token-efficiency study — putting real numbers behind the claim

The honest framing

"We have anecdotal data that wikiTaTa makes your Claude and other-agent work more efficient — but we acknowledge it's anecdotal. Help us gather data to bring factual numbers."

That framing is scientifically honest. It lowers expectations, guards against greenwashing, and is a genuine invitation to the community.

Measure mechanisms, not totals

Don't try to prove that "total tokens drop". Raw totals mislead: a more productive user spends more. There is also an honest tension: a three-agent review spends more tokens per question, while memory and less rework save tokens. Instead, measure three narrow efficiency mechanisms that can each be attributed:

1. Context re-establishment — tokens spent re-explaining the project at the start of each session. Memory should cut this, and it's the cleanest metric.
2. Rework and dead ends — tokens spent on paths later abandoned or corrected. Adversarial review should cut these.
3. Prompt-cache hit rate — structured context that the model provider can reuse is real money and tokens saved.

The design

• Phase A: a normal workflow with token tracking only.
• Phase B: the same user with full wikiTaTa.
• Confounds: learning and project drift are handled by counterbalancing the order, longer windows, a washout period, and ideally a cohort that uses wikiTaTa from day one.

Rigour

• Register the hypotheses, metrics and analysis plan in advance. Report every participant, including nulls.
• Collect token counts and session boundaries only, never prompt or response content. Participants are pseudonymous from the start.
• At small samples, call the result directional evidence. Any environmental claim stays secondary and quantified, never "carbon neutral".

Be ready for a null

If three-agent spend cancels out the memory savings, the honest story becomes better outcomes per token rather than fewer tokens. Deciding now that this is an acceptable result to publish is part of the discipline.