Token Efficiency & Agent Intelligence

What we track

Every Claude Code session that runs inside the wikiTaTa MCP context emits structured telemetry at each turn. No manual instrumentation required — data is captured automatically via the wikiTaTa MCP server (stdio mode, runs locally).

Signal	Table	Granularity
Input tokens, output tokens, cache reads/writes	`service_usage`	per turn
Model ID, provider, cost estimate	`service_usage`	per turn
Session ID, turn timestamp, user	`turn_stats`	per turn
Agent panel runs (graded tasks)	`agent_panel_runs`	per run
Per-agent score, cost, task class	`agent_panel_runs`	per run
Cumulative session burn	`burn_watch_state`	rolling

Known gap: informal Agent tool dispatches that bypass wt_agent_run / wt_agent_panel (e.g. Claude spawning subagents directly) are not yet captured. The fix is routing all dispatches through the wikiTaTa panel so every run is logged. See §Research Design.

Data flow

The chain from a single Claude turn to a routing decision is fully traceable. Every number visible in the UI traces back to a raw row in the database. Nothing is estimated or inferred — it is measured.

The feedback loop

The core claim: the system uses its own telemetry to improve future behavior. This is not a post-hoc dashboard — the loop is architectural.

Measure. wt_agent_scoreboard accumulates real trial data: which agents score highest on which task classes, and at what cost.

Surface. The scoreboard page shows efficiency bar charts (score/dollar), bubble charts (cost vs. quality with run-count radius), and radar charts (per-agent profile across task classes).

Recommend. The Agent Routing Settings tab surfaces a recommended configuration derived from trial data — the preset that maximizes score/dollar given the user's plan.

Apply. The user sets their weights (or accepts the recommended preset). On save, the preference persists to user_preferences.agent_routing. The dispatch layer reads this at runtime.

Re-measure. New runs flow back into service_usage and agent_panel_runs, updating the scoreboard. The loop closes.

This is a genuine feedback loop, not a one-time optimization. The recommendation improves as data accumulates.

Efficiency claims

Based on Trial 1 data (agent-efficiency-pilot-2026-06, 5 agents × 5 task classes):

Gemini Flash

$0 effective

7.4 avg score

Optimal for high-volume routine tasks. Google AI Pro plan = zero marginal cost.

ChatGPT (gpt-4o)

$0 effective

Competitive on synthesis

OpenAI plan = zero marginal cost. Strong on multi-step reasoning.

DeepSeek

~$0.00014/run

Best score/dollar

Excellent for structured reasoning tasks. Cheapest paid option at scale.

Grok

~$0.0018/run

Strong on hard reasoning

Worth using selectively. Higher cost justified on challenging classification tasks.

gem-gpt-deepseek preset: routing ~50% Gemini / 25% ChatGPT / 25% DeepSeek achieves near-equivalent quality to expensive-model-only approaches at a fraction of the per-run cost. This is the current recommended starting configuration.

Scope: these results apply to the tested task classes. Different users, different task mixes, different quality thresholds produce different optimal configurations. The system is designed so each user accumulates their own data and tunes their own routing.

Needed: more data.

What we have not yet proven: that wikiTaTa's MCP context reduces total token consumption relative to a no-MCP baseline. We believe it does. We have not run the controlled experiment.

To rigorously claim that the wikiTaTa MCP context improves token efficiency, we would need a controlled comparison:

Group A: same user, same tasks, same model — without wikiTaTa MCP
Group B: same user, same tasks, same model — with full wikiTaTa MCP context (session memory, card indexing, wt_session_start, tool routing)

We have not run this experiment. Removing the MCP for N sessions would disrupt active development work. The controlled experiment requires a dedicated test cohort, a consistent task battery, and a quality scoring rubric. This is on the research roadmap.

What we can say now: within the MCP-on cohort, we have real measurements of cost and quality per model per task class. The agent scoreboard is real data. The routing recommendation is derived from that data. The loop is real.

The baseline comparison would let us say "wikiTaTa MCP reduces total token consumption by X%." We cannot say that yet. We say instead: we track everything, we optimize within the controlled set, and we have designed the system to support the baseline experiment when we run it.

Research design

Proposed: Controlled MCP Comparison Study

Goal: Measure the delta in tokens/task, cost/task, and quality score between the no-MCP baseline and the full-MCP condition.

1 Recruit N volunteer users (onboarded user, opts in to Token Efficiency Mode only, using Claude Code for 10 sessions*).

2 Phase A (10 Sessions with Claude): Claude Code without wikiTaTa MCP. Log turn stats via a lightweight observer-only hook — token counts only, no MCP tools.

3 Phase B (10 sessions): full wikiTaTa MCP enabled. Same task categories, same quality rubric.

4 Measure: tokens/session · cost/session · sessions-to-task-completion · quality score (LLM-graded against reference output).

* Once onboarded with wikiTaTa and enrolled in the Token Efficiency study, Claude collects token usage only — plus one bonus: insight into your context-window pressure. Most people let their context grow unnoticed, which quietly hurts the efficiency of their conversations with Claude. wikiTaTa prompts you when it's worth starting a new chat (around 60% context — you can always keep chatting and ignore the nudge; we recommend following it). Each new chat counts as a “session,” and after 10 sessions Claude sees you've reached “session 11” and recommends the next phase: exiting the Token Efficiency study and turning the full wikiTaTa experience on.

Token Efficiency Mode: what's on, and what happens at graduation

Capability	While TEM is ON	After you turn it off (full wikiTaTa)
Turn-by-turn token tracking	On (this is the baseline data)	Still on, alongside everything else
Context-pressure alerts (~60%)	On	On
Cards, memory, session context, coordination	Off — nothing else loads	On — the full wikiTaTa experience
Session-start load	Minimal (identity + safety only)	Full context + your card index
Duration	10-session baseline, then you're prompted to graduate	Ongoing

What "quality score" means: a separate LLM judge (not one of the models under test) evaluates each session's output against a reference solution, rating 1–10 on correctness, completeness, and code quality. Same rubric as the agent panel trials.

Expected signal: we hypothesize that MCP context reduces re-derivation tokens — the agent doesn't re-explain project structure, doesn't re-ask for card IDs, doesn't re-establish session state. We estimate 15–30% reduction in input tokens per session based on observed patterns, but this is unverified.

Interested in participating?

If you're a developer using Claude Code regularly, we'd welcome you as a study participant. We'll publish results openly — including the raw data, the rubric, and the full methodology. No black-box claims.

Get in touch

The three pillars

The Agent Efficiency Suite is three integrated surfaces, each addressing a different user need. Together: Track → Evaluate → Route → Measure again.

🔥

Burn Watch

Real-time session cost tracking, daily burn totals, configurable cost alerts. You never get surprised by a large token bill — the system alerts before you hit your threshold.

Open Burn Watch →

📊

Agent Scoreboard

Empirical performance data for each AI model on your actual task types. Score, cost, win rate, efficiency (score/dollar), per-task-class breakdown. Evidence-based model selection — not marketing claims, your data.

Open Scoreboard →

⚡

Agent Routing

A user-controlled routing layer: enable/disable models, set percentage weights, apply recommended presets. The system recommends; you decide. Backed by user_preferences.agent_routing.

Open Settings →

For prospects

wikiTaTa is the only AI workspace that treats your token spend as first-class data.

Most AI tools show you a bill at the end of the month. wikiTaTa shows you — per turn — which model ran, what it cost, what it scored, and then routes future work accordingly. This is not a dashboard bolted on after the fact. It is architectural: the MCP server instruments every session, the data flows through the same database that holds your notes and cards, and the routing layer reads from the scoreboard in real time.

This is the kind of claim every AI tool makes. We are one of the few willing to say: "here is how we would prove it, here is where we are in proving it, and here is what we can measure right now."

Ready to start tracking? Request access →

Explore further

About wikiTaTa

What the system is and who builds it

Pricing

Plans, tiers, and what's included

Architecture

Interactive map of every system, service, and connection

Early access status

What's real, what's uncertain, what early access means

Contact

Reach the team or request a study slot

Responses & reactions

Comments are moderated before publication. We read every one.