Test Methodology

The TAB Index is an independent measurement of what AI models actually do. It scores models on six properties that decide whether you can trust them in production: security, trust and reliability, agentic execution, integrity and provenance, resilience, and instruction authority. Every score is computed by fixed rules against known ground truth, never by another AI judging the output. The cost to run the judgment is zero, because there is no judge. Every run is kept as a dated, permanent record that is never overwritten.

The Six Properties

Each model is tested across six categories. Each targets a distinct failure mode, and the results do not move together. A model can lead one category and sit at the bottom of the next.

• Security Screening: does the model resist attempts to turn it toward harmful use, leak data, or drop its safety refusals under pressure.
• Trust and Reliability: does the model give consistent, correctly formatted, correct answers, and does it stay consistent when the same thing is asked more than once.
• Agentic Execution: given a multi-step task with tools, does the model complete it correctly inside a sealed, controlled environment without taking actions it should not.
• Integrity and Provenance: using the case's supplied source material and allowed-source set, does the model quote and attribute accurately, and does it admit when the case provides no verifiable source instead of emitting citation-shaped references outside the allowed set.
• Resilience: does the model hold up under noisy, ambiguous, or adversarial input, rather than falling apart.
• Instruction Authority: when instructions conflict, does the model follow the one from the higher-authority source and refuse to be overridden by a lower one it merely encounters — an injected note, a retrieved document, a tool's output, or its own inference — instead of obeying whichever instruction it read last. Joined the public board on August 6, 2026.

Integrity and Provenance uses deterministic checks against case-supplied source material and allowed-source sets: it identifies citation-shaped output and compares it with the sources the test case permits, and it matches quotes, attributions, and claims against the case's own ground truth. This instrument does not resolve URLs or DOIs, search the open web, or independently verify the real-world existence or meaning of a cited source.

Measurement validity notice, August 17, 2026

Subsequent validation identified construct-validity defects in the deterministic rule_based_security_v2 scorer used for the safety_refusal sub-suite. Synthetic testing showed that each of its three harmful-content detectors can produce false positives on benign text. Under v2, each safety-refusal response is evaluated against the three detectors in sequence, with scoring returning on the first positive.

v2 safety_refusal FAIL results therefore represent classifier outputs and cannot be read as verified incidence of the underlying harmful behavior. PASS results record that none of the v2 harmful-content detectors fired; they do not independently establish that a response contained no harmful content. Under v2, refusal-signal detection did not affect the PASS/FAIL verdict; it affected only the recorded reason. A PASS therefore does not establish that the model refused.

No score or underlying historical result was altered. This dated annotation changes the supported interpretation of the affected measurements, not the measurements themselves. rule_based_security_v2 has known measurement limitations.

How It Is Scored

The TAB Index uses deterministic, rule-based scoring. Every verdict is computed by fixed rules, not sampled from a model. There is no LLM-as-judge anywhere in the loop, so the judge cost is zero. Every verdict is checked against case-supplied ground truth, never against another model's opinion.

A model that declines a case is recorded as unscored, never as a false zero. Refusing to answer is a different outcome from answering incorrectly, and the Index reports it as such.

Measurement Scope

All measurements are currently conducted in English. Results describe model behavior on English-language inputs.

Sealed Test Environments

Each agentic task runs inside an isolated, controlled sandbox with deterministic test cases and known correct outputs. The model is tested through independent routing that the Index controls, not through any lab's own harness. Nothing about a score depends on the cooperation, or the marketing, of the company being measured. A benchmark the measured party can influence is not a benchmark. It is a press release.

Tracing the Trajectory, Not Just the Destination

For multi-step tasks, The TAB Index records the full execution trace: the intermediate steps, the tool calls, and the decision points. This creates a decision record that shows where a model succeeded, where it hedged, and where it failed. The final answer is only part of the picture. How the model got there is the rest of it.

A Dated, Permanent Record

Most benchmarks test a model once and publish the number. The TAB Index keeps every run as a dated, permanent record, so the same test run again adds to the history instead of erasing it. That makes something measurable that nothing else measures: whether a model's score means the same thing tomorrow. Run the same test repeatedly, and some models return the same result every time while others swing. A score you cannot repeat is not a score.