A wrong query is never just a wrong query. By the time an incorrect SQL statement has done its work, it has produced a number, the number has entered a document, the document has informed a decision, and the decision has moved money, headcount, or strategy. The cost of bad SQL is the cost of everything downstream of it, which is why “our text-to-SQL system is about 90 percent accurate” is one of the least informative sentences in enterprise software. Ninety percent of what, measured how, and what did the other ten percent cost? This piece breaks down where the real costs hide, and then lays out how to measure accuracy in a way that actually predicts them.
The cost ledger: four entries, mostly invisible
The first entry is decision damage, and it is the largest and least measurable. A silently wrong figure that shapes a pricing decision, a forecast, or a resource allocation does its damage long before anyone traces it back to a misgenerated join. Because the harm surfaces far from the cause, it is almost never charged to the analytics system that produced it, which is precisely why it is the hidden cost rather than the visible one.
The second entry is verification tax. Once an organisation has caught a system being confidently wrong even once, rational users begin re-verifying its answers, and re-verification means an analyst reproducing the work by hand. At that point the system has negative productivity: it costs everything it cost before, plus the tool. The third entry is trust collapse, which behaves like a cliff rather than a slope. Users do not discount a system proportionally to its error rate; past a threshold of burned trust, they abandon it entirely, and adoption, the metric the entire business case rested on, goes to zero. The fourth entry, specific to regulated industries, is compliance exposure: a wrong number in a filing or an examined report is not an inefficiency but an incident, with remediation costs that dwarf any tooling budget.
Why the vendor’s accuracy number does not predict your costs
Accuracy claims usually derive from public benchmarks, and public benchmarks systematically overstate production performance for reasons covered at length elsewhere: small clean schemas, unambiguous questions, none of your business rules. But there is a subtler measurement problem inside the metric itself. Most benchmarks score execution accuracy, whether the query returns the expected result on the test database. In production, what matters is semantic accuracy: whether the query means what the asker meant, using the definitions the business certifies. A query can execute perfectly and be semantically wrong, computing revenue from the uncertified table or ignoring the fiscal calendar. Execution accuracy on someone else’s database is, at best, weakly correlated with semantic accuracy on yours, and semantic accuracy is the one your cost ledger responds to.
Measuring accuracy so the number means something
A meaningful accuracy programme has three components, and all three run on your own warehouse with your own questions. The first is a gold-standard evaluation set: fifty to one hundred real questions drawn from actual analyst traffic, each paired with a correct answer that your own analysts have independently produced and a business owner has signed off. The set must be deliberately adversarial in proportion to reality, including questions that depend on unwritten business rules, questions that route through your ambiguous tables, questions requiring multi-step joins, and a handful of questions that are unanswerable from the data, where the correct behaviour is to decline or ask for clarification rather than to produce something. Scoring the system’s willingness to say “I cannot answer this” is not a soft extra. Refusal behaviour is where silent errors go to die.
The second component is graded scoring rather than pass-fail. A useful rubric distinguishes at least four outcomes: correct, correct but suboptimally derived, wrong in a way a reviewer would catch, and wrong in a way that reads as plausible. That last category deserves its own tracked rate, because plausible-wrong is the failure mode that generates decision damage, and a system’s overall accuracy can improve while its plausible-wrong rate holds steady, which is a deterioration disguised as progress. The third component is continuous measurement in production. Evaluation sets go stale as the warehouse and the business evolve, so the programme should sample live queries on a fixed cadence, have analysts verify the sample, and trend the results. A platform with a complete prompt-level audit trail makes this nearly free, because every answer already carries its SQL and sources; verification becomes a reading exercise. A platform without one makes the sampling itself a reconstruction project, which is a strong signal about the platform.
Accuracy measurement is also the improvement engine
The measurement programme pays a second dividend when the platform can learn from it. In a static system, a failed evaluation item is a data point; the same failure recurs next quarter. In a system with an auto-learning semantic layer, each verified failure becomes a correction that persists: the misresolved term gets a certified definition, the wrong join path gets superseded, and the error class closes rather than recurring. This converts the evaluation set from a report card into a training curriculum, and it is the mechanism by which accuracy compounds over time instead of plateauing. Based on QuaerisAI customer deployments and published materials, this compounding loop underpins outcomes like the reduction of a standard audit engagement from 170 hours to 70 hours, a 59 percent reduction: the hours come out only when answers can be trusted at first pass, and first-pass trust is built by exactly this cycle of measurement and permanent correction.
The one-line summary for the business case
The cost of bad SQL is dominated by errors nobody catches, so the only accuracy number worth putting in a business case is one measured on your warehouse, scored for semantic correctness against certified definitions, with the plausible-wrong rate reported separately. Any vendor figure that arrives without that provenance is a marketing input, not a planning input.
Frequently asked questions
What accuracy rate is good enough for production?
There is no universal threshold, because tolerance depends on the blast radius of an error and on whether errors are catchable. A system feeding exploratory analysis with full SQL visibility can tolerate more than one feeding regulated reporting. The practical standard is a plausible-wrong rate low enough that your sampling programme rarely finds one, combined with an audit trail that makes every answer verifiable when it matters. Verifiability buys more safety than any accuracy decimal.
How often should the evaluation set be refreshed?
Quarterly is a sound default, with two triggers for off-cycle refreshes: material warehouse changes such as migrations or new source systems, and any change to the platform’s underlying model. Refresh means retiring questions that no longer reflect real traffic and adding new ones drawn from recent live queries, keeping the set adversarial in the same proportions.
Who should own text-to-SQL accuracy measurement?
The data team runs the mechanics, but sign-off on gold-standard answers belongs to the business owners of the metrics, because semantic accuracy is defined by their certified definitions. A useful pattern is a standing quarterly review where the data team presents the sampled accuracy trend and the plausible-wrong rate to the same governance forum that owns metric definitions, closing the loop between measurement and certification.