Every NLQ demo looks the same. There is a clean schema with tables named orders, customers, and products. The column names are readable English: revenue, order_date, customer_country. Someone types "What was revenue by country last month?" and a correct SQL query materializes in about two seconds. The audience is impressed.
That demo is not a lie. In the narrow scenario it demonstrates, it works. The problem is that the scenario it demonstrates is not representative of real analytics environments. The gap between "NLQ works on the demo schema" and "NLQ is reliable enough to trust in production" is substantial, and the specific gaps are worth understanding if you are evaluating NLQ tooling for your team.
The Schema Gap: Jargon, Abbreviations, and Legacy Names
Real production schemas were not designed for human readability. They were designed incrementally by engineers solving immediate problems, and they carry years of naming decisions that made sense in context and are now opaque to anyone who was not there.
Consider a typical e-commerce or SaaS company with 3 to 6 years of data history. The events table might be called evt_v2. There may be a conv_funnel_legacy table that nobody uses except for one dashboard that finance insists on. The column that represents "activated user" might be called is_onb_complete in one table and activation_flag in another, with subtly different filter conditions in each. Revenue might appear as arr_usd_cents in one table and mrr_normalized in another, and there is probably a Slack thread from 2022 explaining why both exist.
A language model querying this schema has to infer which table and which column the user means from context. For simple aggregations against tables with readable names, it usually does well. For multi-table joins where the join key is something like ue_id (unified entity ID) and the join relationship is not obvious from column names alone, it generates incorrect SQL with meaningful frequency.
The practical mitigation here is a semantic layer or metadata annotation. If you give the NLQ system descriptions of each table and column, the schema gap narrows considerably. But writing and maintaining those descriptions is itself a data engineering task. It is not free. Teams evaluating NLQ should factor in the annotation investment as part of the total adoption cost.
Metric Definition Drift and NLQ Trust
One of the trickier failure modes for NLQ in analytics environments is metric definition drift. A stakeholder asks "What was our conversion rate last quarter?" The NLQ system generates a SQL query. The query is syntactically correct and runs. The number it returns is wrong because the conversion rate definition in the query does not match the canonical definition that the data team uses.
This failure is particularly dangerous because it is silent. The SQL runs without error. The number looks plausible. The stakeholder takes it into a meeting and uses it. The data team finds out later that the number was wrong by 2 to 4 percentage points because the NLQ system used a slightly different user filter or a slightly different attribution window.
The solution is grounding NLQ generation in a semantic layer that encodes metric definitions explicitly. If "conversion rate" is defined in your semantic layer as a specific SQL expression with specific filters, the NLQ system should resolve that term to that definition rather than generating its own interpretation. This is one of the core reasons we built our NLQ capability to reference metric definitions rather than generating raw SQL from scratch. A generated query that contradicts your canonical metric definition is worse than no query at all, because it produces numbers that look authoritative but are not.
Multi-Hop Questions and Reasoning Limits
Single-table aggregation questions are the NLQ sweet spot. "Show me revenue by region for Q1" maps to a straightforward GROUP BY query. Where NLQ systems degrade is on questions that require multi-step reasoning or multi-table joins with non-obvious relationships.
Attribution questions are a good stress test. "Which marketing channel drove the most incremental activations last month, controlling for organic baseline?" This question requires: identifying the activations fact table, identifying the marketing channel attribution table, joining them on the correct key, understanding what "incremental" means relative to "organic baseline," and computing the difference. Each step requires a judgment call that the system may get wrong.
Current LLM-based NLQ systems handle two-hop questions reasonably well when the schema is well-annotated. Three-hop questions involving mix effects or incremental calculations are unreliable enough that we would not recommend relying on them in production without a verification step. The verification step usually means an analyst checks the generated SQL before accepting the answer, which reduces but does not eliminate the time savings.
Where NLQ Actually Delivers for Analytics Teams
Having laid out the limitations, it is worth being clear about where NLQ does deliver real value for analytics teams, because there are genuine use cases.
Ad-hoc dimension lookups work well. "What percentage of activations last week came from mobile?" is a query that a stakeholder can answer themselves with a reliable NLQ interface without waiting for an analyst ticket. The question maps to a single-table aggregation with a filter, and the risk of a definition error on a percentage breakdown is low because the stakeholder can sense-check the output against their intuition.
Trend queries with simple time windowing also work well. "Show me weekly DAU over the past 90 days" is structurally simple and maps to a DATE_TRUNC plus a GROUP BY. An analyst who would normally write this query in 2 minutes can still save that 2 minutes if the NLQ interface reliably handles it, and at scale across a team those small time savings add up.
The pattern we see work best is NLQ paired with explicit metric definitions. When the NLQ interface resolves business terms ("activation," "conversion," "DAU") to canonical SQL expressions from a shared semantic layer, the silent-wrong-answer failure mode is substantially reduced. The remaining risk is mainly in multi-table joins and complex filter logic, which are query types that a human double-check is warranted for anyway.
What Teams Should Test Before Deploying NLQ
The demo schema is not a useful evaluation benchmark. When evaluating NLQ tooling, test it against your actual schema. Specifically, test it against your most ambiguous tables: the ones with legacy names, the ones with multiple columns that could plausibly represent the same concept, and the ones that require a specific join path to get correct results.
Run 30 to 50 questions that your analysts receive regularly. For each, have a senior analyst verify the generated SQL independently. Track the rate of syntactic errors (query runs but is wrong), semantic errors (query runs, returns a number, but the number is wrong), and refusals (system says it cannot answer). A good NLQ system for a real analytics environment should have a semantic error rate under 10% on the question types you intend to use it for. On multi-hop attribution questions, be skeptical if the claimed accuracy is above 70% without clear evidence from your specific schema.
We are not saying NLQ is a bad technology. We use it ourselves at Golden Analytics for a specific class of lookups where the schema is well-annotated and the question type is bounded. The boundary between "NLQ works well here" and "NLQ needs verification here" is real and worth mapping before you commit to relying on it.