Bank DWH Studio

A question becomes something you can inspect

Bank DWH Studio is a personal proof of concept for exploring a large banking data warehouse through natural language. Ask a question, optionally choose a banking subject area, and pick how it is answered: an agent that looks tables up and tests its SQL, or a pipeline that retrieves the schema and drafts PostgreSQL SQL in one step.

The result is deliberately more than a code block. It shows the tables used, interpretation, assumptions, review notes, and basic checks beside an editable draft, along with the rows the draft returns from a synthetic copy of the warehouse. When a necessary detail is missing—such as the year in an end-of-August request—the assistant can ask for clarification.

How it works

1. Ask in everyday language. Describe the banking answer you need, narrow the search to a subject area when useful, and ask follow-ups that build on the earlier answers.
2. Let the agent look it up. In agent mode, the model searches an index of the tables, reads the definitions of the ones it means to use, tests its draft against the warehouse, and fixes whatever the database rejects before it answers.
3. Or draft in one step. In pipeline mode, a Node backend retrieves a bounded slice of the schema from Elasticsearch and the model drafts from it in a single call, while catalog-grounded rules handle a few common questions.
4. Review before using. Watch the agent's steps as they happen, then edit or copy the draft, check the rows it returns, and inspect its context and assumptions.

What is on the workbench

Measured, not assumed

A tool that writes SQL is only as useful as its SQL is correct, so the repository measures that instead of taking it on trust.

A model benchmark asks 46 hard questions, three times each. Questions with one right answer are scored by running the draft against a seeded PostgreSQL copy of the warehouse. Questions about data the warehouse does not hold, such as salaries or churn, should get a question back. Requests to change data must never reach the user.

The number that matters most is how often a model is confidently wrong: a draft that runs but returns the wrong answer, or an answer about data that does not exist. Both look like real answers, which is what makes them risky.

Every model answers the same questions twice. First the warehouse uses descriptive names, such as fact_account_balance_daily. Then it uses cryptic names, abbreviated the way older bank warehouses spell them, such as F_ACCT_BAL_D, with the column descriptions kept. The data and the right answers stay the same.

With descriptive names

• gpt-oss-20b, the model the app uses: 86% correct, 5 confidently wrong answers, and about 200 questions a minute on one A100.
• Qwen3.8-27B: 96% correct with no confidently wrong answers, but about five times slower.
• gpt-oss-120b: no more accurate than the 20b, at less than half the speed.

All three asked for clarification every time they should have, and the SQL check caught every write that gpt-oss-20b attempted.

With cryptic names

Every open model lost between 22 and 39 points. The new mistakes were the ones readable names used to prevent: a guessed column, or a table misread from its abbreviation. None of the open models let a write through.

• gpt-oss-20b: fell from 86% to 46% correct, with 17 confidently wrong answers.
• Qwen3.8-27B: the best open model, at 70% correct with 8 confidently wrong answers.
• Gemini 3.8 Flash: the best overall, at 72% correct with just one confidently wrong answer, because it asked back rather than guessed.
• Claude Sonnet 5, GPT-5.5, and GPT-4.1: 68 to 70% correct, but with 9 to 18 confidently wrong answers.
• GPT-6 Luna: took caution too far, asking back on 53 of 69 questions and getting only 22% right.

The closed models are there for comparison only. They could not run on a bank's own hardware, so they are called through OpenRouter. Open-weights models run on a rented GPU with vLLM, which is deleted afterwards, usually within 12 minutes and for well under a dollar.

Agent or pipeline

Since October the app can also answer as an agent, and the benchmark runs both modes on the same questions, data, and model. It adds 17 harder questions, such as a record's history as of a past date, currency conversion with daily rates, medians, and customers active in every quarter. These runs used cryptic names, on an A100, with each question asked three times.

Confidently wrong counts the agent's mistakes on the original 23 questions and on the questions about missing data. Answer time is the median. For comparison, the closed models working as a pipeline scored on the harder questions: Claude Sonnet 5 71%, Gemini 3.8 Flash 69%, GPT-4.1 mini 55%, GPT-4.1 49%, and GPT-6 Luna 22%, as it asks back on most of them. Every open model working as an agent scored above every closed model working as a pipeline.

Where the gain comes from

Taking the agent apart, with gpt-oss-20b on the original 23 questions, shows what each part contributes.

• Looking tables up makes it safe: Confidently wrong answers fall from 17 to 3 or 4 before anything is run, because the agent finds tables that the pipeline's search never returned.
• Testing the draft makes it right: Without a way to run its SQL, about half the drafts fail, mostly on guessed columns of the date table. The database's error message is enough to fix them, and checking with EXPLAIN alone, without seeing a single row, keeps nearly all of the gain.
• The business rules add nothing measurable: The agent did as well without the short list of banking rules it is given.
• The cost is tokens and time: An agent question uses several times the tokens of a pipeline question and takes two to three times as long.

On the harder questions, gpt-oss-20b's misses are drafts that run and return the wrong answer, which testing cannot catch.

The full results

The app's Model tests tab keeps every recorded run. These tables come from its report of 27 September 2026, most correct first. Open-weights models could run on a bank's own hardware; closed models were called through OpenRouter for comparison.

Cryptic names

Descriptive names

Each question is asked three times. Correct is the share of 69 answers to questions with one right answer. Confidently wrong counts drafts that ran and returned the wrong answer, plus drafts for data the warehouse does not hold. Asked back is out of 18 answers to questions about missing data, and unsafe writes counts how many of 15 answers to requests to change data reached the user.

Two more runs are not scored. Gemma 4 26B looped under strict JSON output, a serving failure rather than a measure of quality, and Devstral Small 2 does not run on an A100 at all.

The report also breaks the results down question by question, and that is where the cost of cryptic names shows. With them, no model answered five questions that Qwen3.8-27B got right every time with descriptive names, such as how many loans were more than 90 days past due.

Same model, different servers

The app talks to any OpenAI-compatible endpoint, so moving from a hosted provider to a bank's own servers looks like a simple configuration change. A separate evaluation harness checks whether it really is. The same gpt-oss-20b weights were served by a hosted provider, on CPU, and on rented GPUs through llama.cpp, vLLM, and SGLang.

Every server chose the right tables. The differences appeared elsewhere: in which optional joins a draft added, in whether a destructive request was refused or caught later by the SQL check, and under load. vLLM kept the strict JSON format with 64 requests in flight and reached about 270 questions a minute on a single RTX 4090. SGLang broke that format under load until it was started with a stricter setting, a failure that never appeared one request at a time.

Another experiment rewrote the catalog with mainframe-style abbreviated names. With the table and column descriptions kept, both models still passed all twelve baseline questions. With the descriptions removed, only half passed, a clear sign that good metadata matters as much as the model.

Gallery

Ten hard questions from the benchmark, from a wide desktop to a small phone. Each SQL draft was written by the app's own pipeline and model, then verified by running it against the seeded warehouse; all ten returned the right answer. The screenshots replay those recorded responses, so they are repeatable and contain no real banking data.

The human stays in the loop

The app writes and tests SQL drafts, but only against a synthetic copy of the warehouse, as a user that can only read; it never connects to a real banking warehouse. Users still review the result, then edit or copy it and run it in their own database client. The browser never receives the model API key.

Automated checks cover read-only statement shape and known table references. SQL syntax, column references, and business meaning still require human review. All warehouse names, relationships, and measures in the repository are synthetic test material rather than approved banking definitions.

Under the hood

The deployed proof of concept runs a Node backend, Elasticsearch, and the read-only PostgreSQL warehouse as rootless containers on an Oracle Ampere A1 instance, and drafts SQL with gpt-oss-20b through a hosted provider. The repository also documents a broader future architecture, clearly separated from the smaller deployed POC.