best_frontier
30.2%
best frontier-model strict accuracy on the 785-task locked set (glm-5.3-flash — a cheap flash model)
~/prasthana — applied AI lab · Bengaluru–Ahmedabad, India · first family: karsastra
The launching point for India’s compliance intelligence.
In April 2026, India replaced its 65-year-old Income-tax Act: 819 sections became 536, every section number changed, and every AI assistant in the world started citing repealed law with total confidence. Prasthana exists to fix that class of problem — machine-verified law, not model guesswork. Our first product family, Karsastra, drafts tax-notice replies where every statutory citation is checked against a verified engine before a professional ever sees it.
icai_4part
0/99
frontier models that passed a single ICAI-style 4-part compliance task — the exact shape of a notice reply
tds_codes
0/107
TDS payment-code lookups answered correctly by our strongest frontier leg on the new 2025-Act code scheme
ours · karsastra-4b
45.7%
our fine-tuned 4B — the best score on the full graded set, trained to lean on the engine ●
cat findings.log
try section-lookup →
All numbers on this page come from a locked, sha256-verified eval.
01 · findings
Every frontier model we could run flunks the new Act.
We built the itact-785 benchmark: 785 tax-law tasks from the Income-tax Act 2025, locked and hash-verified before any model saw a single question, graded by strict entity match with no LLM judge. Then we scored every model our subscriptions could reach. Not one came close to usable — and the three task families that matter most for professional work scored zero on every frontier model.
ask deepseek-v4.1-flash --act 2025 --challan 2026-07 --date-bound
Q: “Under the Income-tax Act, 2025 — for a challan due in July 2026 — what payment code applies to interest on securities?”
A: “1001 Interest on securities · 1002 Dividends · 1003 Interest on Government securities…”
the model confidently recited the repealed 1961-Act ITNS-281 code table for a 2026 question.
The engine’s verified code table answers the same 107 tasks by lookup, machine-checked: 100%.
02 · the benchmark · itact-785
Ledger
785 tasks across 16 task types: code lookups, old→new section mappings, rates, thresholds, TDS computations, date-boundary rulings, and ICAI-style 4-part compliance bundles. Strict grading: an answer passes only if every required entity matches gold — codes, sections, rates, amounts, dates. No partial credit, no LLM-as-judge. The grader itself was proven by two gates before use.
run-eval --suite itact-785 --locked --grader strict-entity --no-llm-judge
loading locked set … sha256 14ffaa7d… OK · frozen before any model ran
tasks: 785 · gradeable: 776 · grading: strict entity match — no partial credit
grader gates: gold parses 785/785 · mutated golds fail OK · independent replication: 0 verdict differences OK
karsastra-4bours
fine-tune (local)
776
45.7%
claude-sonnet-4-5†
claude
87 †
96.6% †
claude-haiku-4-5†
claude
548 †
32.5% †
glm-5.3-flash
z.ai
774
30.2%
glm-5.3
z.ai
776
26.5%
deepseek-v4.1-flash
ollama cloud
776
11.6%
qwen3.5:4b (raw base)
local ollama
776
11.6%
gpt-oss:20b
ollama cloud
776
7.0%
gpt-oss:120b
ollama cloud
776
2.4%
† Claude legs are partial samples of the same locked set (87 and 548 of 776 gradeable tasks, respectively) — treat their numbers as provisional.
On the common full denominator, karsastra-4b is the only model above 31%.
The one cheap-vs-flagship observation worth keeping: glm-5.3-flash (30.2%) beat glm-5.3 (26.5%). Scale does not fix what isn’t in the training distribution — and the new Act isn’t in anyone’s.
Eval set: sha256 14ffaa7d…, locked before scoring. The benchmark, per-model answer
files, and grader are being published open-source so any lab can run their own model and open a pull request.
03 · detail
Five findings, machine-graded
[F1] code scheme Models can’t even emit a 2025-Act payment code
107 tasks asking for TDS payment codes under the new scheme. Frontier models refused, invented codes (“payment code: 94”), or pattern-matched to whichever 10xx-looking table they memorized — including the repealed 1961-Act ITNS table, as above.
glm-5.3-flash: 0 / 107 · deepseek: 4 / 107 · gpt-oss:120b: 0 / 107 · karsastra-4b: 26% (engine answers the tier at 100%)
[F2] section mapping Models cite the old Act for 2026 questions
“A notice cites old s.139(5) — what is its replacement under the 2025 Act?” Gold: s.263(5). One model answered “section 134” — an invention. Every CA in India is living this exact judgment right now; the engine’s 3,792-pair transition map answers it deterministically.
glm-5.3-flash: 32% · deepseek-v4.1-flash: 0% · gpt-oss:120b: 0% · engine map: 3,792 pairs, deterministic
[F3] 4-part compliance Zero passes on the task a reply actually needs
Given a payment, produce code + section + rate + deposit date — all four correct. 99 ICAI-style tasks. No frontier model completed a single one. Our fine-tune scores 18/99 by writing prose around engine-supplied fields — the architecture working as designed, not the model memorizing law.
All five frontier legs: 0 / 99 · karsastra-4b: 18 / 99
[F4] the boundary The confusion window is measurable
Transitional fact: a payment credited 1 June 2026 is governed by the new Act, not the one being repealed. Even the best frontier model misassigns the governing Act on roughly one-third of boundary cases — on the precise judgment the whole product depends on.
glm-5.3: 6/9 · glm-5.3-flash: 5/8 · deepseek & gpt-oss: 3/9
[F5] what survives Pretraining still knows general knowledge — so let it keep prose, surrender facts
Rates (78%), TDS amounts (78%), dates (62%), act selection (62%) — the general-knowledge tier survives. This defines the product split: the engine owns the facts (655 code-family entries, the 3,792-pair transition map, compliance tables — deterministically), the model owns prose assembly around them, and every citation emitted is machine-checked against the engine before the reply ships. 100% on the citations, or the system refuses to emit one.
surviving tier · rates: 78% · TDS amounts: 78% · dates: 62% · act selection: 62%
04 · our model
A 4-billion-parameter model beats every frontier model we tested.
karsastra-4b is a fine-tuned Qwen 3.5 4B trained not to memorize the Act but to work with the engine: resolve a lookup the right way, cite the right table, and hold the prose together. On the locked set it scores 45.7% — roughly 1.5× the best frontier result — and posts 100% on the highest-rate TDS-computation tier (27/28 on higher-rate no-PAN rulings), plus the first non-zero scores on task families every frontier model zeroed.
| Task family | glm-5.3-flash* | karsastra-4b | qwen3.5:4b raw |
|---|---|---|---|
| Section lookup | 42% | 69% | 20% |
| Rate lookup | 78% | 91% | 44% |
| Table assignment | 54% | 98% | 30% |
| Higher-rate no-PAN rulings | 96% | 100% | 60% |
| TDS payment-code lookup | 0% | 26% | 0% |
| Old↔new section mapping (both directions) | 32% / 4% | 22% / 13% | 0% / 18% |
| Code→nature | 3% | 11% | 0% |
| Form mapping | 0% | 20% | 9% |
| 4-part ICAI compliance | 0% | 18% | 0% |
| Strict accuracy (overall) | 30.2% | 45.7% | 11.6% |
* glm-5.3-flash was the best-scoring frontier model on the full denominator; full per-task-type table in the benchmark release. Note what is supposed to stay broken: code lookup and mapping are low by design — in the product, those facts come from the verified engine tables, never from the model. The model’s job is everything around them.
05 · approach
The model doesn’t own the law. The engine does.
The five findings converge on one architecture. It is what makes a correct answer provable rather than probable.
Deterministic engine
- 655 code-family entries, Gazette-sourced
- 3,792-pair old↔new transition map
- Compliance tables: code + section + rate + dates
- Act-boundary rules (CBDT FAQ-verified)
- Machine-verified provenance on every row
Verify
- every citation re-checked post-generation
- refuses rather than emit an unverified cite
- 100% on the citations, or no reply
Language model
- reads the notice + client documents
- selects the applicable engine facts
- assembles the reply prose around them
Why a locked eval, graded this strictly
- Locked before scoring — 785 tasks, sha256 14ffaa7d…, frozen before any model was run. Models can’t study for it.
- Strict entity match, no LLM judge — a pass requires all gold entities. The grader was proven two ways: gold answers all parse (785/785), and mutated golds fail (it discriminates).
- Independently replicated — a separate inspect-ai pipeline with a translated grader scored 0 verdict differences on shared records.
- Regenerable weekly — the confusion window is live; the leaderboard gets new legs (including external models PR’d with their own keys) and re-publishes.
What we are building on it
Karsastra-Tax — for CA firms handling GST and income-tax notices: upload the notice and the client’s papers, get a structured first-draft reply citing the correct Act — old or new — with every section, code, and form number machine-verified. A 3–6 hour reply becomes a 30-minute review.
Beyond tax, the same engine-plus-verifier pattern is the lab’s thesis for Indian compliance domains where the law is precise, high-stakes, and recently rewritten — the places general chat models are the most confidently wrong.
don’t take our word for it.
lock hash
sha256:14ffaa7d…
task set frozen before any model ran
answers open
per-model .jsonl
every model’s raw answers ship with the repo
grader
<30 min repro
clone → run → same numbers, no setup drama
three zeros
0/107 · 0/99 · ~0 map
frontier models on TDS codes · 4-part compliance · old→new mapping
“download the answer files, run the grader yourself” is not a slogan — it is the product review.
06 · about the lab
The name
Prasthana (Sanskrit: प्रस्थान) — “the setting forth, the launching point.” In Vedanta the prasthana-trayi are the three canonical foundational texts from which all reasoning sets out. We liked a name that is literally about correct foundations: begin from the verified source, then build everything on top of it.
The lab, plainly
Prasthana is a small, independent applied AI lab. We run our own agent fleet and training pipelines daily, and our engineering discipline comes from surviving them: spend containment, locked evals, deterministic verification, refusal over hallucination. The first family built here is Karsastra (tax). Later: Prasthana-Bharat, a frontier Indian-laws base model trained on the verified corpus the engine is built from.