Model evidence benchmarks
Compare Jylus with standard and tuned RAG.
Select a cohort to inspect its complete same-model result. The frozen benchmark and supporting real-data cohort remain separate because their datasets, sample sizes and evaluation contracts are different.
528 / 528passed · 100% observed strict accuracy
1,584completed model evaluations
99.28%Wilson 95% lower bound
98.77%less model input than full context
| Evidence path | Strict accuracy | Evidence recall | Hallucination trials | Avg model input | P95 end-to-end | Cost/query |
|---|
| Full context | 78.79% | 86.74% | 7.58% | 205,129 | 9,352.59 ms | US$0.053305 |
| BM25-only RAG | 76.14% | 90.03% | 8.71% | 8,128 | 6,359.19 ms | US$0.003149 |
| Jylus Context Pack | 100.00% observed | 100.00% observed | 0.00% | 2,532 | 3,404.00 ms | US$0.001160 |
528 unique adversarial questions across four data domains. Full context, BM25-only RAG and Jylus used the same Gemini 3.1 Flash Lite settings and deterministic scorer.
What changed
The model did not improve. The evidence supplied to it improved.
Full context mixed current facts with stale values, conflicts, unrelated records, nested-field collisions and untrusted text. BM25 reduced volume but could omit or mis-rank required evidence. Jylus supplied a compact, proof-bound Context Pack around the requested scope. On this defined benchmark that changed accuracy, latency and cost.
+21.21 pointsstrict accuracy vs full context
97.82%lower measured model cost/query
98.77%less model input than full context
0%scored hallucination trials in 528 Jylus evaluations
528 / 528 passed — 100% observed strict accuracy. The Wilson 95% lower bound is 99.28%. Hallucination-trial interval: 0.00-0.72%. Zero observed failures is not a universal guarantee. Results apply to this defined benchmark, and the evaluation has not been independently reproduced.
Measured economics
One million requests on this benchmark workload
- Full context
- US$53,305
- Jylus Context Pack
- US$1,160
- Measured difference
- US$52,145
Scaled linearly from provider-reported usage of US$0.053305 versus US$0.001160 per query at the frozen standard price. Actual savings vary by model, workload, output length, caching, discounts and usage.
Frozen comparison contract
No tuning between accepted arms
- 528 unique designs across four data domains and 11 adversarial classes
- Zero missing pairs, duplicate cells, execution errors or malformed response hashes
- Raw answers, provider request IDs, timings, hashes and checkpoints retained
- No core, portal, prompt, scorer, dataset, budget or model-setting changes between arms
Public methodology exposes the evaluation contract and limitations required for credibility. Compiler algorithms, ranking and fusion logic, evidence selection, temporal and relationship processing, optimisation and tuning remain proprietary.
Public benchmark comparison
TEMPO, QASPER and FinQA retain their own metric contracts.
These complete public test-set reruns are reported separately from Jylus's frozen operational benchmarks. A retrieval score, an execution score and strict answer accuracy are different measurements and are never substituted for one another.
TEMPO · NATIVE JYLUS PUBLIC V1 APITemporal and state retrieval
40.048%NDCG@10 · 1,730-question live retrieval run
39.745%Recall@10 · same live retrieval run
56.029%TC@10 · latest fully judged 680-question live cohort
NDCG@10 and Recall@10 are all-question means from the latest complete authenticated public-v1 retrieval run. TC@10 is reported from the latest fully judged frozen live temporal cohort. The newer retrieval run made no paid judge calls, so these scopes are kept explicit rather than presented as one fresh judged result.
QASPER RETRIEVAL · 1,335 QUERIESUntouched MTEB task and supplied qrels
| Evidence path | NDCG@10 | Recall@10 | MRR@10 |
|---|
| Jylus native stack | 50.695% | 75.130% | 45.663% |
| Tuned hybrid RAG | 48.561% | 74.226% | 42.908% |
| NVIDIA Blueprint agentic RAG | 54.386% | 76.419% | 50.388% |
Dataset: mteb/QASPER test split, 20,508 corpus IDs, 1,335 query IDs and 2,319 supplied qrels. No query or corpus deduplication and no qrel modification.
FINQA · 1,147 OFFICIAL TEST CASESOfficial execution and program evaluator
| Evidence path | Execution accuracy | Program accuracy |
|---|
| Jylus native stack | 36.443% | 31.473% |
| Tuned hybrid RAG | 35.833% | 31.125% |
| NVIDIA Blueprint agentic RAG | 34.525% | 29.468% |
Every frozen evidence ranking passed through the same deterministic provenance-bound program synthesizer and the official FinQA evaluator. Runtime inputs excluded gold evidence and answers. These controlled local measurements are not public leaderboard placements.
Comparison controls
Same test cases, explicit runtime boundaries
- All 1,335 QASPER queries and all 1,147 FinQA test cases completed
- Zero failed cases in the tuned and NVIDIA FinQA retrieval runs
- FinQA official execution/program scores are the headline; retrieval-only diagnostics are not presented as leaderboard scores
- NVIDIA values come from the tested local Blueprint implementation pinned to commit
cc84e6a, not from an NVIDIA-published result
QASPER timing: Jylus 34.46 ms mean / 48.77 ms P95 with zero model tokens; tuned RAG 423.21 / 758.92 ms with zero model tokens; local NVIDIA Blueprint 5,326.16 / 6,612.28 ms with 6,367,554 planner tokens. FinQA timing: tuned RAG 92.12 / 157.81 ms with zero planner tokens; local NVIDIA Blueprint 4,889.16 / 5,926.78 ms with 3,326,499 planner tokens. Jylus FinQA latency was not captured in the same harness, so no FinQA speed comparison is claimed.
Test conditions
Reproducible workload definition
- Host
- AMD EPYC 4464P, 12 cores / 24 threads
- Memory
- 128 GB DDR5 ECC
- Storage
- 2 x 960 GB NVMe, software RAID 1
- Network
- 1 Gbps host interface
- Event shape
- Structured device telemetry with nested identity, state and measurements
- Representative raw size
- 1,342 bytes per JSON envelope; values vary by sequence and payload
- Load generation
- Isolated server-side generator using a fixed, declared test profile
- Working set
- 7,000 synthetic device identities in a deterministic workload
- Duration
- 30 minutes at the 1M-events/s profile; 1.8 billion published events
- Reliability gate
- Accepted-event accounting, zero-failure checks and verified final zero backlog
Measured result
Latency and reliability gates
Accepted rate1,000,135.7 events/s
Published events1,800,000,000
Concurrent API requests18,659,349
P505.44 ms
P9010.78 ms
P9513.30 ms
P9920.99 ms
Failures0
Redeliveries0 added by the run
Dead letters0 added by the run
Pending queue0 after verified drain
Queryability
Reads remained active during ingest.
The gate ran concurrent API requests while events were being accepted, then required the accepted workload to settle without failures, new redeliveries, new dead letters or remaining backlog. The test verifies continued query service and final zero backlog. It does not claim that every individual event was queried immediately after its own publish.
Context compiler
Reduction is published with correctness gates.
This benchmark measures the deterministic decision-context compiler only. Retrieval and model inference are excluded. Source records are deterministic arbitrary-schema fixtures covering structured JSON records, events, telemetry and document-like payloads across AV, payments, healthcare and industrial scenarios, including conflicting and unverified evidence cases.
Source tokens74,414
Final packet1,892
Reduction97.46%
Compiler P9521.04 ms
Representative payments case: 100 documents, 300 runs, 2,000-token budget. Token counts estimate UTF-8 JSON characters divided by four; model tokenizers vary. The full seven-case suite completed 2,110 runs with zero correctness failures.
✓Expected decision readiness returned✓Expected latest values retained✓Zero irrelevant entities admitted✓Zero secret-marker leakage✓Every retained fact carries proof IDs✓Final packet remains within the 2,000-token budget✓One context identity across shuffled input order
Prior evaluations
Earlier multi-model evidence comparison.
Before the current frozen benchmark, this evaluation used 168 unique questions across seven failure-oriented scenarios and four arbitrary data schemas. Each successful call compared one of three evidence paths without exposing the path label or expected answer.
293/299strict Jylus answers across both models
95.69-99.08%combined Wilson 95% interval
98.81%fewer input tokens than full context
What this means
Jylus did not change the models. It changed the evidence they received.
Instead of asking a model to reason across the entire evidence universe, Jylus supplied a bounded Context Pack containing the relevant current state, historical evidence, relationships and provenance. On this defined workload, average model input fell from 175,289 to 2,093 tokens while strict accuracy increased from 70.81% to 97.99% compared with full context.
Combined successful evaluations
| Evidence path | Strict exact | Passed | Avg input | Hallucination trials |
|---|
| Full context | 70.81% | 211/298 | 175,289 | 7.38% |
| BM25 RAG | 75.92% | 227/299 | 7,515 | 6.35% |
| Jylus Context Pack | 97.99% | 293/299 | 2,093 | 1.34% |
OpenAI GPT-5.4 mini
| Evidence path | Exact | Passed | Avg input | Avg cost |
|---|
| Full context | 40.00% | 52/130 | 154,529 | US$0.116257 |
| BM25 RAG | 48.85% | 64/131 | 7,150 | US$0.005730 |
| Jylus Context Pack | 95.42% | 125/131 | 1,948 | US$0.001839 |
Gemini 3.7 Flash
| Evidence path | Exact | Passed | Avg input | Avg cost |
|---|
| Full context | 94.64% | 159/168 | 191,353 | US$0.145387 |
| BM25 RAG | 97.02% | 163/168 | 7,799 | US$0.007239 |
| Jylus Context Pack | 98%+ | 168/168 | 2,206 | US$0.002399 |
GPT-5.4 mini: 98.4% lower average model input cost while strict accuracy increased from 40.00% to 95.42%.
Gemini 3.7 Flash: 98.35% lower average model input cost while strict accuracy increased from 94.64% to 98%+.
Defined evaluation only. Costs use the published provider-reported input usage and list rates used for this run. The strict gate required all three expected values, no unsupported structured claim, no invented source proof ID and a current proof citation when required. Across 299 successful Jylus trials, evidence recall was 99.78% and structured hallucination-trial incidence was 1.34%. Gemini completed the full 504-call matrix. GPT completed 392 successful calls from the deterministic seeded order before API credit exhaustion; 111 GPT calls were not run and are not counted as failures. These 896 successful evaluations measure the defined workload, not universal accuracy.
Public methodology describes the evaluation contract, measured inputs and outputs, failure rules and limitations. Context Compiler algorithms, ranking and fusion logic, evidence selection, temporal and relationship processing, optimisation and tuning remain proprietary.