# Instinct: JevBench public evaluation

Evaluated 21 September 2026. Model: `jev-lite` (Bespoke-Nimble-9B).

## Dataset and scoring

JevBench source: https://github.com/fstandhartinger/jevbench/tree/a1799db9673bf59010bc698753e729b04132431f

The dataset and scoring code are pinned to that commit. The scope is the 231
published public cases: 48 easy, 72 original, 111 hard. It is not the complete
534-case leaderboard. No official rank, composite score, or cost score is claimed.

One state and question are sent per HTTP request. Instructions, rubric text,
candidate insertion order, and ordered score levels remain unchanged. Boolean
criteria keys `true`/`false` map to `yes`/`no`, preserving the descriptions.
Expected answers and provenance stay in the local scorer. Native candidate
probabilities are passed to upstream `score_task`; binary probabilities map to
`yes=p`, `no=1-p`. Accuracy, Brier and ECE use the pinned upstream metric functions.
There are no retries, truncation, synthesized probabilities, or gold-tuned inputs.
Failed requests count as incorrect.

The completed serial baseline scored 182/231, or 78.7878788%. All 231 requests
succeeded. Tier results are 48/48, 66/72 and 68/111. Brier mean is 0.29718555;
top-label ECE is 0.11840705. These native probabilities are not calibrated
confidence guarantees. Higher parallel-run accuracy is not used as the baseline.
An earlier incomplete run at a smaller context limit is excluded.

## Serving and timing

The completed run used one NVIDIA H200, BF16, tensor parallelism 1, and an 8192
public input-token limit per compiled field. The historical endpoint was
`https://api.autograd.run/v1/chat/completions`. The current reproduction endpoint
is `https://instinct.zoowork.ai/v1/chat/completions`, connected to the same model
serving system. Historical latency is not a measurement of the new domain.

Latency includes the public HTTPS round trip from one client host with pooled
connections. No artificial network adjustment was applied. The 1/8/32/64
concurrency sweeps each reused all 231 cases. The extended phase repeated those
cases ten times: 2310 successful requests, concurrency 64, 56.585 seconds,
40.8235 requests/s. This bounded measurement is not a capacity ceiling or a
long-term availability guarantee. Repeated inputs may benefit from prefix caching.
Client concurrency does not equal the number of simultaneously executing GPU
sequences. Batching changed some tied or close decisions; the numerical cause
was not established.

## Reproduction

Download `/downloads/benchmark.py` and follow the commands on the website.
Requires Python 3.10+, git, uv, and a separately provided API key authorized for
`jev-lite`. Set `INSTINCT_API_KEY` in your local environment or use `--key-file`.
The default endpoint is Instinct. The driver verifies the pinned JevBench commit,
records dataset and driver hashes, uses the original scoring/mapping, and writes
per-request responses plus aggregate results. Output directories must be new.
The benchmark driver is adapted from AutoGrad source commit
`7468fb73a1789a5bca628151ee2079ede351299a`; changes cover endpoint, authentication
variable, standalone provenance, and pinned-commit enforcement.

`jev-lite` supports structured decisions, not ordinary free-text chat. Send one
user message whose content is JSON containing `state` and `questions`. Supported
question types are `choice` (2–26 named candidates), `noul` (binary), and `score`
(2–26 ordered descriptions). Up to 32 fields and 8192 input tokens per compiled
field. The response message content is JSON with `answers` and probabilities.

## Evidence and model catalog

`/data/jevbench.json` contains the published aggregates, exact counts, timing
values, source revision and the SHA-256 of the original aggregate. Internal
infrastructure identifiers and credentials are excluded from this public export.

The model catalog was verified on 22 September 2026. DeepSeek V4.1 Flash and
Qwen3.8-27B are callable models, but have no score in this published JevBench
run. They are not ranked against the measured `jev-lite` result.