[Side Project] 'Jev-compatible' had no spec: building jevcompat

What I built

jevcompat is three things:

1. SPEC.md: an unofficial, numbered spec of TypeSafe's System One API (POST /v1/systemone). It has 48 requirements, each with a level (MUST/SHOULD) and the official source it comes from.
2. jevcompat test URL: runs the spec against any server. Each failure comes with the request that caused it and the response bytes that broke the rule.
3. jevcompat proxy URL: sits in front of a non-conforming server, fixes what can be fixed, and refuses what can't instead of passing a wrong answer through.

Why

TypeSafe released Jev on September 15. It takes typed questions (yes/no, choice, score) and returns typed answers with probabilities. The model was closed and the API waitlisted. Within nine days, about a hundred open-source servers called themselves "Jev-compatible".

I read the code of the 26 most-starred ones:

• Only 14 actually serve /v1/systemone.
• confidence meant different things on different servers: the top probability, the top-two margin, or 1 − entropy.
• The choice limit was 26, 50, 64 or 128. The docs promise 255.
• The official SDKs' default, "model": "jev-latest", was handled four different ways.

There was nothing to check the claim against, and the would-be references: TypeSafe's docs, its OpenAPI file and its two SDKs: disagree in eight places.

I checked what already existed first. There were leaderboards (JevBench), calibration benchmarks (sys1bench), semantic grep tools (17) and SQL integrations (25). The empty slot was the API contract itself: the same situation Markdown was in before CommonMark gave dozens of implementations a spec and a test suite.

Results: the eight most-starred servers

Two of eight conform. The most striking finding was jeff: reversing the order of the questions in one request moved the team answer from Billing 0.768 to 0.351, because the questions share one encoder pass.

The hard part: not grading anyone wrong

Writing the checks was quick. What came after took much longer. This tool grades other people's open-source work in public, so the worst bug isn't a missing feature. It's a false positive: a "MUST violation" stamped on a server that did nothing wrong.

So it was verified in three layers.

1. Every check has to be able to fail. The reference server (jevcompat mock) implements the spec exactly and can break any requirement on purpose (47 faults). CI checks three things: every check passes on the clean server, each fault fails its target requirement, and no single fault spreads into unrelated MUSTs. A check that can never fail is no check.

2. Three independent reviews. The first found a critical problem. "Does renaming a question id change the answer?" was judged from the difference between two requests, so correct servers with noisy probabilities failed about 25% of the time. The tests passed only because they used one lucky seed. Now requests are sent several times, means are compared against a limit from the pooled standard deviation, and a difference counts only if a fresh second round shows it in the same direction. The false-positive rate is tested over many seeds. The reviews also found:

• a 502 on the very first request, while the server warmed up, was blamed on the model name;
• float32-rounded values weren't recognised as rounded;
• rate limiting (429) was counted as a server defect.

3. Every real failure checked by hand. Each of the eight servers' failures was compared against the recorded request and response. That found five more bugs in the tool itself:

• a structured score level returned as a string was treated as a violation, although the API reference types legend values as strings;
• only one of the two confidence formulas that reproduce every documented example was accepted;
• a new check never ran, because of the order it was registered in.

All were fixed before publication. The 42 failures in the final reports are all real (REVIEW.md).

Two rules for writing the spec

• One rule sets the levels. If a client written against the official docs or SDKs breaks (an exception, a crash, a wrong answer that looks right), it's a MUST. If the server only behaves differently from the official one, it's a SHOULD.
• When official sources disagree, the robustness rule decides. Accept any request that any official source calls valid. Produce only responses that every official client parses. That one rule settled all eight disagreements.

What it doesn't do

• It doesn't grade accuracy or calibration. It asks "will my client code work?", not "is it smart?".
• It doesn't test TypeSafe's hosted API. The suite sends malformed requests on purpose, and TypeSafe's terms prohibit security testing.