Ticket explorer

These are 150 synthetic support tickets from the frozen test split (seed 1337). When you pick a ticket you see four answers: Jev's recorded response from 21 Sep, Jev live, and two live typesafe engines (zero-shot and trained). Jev and typesafe zero-shot see only the criteria text.

Playground

The request body is Jev's POST /v1/systemone. The server exposes the same endpoint, so an existing Jev client can point here unchanged. Question IDs department, urgent and frustration hit the heads trained at boot. Any other ID is answered zero-shot.

npm i @ruvector/typesafe
node node_modules/@ruvector/typesafe/scripts/fetch-models.mjs

import { createTypesafe, choice, noul } from '@ruvector/typesafe';
const ts = createTypesafe({ embedder: { kind: 'onnx',
  modelDir: 'node_modules/@ruvector/typesafe/models',
  manifest: 'node_modules/@ruvector/typesafe/models/manifest.json',
  model: 'bge-small-en-v1.5-int8' } });
// ⚠ createTypesafe() with no options uses the "hash" test double:
//    word-overlap only, answers are near-meaningless.
await ts.train('dept', [{ text: '…', label: 'billing' }, …]);
const r = await ts.decide(state, { dept: choice({ billing: '…', fraud: '…' }) });
r.dept.choice; // typed "billing" | "fraud"

Experiments

Five hands-on exercises. Each one tests a specific claim. Results are computed live on this server.

Fresh benchmark

These are 150 test messages neither system has seen, across three unrelated tasks, with a separate training set. It was written for this lab to avoid the package author's synthetic tickets. Four setups are compared: typesafe with criteria only, typesafe trained on the training split, Jev with criteria only, and Jev with the whole training split passed in the state.

Reference run

Where each setup breaks: accuracy by difficulty tag

Reproduce it live

Browse the items

Cascade: local first, Jev last

Each message goes through a chain of stages, cheapest first. A stage answers only when its confidence clears the threshold you set. Otherwise the message moves to the next, more capable stage. All three local stages run on our own servers, so nothing leaves them until the final Jev step.

Each stage on its own

Tune the cascade

How little Jev do you need?

For each cap on the share sent to Jev, this is the best mix of thresholds found on the 150 test items. The thresholds were chosen on the test data itself, so treat these figures as an upper bound. Real traffic needs thresholds picked on held-out data.

Try a message

Graph + cascade: does a knowledge graph help?

Real routing traffic usually comes with history: which company a complaint is about, which council a report goes to, and how past cases were routed. We added that history to the cascade as an extra signal, in two ways. One is plain counts ("how often did this company's past complaints fall under each product?"). The other is @ruvector/kge, which learns embeddings from the same history as a knowledge graph. Both were tested on real public data.

Each signal on its own

How little Jev do you need?

Walk through real cases

What's next

Method and caveats

Download the results file (all arms, frontiers, sample items).

Evidence

All runs use the package's own frozen fixtures and harness (commit 5356a84), plus a few new experiments. Provenance tags: reproduced = re-run here and matches the README, new = experiment added for this lab, recorded = the author's capture and not re-runnable here.

Tickets (8 departments, 150 test items)

Out-of-scope detection (CLINC150)

This is the package's own release gate (OOS AUROC ≥ 0.85), and abstain passes it: 0.90 with 150 intents after 8-shot training, 0.90 zero-shot, and 0.94–0.97 on 8-intent subsets. What doesn't carry over is the scale. abstain is one slot of a 151-way softmax, and training fits a sharp temperature that is applied to it too, so values sit near 1e-8 and a fixed threshold can't move between questions. A proposed opt-in abstainMode: 'sigmoid' reports the same signal on a fixed 0–1 scale. With it, one threshold tuned on the 150-intent validation split caught 78% of out-of-scope test items at 14% false alarms, and 92–97% at 11–19% on the 8-intent subsets.

Speed and cost

Code-level observations

Where it sits in RuVector

RuVector describes itself as a Rust substrate for "high speed decisions and persistent memory for AI agents". typesafe is its decision layer. These are the integrations that already exist and the ones that look realistic, tagged by maturity.

A reference architecture worth prototyping

Verdict

These are not substitutes. Jev is a foundation model that makes typed decisions from a description, and on accuracy it wins almost everywhere we measured, live. @ruvector/typesafe is a well-engineered, API-compatible toolkit for training small, fast, calibrated classifiers on your own labels. Its case rests on latency, locality and calibration, not accuracy.

Recommendations

Before relying on it in production

1. Re-run both on your own, non-synthetic data. These tickets are templated and come from the package author's generator.
2. Always give choices an other option, and test against your real off-topic traffic. For typesafe, set an abstain threshold empirically.
3. Always pass an ONNX embedder explicitly. Never ship the default hash.
4. Measure on a non-templated dataset. Banking77 and HWU64 fetchers exist in bench/datasets.
5. Tune noul thresholds on a labelled validation slice. Jev's urgency jumps from 58.7% at 0.5 to 91.3% at 0.91.
6. Pin the package version. It is 0.1.x and the API may still change.