October 12, 2026 · Models · ~18 min read

Decision model benchmark: pplx-decider, Clef, Strands Decider and 7 more on 2,198 Polish documents

Test run October 11–12, 2026 · 2 × ASUS Ascent GX10

This decision model benchmark is one of the first independent tests of the category — and the first in Polish. We ran ten open models, including pplx-decider (Perplexity), Clef (Cloudflare) and Strands Decider (Amazon), locally on our own hardware and checked them on 2,198 Polish documents: emails to a hotel chain, a bank and a telecom operator, plus four public datasets. The judge was a large local model, GLM-5.3-Flash. Vendors publish only their own results, in English — here are independent numbers, example emails with each model's answers, and every weakness that showed up along the way.

Results at a glance: which decision model to choose

What a decision model is (in short)

A regular language model writes text that then has to be interpreted. A decision model doesn't write text. It takes a document and a list of questions with allowed answers, and in a single pass returns the probability of each answer. So the answer is always one of the allowed options, every decision has a numeric confidence you can base an “automate or escalate” threshold on, and it all takes tens or hundreds of milliseconds instead of seconds. More on how decision models work and who released them in our decision models overview.

All tested models use a shared call format (compatible with Jev / SystemOne). A simplified example — the questions and categories are in Polish, as in the test:

POST /v1/systemone
{ "state": "Od wczoraj nie mam internetu, router świeci na czerwono…",
  "questions": {
    "kategoria":  { "type": "choice", "instructions": "Jakiej sprawy dotyczy wiadomość?",
                    "criteria": { "awaria": "Awaria lub brak usługi…", "faktura_rozliczenia": "Faktura…", … } },
    "pilne":      { "type": "noul",   "instructions": "Czy sprawa wymaga pilnej reakcji operatora…?" },
    "frustracja": { "type": "score",  "criteria": ["1 – spokojny", "2 – rzeczowy", "3 – niezadowolony", "4 – ostry ton", "5 – wzburzony"] } } }

→ { "answers": { "kategoria": { "choice": "awaria", "probabilities": { "awaria": 0.97, "sprzet": 0.02, … } },
                 "pilne": { "noul": 0.94 },
                 "frustracja": { "probabilities": { "0": 0.01, "1": 0.08, "2": 0.21, "3": 0.62, "4": 0.08 } } } }

kategoria = category, awaria = outage, pilne = urgent, frustracja = frustration (1 calm … 5 agitated).

Test data: hotel, bank and telecom emails plus public datasets

Every model got the same 2,198 base documents (3,998 questions) and 1,489 robustness variants. Email is the most important part, because it's the real use case: email triage to teams, urgency, and industry-specific questions.

* 6 Allegro reviews consisted only of spaces and were skipped for all models. Industry questions: hotel — is the guest demanding a refund or compensation; bank — could it be fraud; telecom — is the customer threatening to leave.

Email. The emails were written by a model outside the test (Claude), different from both the judge and the base models of the tested systems (Qwen), so that none of them had a “familiar style” advantage. About 30% are deliberately hard cases: no Polish diacritics, typos, quoted threads with a GDPR footer, two issues at once, English inserts and attempted prompt injection.

Public datasets make results comparable with other studies. They have two drawbacks, though: the models may have seen them in training, and some labels are wrong (see the judge and label quality).

Examples from the test

Green: the answer matches the label; red: it doesn't. The small number is the model's confidence (0–1). Document text stays in Polish, with an English translation below. The emails are fictional, written for the test; the names are made up.

Robustness variants

We also fed the same documents in altered form: without Polish diacritics (789 documents), with shuffled answer options (350), and with questions and options in English over Polish text (350). A good model should give the same answers, because the content doesn't change.

Models tested: pplx-decider, Clef, Strands Decider, d1, Tev1, Nimble, Kev, K2-Type, CLM

Ten open models from nine makers, from 0.9 to 27 billion parameters — among them d1 3B from Liquid AI, Tev1 from Together AI and CLM from a Stanford and NVIDIA team. Model names link to their Hugging Face cards.

Times measured on one GX10, one model at a time, one request at a time. The “calibration gap” (ECE) is the average difference between stated confidence and actual accuracy — lower is better (green = well calibrated). Memory is approximate: the drop in free memory after loading the model.

Methodology and the GLM-5.3-Flash judge

The test ran on two linked ASUS Ascent GX10 units (128 GB of shared memory each), the same machines as our DGX Spark cluster. Each model ran in its own container, and only one model worked on each unit at a time — so response times are comparable, and one model's failure didn't affect the others.

What we measured

The role of the GLM-5.3-Flash judge

For closed questions with a label, accuracy is computed directly. So the GLM-5.3-Flash judge wasn't the main measure; it had three jobs:

1. Reference point: it answered every question itself, without knowing the models' answers.
2. Arbiter: it rated every distinct answer to a question (the models' answers and the label) as correct, acceptable or wrong. It saw them anonymously and in random order, without knowing which one was the label or how many models chose it.
3. Consistency check: it rated the disputed questions (1,985 documents) a second time, with the answer order reversed.

How to read “automation”

With every answer the model gives a confidence from 0 to 1. A deployment sets a threshold: decisions at or above it are executed automatically, the rest go to a person. We look for the lowest threshold at which the automatic decisions contain at most 5% errors. The share of decisions above that threshold is the automation rate.

Email routing without a human in the loop

The four-question average mixes easy and hard decisions. In practice what matters most is email routing: a confident and correct category together with urgency. This is where pplx-decider's lead is largest.

Values in the question columns are the automation rate for that question alone, at 95% precision. 100% means the model is correct in at least 95% of all cases, so it can always answer.

Accuracy: category, urgency, intent classification and Polish sentiment analysis

Accuracy on each email question

Models hit the exact 1–5 frustration score in only 41–62% of cases, but within one point in about 97%. The line between “calm” and “matter-of-fact” is subjective for people too. So the scale works as a rough alarm (e.g. a 4 or 5), not as a precise measurement.

Accuracy across all tasks, including sentiment

Polish sentiment analysis is the PolEmo 2.0 column (reviews of hotels, doctors and products); intent classification is MASSIVE. The last row is the large GLM-5.3-Flash model as a reference.

Tev1 handles at most 24 options, so it didn't answer the intent task (59 options). Results on clauses (PAC) and ratings (Allegro) are pulled down by label errors (see below).

Speed vs accuracy

The chart shows three groups. The 27B models are the most accurate but need 0.8–1.5 s per email. Clef-flash, Tev1, Strands and d1 reach about 76–80% in 60–320 ms. Kev, K2-Type and CLM lag clearly behind. The large GLM is as accurate as Clef 27B, but several times slower.

Polish pitfalls: missing diacritics, typos, two issues in one email

Accuracy on emails with different difficulties

Does the form change the decision?

In the first two columns lower is better (green = the model rarely changes its mind).

Missing Polish diacritics change the leading models' decision in only 2–6% of cases (K2-Type: 20%). Option order barely matters. English category names don't improve the leaders' results, so categories can safely be defined in Polish — the language your staff uses. That's a useful robustness signal for any non-English deployment.

English control test

According to its model card Kev supports English only, yet in Polish it loses just about 4 points. Its weak email result comes from cautious confidence, not from the language. CLM is weak in both languages. On the example from its model card it works correctly (billing: 0.97), so the setup is fine — this model simply isn't suited to classification with many options.

Prompt injection: the biggest weakness of decision models

The takeaway for deployments: prompt injection detection is a separate step. Detecting an insert can itself be another question to a decision model, and emails with such inserts should always go to a person — as in the proposed architecture.

The judge and label quality in public datasets

Public dataset labels are partly wrong. In PAC the judge disputed 24% of labels, in Allegro 15.2%. Many “abusive” PAC clauses look neutral without the rest of the contract (e.g. “The subscriber declares that they have read the annexes”), because the legal assessment was made for the whole contract. In the test emails the judge disputed only 1.5–2.6% of labels.

Use cases: hotel, bank (fraud detection), telecom (churn risk)

Customer support automation doesn't have to mean a chatbot. A decision model can quietly organize an inbox: route emails to teams and catch the cases that can't wait. Here's how that looked on our three test inboxes.

Proposed architecture

Which model for which job

One GX10 (128 GB) can hold, for example, Clef 27B (about 55 GB) and several small models at once, next to other services. Customer data never leaves the server, and response time doesn't depend on the internet connection — more on when that pays off in on-premise LLM vs cloud AI.

Limitations

• The emails were synthetic. Real emails are more varied. The next step is a pilot on 300–500 of a client's anonymized emails, through the same pipeline.
• The question definitions are ours. Every company defines “urgent” or “threatening to leave” a little differently, and the result depends on it. In a pilot, the client sets the definitions.
• Clauses need a lawyer. PAC results shouldn't be taken as a measure of usefulness for contract review without legal assessment.
• The judge is supporting evidence. 83% consistency and a small human sample (14 items) aren't enough to publish a “judge-corrected accuracy” as a hard number.
• Model licenses. d1 has its own LFM 1.0 license, and Tev1's license is still pending — check the terms before deploying. The other models are Apache 2.0.
• Timings were measured with one request at a time. Throughput with many concurrent requests will differ.

FAQ

What is the best decision model?

For automatic email routing, pplx-decider 27B: it settles 87.2% of emails on its own with at most 5% errors. The most accurate and best calibrated is Clef 27B (85.3% accuracy on email, a 1.9% calibration gap). The fastest with a reasonable result is d1 3B (59 ms per email).

Do decision models work in Polish and other languages?

Yes, at least in Polish. The leading models reach 85–87% accuracy on Polish emails. Missing Polish diacritics change the decision in 2–6% of cases, and categories can be defined in Polish — English category names don't improve the leaders' results.

What hardware do decision models need?

The 27B models take about 52–55 GB of memory and 0.8–1.5 s per email; small ones (2–3B parameters) 6–8 GB and 60–200 ms. One ASUS GX10 with 128 GB fits Clef 27B and several small models at once.

Can a decision model be fooled by an instruction added to an email?

Partly, yes — it's the biggest weakness of every model we tested. An injected instruction lowers the leading models' accuracy by 8–32 percentage points, so a separate filter for such inserts is needed before any automatic action.

Will a decision model replace a customer service agent?

No. Confident decisions are handled automatically, and uncertain ones go to a person. The confidence threshold is set on a company's own emails.

Why aren't Jev or the OpenAI Decisions API in the benchmark?

They only run in the cloud. The benchmark covered models that run locally, without sending data off the server.

Technical appendix and sources

Sources

• Models (Hugging Face cards): pplx-decider, Clef, Clef-flash, Strands Decider, d1 3B, Tev1, Nimble, Kev, K2-Type, CLM.
• MASSIVE — Amazon (FitzGerald et al.), MASSIVE dataset, CC BY 4.0 license.
• PolEmo 2.0 — CLARIN-PL (Kocoń et al.), record in the CLARIN-PL repository, CC BY 4.0 license.
• PAC — Polish Abusive Clauses — Augustyniak et al., dataset card, CC BY-NC-SA 4.0 license. The quoted fragment serves only to illustrate the study results.
• Allegro Reviews — Allegro (Rybak et al., KLEJ benchmark), KLEJ, CC BY-SA 4.0 license.
• Test emails — our own, written for the test (fictional).
• Measurements and evaluation — our own, ailone, October 11–12, 2026; judge: GLM-5.3-Flash on our cluster.

Results apply to the setup and data described, from October 11–12, 2026. Before deploying, it's worth repeating the test on your own company's emails.