Prompt-and-parse is a prototype. Typed decisions are production.
Open almost any AI feature in production and you will find the same shape: a prompt that begs a large language model to "respond with JSON only," a parser that hopes it listened, and a retry loop for when it did not. It demos beautifully. Then real traffic shows up with edge cases the prompt never imagined, the model gets creative with your schema, and your code is left holding a paragraph where it needed a decision.
We hit that ceiling enough times that we changed the pattern. The decision points in our systems now run through a typed decision layer: small, purpose-built models that take natural language plus application state and return structured judgments, a label, a score, a probability, that ordinary code can test, combine, and log. The tool we are integrating for this is TypeSafe's Jev, and the shift in thinking matters more than the vendor.
What changes in practice
- The output is a type, not a hope. A routing call returns one of the routes you defined, with a confidence score. Your code branches on it the way it branches on any other value. No parsing, no prayer.
- Small models for small decisions. "Is this lead a good fit, and which bucket does it belong in" does not need a frontier model. It needs a fast, cheap judgment that is right often enough and honest about its confidence.
- Probabilities compose. Scores can be weighted, thresholded, and combined with hard rules you control. When the model is unsure, your rules decide, and the uncertainty is visible instead of hidden inside fluent text.
- Every decision is auditable. A typed judgment with a score is something you can log, sample, and argue with. A paragraph of model prose is not.
Where we are applying it
- Lead routing and scoring for a local-service marketplace: which provider gets the job, what gets escalated to a human, what gets a second look.
- Live-chat moderation for a retail client: flag what needs a person, let the ordinary conversation flow.
- Ops guardrails for our own agent systems: cheap checks that run before an automated action is allowed to proceed.
How we roll it out: shadow mode first
None of this goes live on faith. The pattern we use for every decision point:
- Run the typed model in shadow mode alongside the existing behavior. It scores real traffic but changes nothing.
- Compare its judgments against what actually happened and what a human would have done.
- Promote it to a live decision only for the cases where the data says it has earned it, with thresholds and fallbacks set by us.
- Keep the logs. If quality drifts, we see it in the numbers, not in a customer complaint.
This is the unglamorous middle of real AI work. The demo is the easy part. A decision you can type, test, score, and audit is what survives contact with production, and it is the difference between an AI feature and an AI system.