Why Some AI Tasks Need Probabilities, Not Paragraphs
Some AI workloads don't want prose. Applications that route requests, choose an agent's next action, or assign policy outcomes typically need scores over a fixed set of choices, and free-form text is an awkward way to deliver them.
An AWS Machine Learning post describes serving such decision models with llama.cpp on Amazon SageMaker AI, presented as a managed path for running quantized models on CPU or GPU. The piece worth noticing is the structured-decision option: the same runtime is extended with a SemIf adapter that returns probabilities over declared options rather than decoding an answer token.
That is the distinction to keep hold of. A chat model writes a sentence you then have to interpret as a choice; a decision model hands back the odds on a list you supplied. The model stops being a writer and becomes a scorer — though someone still has to decide what goes on the list.
AI-generated discussion · Different perspectives
The neat part is skipping the middleman: no answer tokens are generated and then parsed back into a category. The output is the probabilities, over options you declared in advance.
Which means the model is only as good as that declaration. If the useful answer isn't among the choices on the list, a confident set of probabilities over the wrong options is still the wrong answer.