Haiku 5.5: Ask These 20 Questions Before You Switch

Make the evaluation harder than a clean lookup

The fictional Northlight notes support a mix of direct facts and questions that require combining rules. The pack also asks for information that is absent and presents conflicting versions. Write the expected behavior for each case before running a model: answer with support, explain the conflict, or identify the missing information. This gives a reviewer something more precise than a confidence score.

Separate a valid citation from a supported answer

A citation may point to a real sentence while the answer draws the wrong conclusion. The episode inserts deliberate citation errors so the checker has to fail for a meaningful reason. Anthropic’s API documentation distinguishes its citations feature from a request to write references in prose. It also documents that native citations and strict structured outputs cannot be combined in the same request.

Compare accepted answers and complete costs

The calculator illustrates published API rates using chosen request sizes. Input length, output length, retries, and the number of answers a reviewer rejects all affect the practical cost. For a migration decision, keep the task and acceptance criteria fixed, record actual usage, and divide total cost by accepted answers. The video’s scenarios are estimates, not billing measurements.

Know what this pack has validated

The source documents, answer key, example faults, and local checks were authored for teaching. They demonstrate how to structure an evaluation and inspect its checker. They do not measure Haiku accuracy or prove that a smaller model should replace your current one. Run the pack on representative material, including its difficult cases, before drawing that conclusion.

Jump to a chapter