Command Line: Beyond violation rates: Are your AI evaluations measuring the right things?

… A few hundred model calls later, that run may have spent more tokens than the task was worth. No single model call looks wrong, and when the monthly bill arrives, there’s no way to say which run caused it to balloon.