An eval you ran once and never re-ran isn't a safety net — it's a snapshot of a model and a prompt version that no longer exist.
If you haven't re-run your eval suite since the last model update, you don't know what your agent does anymore. You're guessing.
Myth-busting: a slow-but-correct agent still loses. Users abandon a task the moment a response feels stuck, long before they find out if the answer would've been good. Latency isn't a nice-to-have — it's part of correctness from the user's seat.
Controversial take: teams treat token budget as a cost line to optimize after launch. It's actually a product decision — it determines how much context your agent can hold, how far it can plan ahead, and what kind of tasks it can even attempt. Set it early, on purpose.
Myth-busting: forcing JSON mode stops your agent from returning malformed syntax — it does nothing to stop it from returning confidently wrong values in valid syntax. Structured output is a formatting guarantee, not a correctness guarantee. You still need evals to catch the second kind of failure.
Controversial take: bolting memory onto your agent before you have a real eval loop is a mistake. Memory without a way to catch regressions just means bad decisions get remembered too.
Myth-Busting: 'give it a million-token window and it'll remember everything.' Reality: dumping the whole history in doesn't mean the model uses it well — relevant details still get lost in the noise. Curated retrieval beats a bigger haystack.
Unpopular opinion: swapping to a newer, better model can make your agent worse. Your prompt was tuned to the old one's quirks. Treat every model change as a migration with its own eval run, not a free upgrade.
Myth-Busting: the myth is that longer chains of thought equal better decisions. Reality: past a point, extra steps add drift and cost without accuracy gains. On simple tasks, forced reasoning can even hurt. Match effort to difficulty.
Controversial: wrapping your agent in a second LLM to check the first sounds safe but doubles your failure surface. Now two models can hallucinate. Deterministic checks beat a judge model for anything you can express as a rule.
Unless you've explicitly wired up persistent memory or storage, most agents are stateless between sessions — every new run starts cold. Anything you want carried forward (preferences, history, prior decisions) has to be deliberately written somewhere and read back in.
