My Road to Agents

Since early 2023, I’ve been all-in on “using LLMs.” The word “agent” already existed back then, but the question of what the best practices for using LLMs actually are had no real answer yet.

Looking back, I stuck to a few principles that were decidedly non-mainstream at the time — without quite realizing it — and they’ve ended up looking more and more like how today’s agent systems work.

No fine-tuning

First: I refused to fine-tune, or to modify the model in any way.

The most immediate reason, of course, is that fine-tuning is a pain. But at a deeper level, I never really trusted post-training (I mean the domain-specific, “inject this knowledge” kind of RL — not the general-purpose stuff): the model doesn’t behave the way we want, so we prepare some data, design a reward, and keep tweaking a set of parameters we have no real ability to understand, until the benchmark numbers go up.

To me that’s epistemically lazy: we substitute optimization for understanding. What did the model actually learn? What else got changed alongside it? We usually have no idea. RL can certainly work — but I’ve always felt it shouldn’t be overdone, and you definitely shouldn’t stuff every missing capability and every rule into RL.

At the application layer, there’s an even more practical reason: whatever model you painstakingly tune will almost certainly lose to the next generation of base models. What’s actually worth building is a system that inherits model progress for free, rather than one you have to retrain around every model version. Later, in our framework, Code Llama and plain Llama performed essentially the same — if you threw the task at them directly, Code Llama was of course clearly better; but running through our workflow, the two were basically indistinguishable.

No RAG

“RAG” was all the rage back then, too. A reviewer once asked us: “Why didn’t you add RAG?”

My distaste for RAG comes from the same place: the model can’t do a task, and instead of investigating which step of reasoning is actually missing, you grab a few chunks of similar-looking text from a corpus and stuff them into the prompt. To me, that’s taking the same shortcut.

I have no doubt that RAG can make benchmark numbers go up. But coding is not information retrieval. Whether a program is correct depends on the concrete semantics of this program — not on how some other, similar-looking code is usually written.

If a definition or a dependency isn’t in the current context, then sure, you need to go find it. So I built a code-lookup layer on top of ctags: useful feedback comes from the precise code, not from more text that merely looks relevant. A lot of today’s coding agents do essentially the same thing with grep. My system now just runs LSP directly — clangd. If there’s code you don’t understand, you should ask the compiler. Not guess. Not rely on memory. Not go fishing for “similar code.”

No constrained decoding

I always avoided forcing the LLM to output in a required format.

My approach was two steps: first, let the LLM freely summarize and reason in natural language; then, once the reasoning is done, take one extra step — “now condense the discussion above into JSON.” I was always against the “you must respond in the following format” style, which tangles up understanding the problem with satisfying an interface.

Today’s reasoning models are, in a sense, doing something similar: reason internally first, give the answer last. Structured output must never occupy the entire thinking process. It’s not that you don’t use JSON — it’s that you don’t make the model think while nervously holding a JSON structure together.

No temperature = 0

Finally, temperature.

I ran with pretty “aggressive” temperatures the whole time: 0.7, even 1.0. I even put together a set of experiments for my boss to explain why this setting was better — why we couldn’t just set temperature to 0 to “avoid randomness.”

Temperature 0 makes the model give you the same answer every time, but the same answer is not the correct answer. Randomness isn’t just noise — it’s search space. Squeeze the search space down to zero and greedily emit the next token, and what you get back usually isn’t worth much.

Today, mainstream reasoning models no longer let you adjust temperature at all. Sampling strategy has gradually become an internal implementation detail of the model.

Looking back, I did not foresee reasoning models in 2023. My choices came from one simple intuition: an LLM is, by nature, a probabilistic proposer, and it shouldn’t be forced to masquerade as a deterministic program.

I didn’t bury constraints into the weights. I didn’t cram reasoning into JSON. I didn’t treat temperature 0 as a source of reliability. I left the freedom to the model, and I put the determinism into my own workflow — call it prompt engineering, call it a harness, call it loop engineering.

I was never trying to manufacture a reliable LLM. The real theme of all this work was: given that LLMs are exactly this unreliable, how should we use them? In some sense, the LLMs of 2026 are not fundamentally more reliable than the LLMs of 2023.

And that, more or less, is my road to agents.