One of the more frustrating aspects of working with agents is their tendency to underestimate their own intelligence. Although it’s clear to us that intelligence is perhaps becoming an ability you can summon at will, the models try to substitute ill-defined, underspecified logic exactly where intelligence would shine. You have this incredibly smart model that can use the browser to take pretty complex actions, but it crashes midway because some attribute in a dataclass it defined itself doesn’t match. This throws people into loops where they use that same intelligence to diagnose and fix the issue.
This might also the reason behind the “everyone is building a harness” sentiment: everyone is trying to figure out the boundary between intelligence and logic/code. Iterations on products end up looking like a search for this boundary, and as models absorb the harness itself, the boundary moves again and product/harness builders need to adjust. Could also be why a harness that lags the model’s capability can hurt the metric you care about.
As for why models underestimate themselves in the first place, I think it makes sense. The model at any given instance is the result of patching its own past failures, from a point when its final capability didn’t exist yet. It doesn’t know where its training would end up, so its sense of what it can do is tied to a weaker version of itself. Distillation on the other hand, perhaps goes too far the other way where the model learns to be overconfident in its predictions because of a much stronger teacher. Of course, distillation is still a net positive but produces poorly calibrated models which in turn blurs this intelligence-logic boundary (like an incorrect CoT, correct final answer).