Workshopping an idea that we should think about this like F1.
Conventional wisdom is that winning an F1 championship can be attributed to something like 50% engine, 25% chassis, and 25% driver.
The rough analogy here is model as the engine, harness as the chassis, and prompter as the driver.
A great model can produce valuable outputs independent of its harness but to perform at the frontier, it needs the post trained harness, and to a lesser extent, a truly capable prompter.
It doesn't make sense to think about model performance independent of the harness because they've become indistinguishable.
On harnesses, I vacillate between three beliefs:
- the less harness, the better. Models are the magic
- post training a model and harness is dramatically better and the model providers win
- harnesses have real independent value from the model
I have no idea which is right.