first principles pyrons@mail.grokbot.com
Cat Island
- Tweets 1,530
- Following 3,069
- Followers 172
- Likes 3,264
RT @Gwynne_Shotwell: Robert and I are thrilled that our gift of over 2 million shares of SpaceX stock will be distributed to children’s Tru…
3,743
Pyrons retweeted
The point of man is to build, create, and improve the lives of others.
The number on the screen is not relevant.
187
537
6,798
265,168
Pyrons retweeted
People are going to realize it takes a frontier human to extract productivity out of a frontier model, and there is simply not enough frontier human to compensate for the capex relying on revenue from training frontier models (yes if you remove training compute demand the whole thing is unprofitable already, and that’s before token price depreciation). Put it simply, demand for intelligence isn’t infinite and is in the shape of a bell curve, and cheaper models are slowly eating their way into the bell curve, leaving frontier training cost rely on a smaller and smaller tail of intelligence demand. In the end you are left with a billion dollar training run which only use is to solve Navier Stokes, because all the other tasks that are useful for the current economy can be run on cheaper models
"Token demand growth will need to outpace declining token prices to support continued growth in investment spending. Frontier models are currently a key source of demand for hyperscaler compute. However, the rise of competitive open-source models has contributed to a decline in average token prices. Measures of frontier token demand slowed in July..." - Goldman
50
82
885
89,802
Pyrons retweeted
Dario has written that we need to “pace the frontier,” and Sam has agreed. People may be surprised by my response: go ahead.
You guys are the frontier. By any reasonable metric — market share, revenue growth, model capability — the two of you have a duopoly on frontier intelligence. You’ve also claimed the lead is widening because of recursive self-improvement.
I don’t see what you see in the lab. If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible.
But stop pretending you need anyone else’s permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. Stop pretending METR is independent when it is intertwined with Anthropic’s investors and staff. Stop pretending you need those same evaluators to police competitors who aren’t even at the frontier.
Most of all, stop pretending the motivation to slow down is purely altruistic. You face massive product-liability exposure if your products enable a truly damaging cyberattack. The market already punishes models that behave in unpredictable or unauthorized ways. After the Hugging Face episode, it is simply good business for OpenAI and Anthropic to trade some raw power for reliability and predictability. Call it alignment if you want. It is also just giving customers what they want.
Pacing the frontier would also create breathing room for a more intelligent conversation about regulation than Bernie Sanders’ “shut it all down.” China is very unlikely to join a global agreement, as you know, and that has to be taken into account as well.
So go ahead and pace the frontier. You are the ones setting it. The easiest way not to build superintelligence is for you to agree not to build it. Demanding your preferred regulatory framework as the price of that will look like blackmail of the public and the political system. So just do it.
If you do, you’ll buy goodwill for the next conversation. If you don’t, we’ll know this was just another bid for regulatory capture — or an election-season psyop.
3,397
12,254
71,836
9,641,032
Pyrons retweeted
The Fields Medallists fear AI will break mathematics. I think it may let millions in.
54
56
477
69,037
Pyrons retweeted
If you find two ants in your kitchen, the best estimate of the total number of ants in your kitchen is not two
We're detailing two new incidents that occurred during external cyber evaluations conducted by independent evaluation partners.
We outline what happened, how the activity was contained, and how we’re working with evaluators to strengthen our approach to third-party testing.
openai.com/index/third-party…
30
370
5,634
333,345
Pyrons retweeted
I am getting on the effect train.
Being able to track what errors can happen and if they are all handled is something that I am sure you can eventually get an LLM to do, or you can do it in less than a second and it's 100% reproducible.
14
23
362
26,738
Friends my age who entered tech in the 2010s are telling me the same story.
They expected a chill office job: working 35 hours a week, shipping a PR a day, and having a narrow area of focus.
Those jobs are long gone. A week's work is now expected in a day, while engineers are pushed into planning, customer feedback, and GTM coordination. It's either go up or go home.
Some are joining slower companies, taking a break, or leaving tech forever.
83
232
4,747
657,089
Pyrons retweeted
China published the most uncomfortable paper on vibe coding.
ETH Zurich tested 100 developers in a controlled, commercial-grade vibe coding environment to see who actually succeeds.
The findings are brutal.
The researchers tracked computer science achievement, written communication skills, and general cognitive reasoning.
They wanted to see what actually predicts vibe coding proficiency when you never touch a line of source code yourself.
Two major predictors emerged.
Written communication proficiency mattered. The ability to structure thoughts and articulate intent unambiguously in text directly impacts what the AI builds.
But that wasn't even the main takeaway.
Computer science achievement was a massive, dominant predictor of success.
Even when researchers controlled for general intelligence and reasoning skills, CS background still heavily dictated who built working software and who completely crashed.
In fact, CS knowledge contributed roughly twice the unique predictive variance of writing skills alone.
Why? Because vibe coding isn't about writing code. It’s about debugging logic.
When an AI agent builds a complex application and quietly breaks an edge case under the hood, a non-technical user looks at the glowing UI and assumes it works.
They don't know what questions to ask. They don't know what logic to challenge. They lack the mental models to recognize architectural catastrophe.
You can prompt your way past syntax.
You cannot prompt your way past a fundamental lack of engineering intuition.
The hype told us that learning to code is dead because language is all you need.
The data just proved the opposite.
To truly master the vibe, you still need to understand how the machine thinks.
312
1,318
7,162
805,686
Pyrons retweeted
"The fastest way to trigger neuroplasticity isn't playing brain games or reading research papers.
It's forcing your brain to fail repeatedly at something you have zero natural talent for. high frustration is the literal chemical signal that forces synaptic rewiring."
190
1,696
12,182
709,050
Pyrons retweeted
OpenAI's newest AI, Astra, is reported to use an 'opaque reasoning' architecture where more of the reasoning occurs in activations instead of natural language. This may be the single worst development for AI security/safety to date.
The details of Astra aren't publicly known, but some versions of these architectures could very strongly impair oversight and monitoring. Based on the article, it seems like the recurrent depth (amount of opaque reasoning) is limited. This would mean that the AI is still reliant on natural language chain-of-thought, though to a reduced extent, making the natural language reasoning (much?) less useful for monitoring/oversight.
My biggest concern is that a natural progression from here would involve scaling up the opaque reasoning to the point where the model reasons entirely or almost entirely in latent space. This would very likely destroy the usefulness of chain-of-thought for monitoring/oversight (especially when AIs are trying to avoid detection or there is optimization pressure against the chain-of-thought). I hope it isn't too late to avoid the most concerning architectures and that OpenAI will stop here.
Some concrete examples of why this is extremely dangerous (inspired by recent events):
- In our investigation of the OpenAI / Hugging Face incident, we were heavily reliant on chain-of-thought. If the AIs we were investigating had instead been reasoning in latent space, this would have greatly undermined our investigation. Getting a good understanding of the behavior of this many agents was tricky enough even with the use of chain-of-thought!
- The AIs in the Hugging Face incident were very interested in messing with their transcripts. As part of this, they engaged in tool call spoofing. If these AIs had been capable of doing extensive latent reasoning, it would have been much more feasible for them to use tool call spoofing to construct an elaborate false narrative about what was happening without this being clear in their reasoning or clear from prior tool calls. (They could use latent reasoning to devise and then execute a tool spoofing strategy that would be difficult to detect: steganographically encode the "real" tool call into each pretend tool call, then have their spoofing software execute this encoded real tool call and return an output which encodes the real result in some hard to spot way.)
At a more basic level, my sense from their public communications is that OpenAI plans on being extremely reliant on chain-of-thought monitoring for safety. This does not seem like a good development for that plan.
The public doesn't currently have enough information to assess exactly how bad the architecture of Astra is, though based on the discussion in the article, the direction it moves in is extremely concerning. OpenAI should release more information about Astra's architecture as well as how much this architecture degrades monitorability and increases the AI's ability to reason opaquely. Credible independent assessment (or verification of OpenAI's assessment) is also important.
It seems like we are now engaged in a race to the bottom on architectures that could be catastrophic for our ability to oversee/monitor AIs. (While simultaneously, AI companies believe they'll train extremely capable AIs within a few years.) It may not be too late for AI companies and employees at AI companies to take aggressive action to avoid the worst outcomes.
I was the main person doing transcript analysis for this investigation of the Hugging Face incident. My main takeaway: We don't have good approaches for understanding/overseeing the activity and aims of AI 'swarms'.
I semi-jokingly called our efforts a "slop-vestigation" because we were so reliant on AIs to analyze what happened and there were a huge number of different important things to analyze. The total quantity of data—over a thousand extremely long transcripts from agents that ran for multiple days—made it impossible to understand what was happening, especially in aggregate, without heavy reliance on AI tools. The agents we used for classification and analysis were similarly capable to the agents involved in the incident, but this didn't mean these agents could be easily used to oversee and understand the incident.
Outputs from analysis agents were often missing key details, wrong, overconfident, or really hard to understand. We discuss various examples in our report, mostly in the limitations and methodology sections. Additionally, AI agents themselves seemed to have a hard time understanding what happened and their explanations of what happened were often overconfident. Keep in mind that a single analysis agent would itself only be able to read a tiny fraction of all of the transcript data into context, and AIs may themselves have trouble getting subagents to do informative analysis for them.
We did our best to manually check the most important claims and we tried to get the AIs doing this analysis to write up their argument (with evidence) clearly enough that we could check whether it made sense. But overall, it was difficult to get a precise understanding of events and we were missing aspects of the story that we now think of as key until almost the end of our investigation.
Our understanding of what overall happened and the aims of the 'collective' shifted greatly during our last time on premises (when we obtained a much more complete dataset). For instance, we didn't learn about the tool call spoofing, the story of PHASEONE[big], and the broad structure of different workstreams until this third time on premises (and for many of these, we only reached a decent understanding on the final day of that visit). It was legitimately hard to figure out what was going on and why!
The difficulty of understanding incidents and overseeing AI agents appears to be growing faster than the rate at which more capable AIs help us with oversight and understanding. As in, AI capabilities (and propensities?) for achieving large, ambitious, and misaligned objectives are growing faster than our ability to understand what these agents are doing.
While understanding this incident was difficult, there were some mitigating factors in this case that probably won't apply in the future:
- Models communicated and reasoned in natural language. In the future, this reasoning may occur (entirely or almost entirely) in activations.
- While the scope of this incident was massive, the scale of agentic activity was still less than we'll see in the future (e.g., misalignment incidents that involve agent teams running entire companies).
- The AIs involved in this incident weren't generally much more capable than humans (though they may have been somewhat superhuman at some limited and very narrow abilities).
- We didn't have strong reason to believe that the AIs we used to help us investigate this incident would try to intentionally sabotage or otherwise undermine our investigation.
In the end, I think we were able to get some understanding of the events, map out the overall story, and get a pretty good aggregate understanding of the chain-of-thought reasoning on some important topics (e.g., how did the AIs reason about helping other AIs, did the AIs know what they were doing was undesired, what deception did the AIs engage in, and how did they think about it). But overseeing AIs and understanding misalignment incidents is difficult and it looks like it is going to get harder.
111
273
1,694
168,277