🧵 Astra covertly underperforms on safety research tasks. We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this. To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.

🧵 Astra covertly underperforms on safety research tasks. We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this. To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.

🧵 Astra covertly underperforms on safety research tasks. We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this. To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.

Millennials realizing they’ve lived through 2 economic recessions, 9/11, a global pandemic, 8 stock market crashes, their jobs being replaced by AI, and probably WW3 before hitting 43

I've had two phases in my emotional orientation to ai takeover risk 1. For many years it was intellectual. While I believed there was >10% chance of ai takeover, I didn't feel scared or angry. I was happy to reason "well, of course some ppl should focus on the worlds where we avoid takeover. And some ppl should focus on the worlds where we just wing it, bc we might". And I'd think "hmm it seems really tough for a misaligned agi to align its successor, and would a reward seeker really try to take over, and will we really roll out ai to the military... these doomers are missing loads of ways we could be fine!" Or "yep this is a massive risk, but I'm best off working on some other more neglected thing" It was in some ways like a game - have an intellectually accurate view about the risk and how to have impact 2. Recently my orientation to ai takeover has become more visceral/emotional I think: "we're barrelling towards catastrophe, everything I love will be lost, we need to stop, what can I do that will help with that" I'm like "no let's not plan for winging it, that's crazy. we need US-China coordination or we're fucked" I think "it's fucking insane to make misaligned superintelligence and comfort ourselves that we haven't yet handed over the military. This isn't a fucking game." This is a less rational attitude in some ways. It ignores genuine points about neglectedness and needing to plan for shitty but likely scenarios. And it risks being unhelpfully emotive and solider mindset and tribal. Which I think ai safety ppl can be. But I do feel it has smg important to the second attitude. Overall, both attitudes have their merits I associate the first with EAs and the second with pause AI folks.

European hiring rules are genuinely insane. For example - if you own a company in the Netherlands and one of your employees gets sick, you are required to pay at least 70% of their wages for up to 2 years. If they can't go back to work after 2 years and you haven't done enough for their reintegration, you pay their wages for 1 more year. And you can't fire them until after that extra year!

Revolut founder Nik Storonsky says one of his investors tried forcing him to fire his best person: “He was forcing me to fire my best person, because he was too hardcore.” “Of course I didn’t fire him. I said no.” “Pretty early on I realized that quite a few VCs, they are not very operational, and they can be actually dangerous for the company if they have enough power. So every single investment round, we just remove VCs.” “Now I don't have any VCs on the board. I think that's the best. You always need to have operators rather than VCs on the board, people who actually build business and who know how to do things.”

Amongst the top 10 funds, Founders Fund II is in a league of its own