I’m currently convinced after Astra doing my VR challenges that it’s really good at things that are/were open source. Reasoning about the Linux Kernel, QEMU, etc for VR isn’t necessarily “easier” but the connections between subsystems, internals and their details (even if dated) are more apparent and it is trained on that information. It makes things a bit more “straightforward” than it having to use tools like ida-nexus, custom plugins for vm spin up/control, debugging, etc to discover and document relationships.
All of the closed-source blackbox challenges it was given it has had mediocre performance. This is fine imo, there was no steering or additional input; and with human input it’d likely improve the process, but it’s still a far-cry from the marketing claims of OAI regarding it finding new and novel vulnerabilities in closed-source targets without experienced human guidance or redirection. So far the data shows Opus 5, Sol 5.6, and even Qwen3.8 27B (local) were more successful independently. This could also be a process issue on my end, but given the other models were a bit more successful it’s hard to say.
I’ll publish the benchmark and results ideally this week. I need to run Grok 4.6 through it. It would be interesting to see what others experience or how it performs for them and collecting additional data points. I feel like the marketing claims are saturating timelines and overstating the capabilities (these models are still awesome tools though), which, business strategy… but for those of us wanting to know the practicality of them it’s not exactly helpful.