MMNE (make model names easy)
Which model names (-m) can you use in Harbor? Now there's one answer: LiteLLM's. harbortoolkit.com/model
6
509
Harbor 🤝 Docker Sandboxes
Agents that can edit files and run commands can also game their own evals. Harbor + Docker Sandboxes fix that: every eval task runs in an isolated microVM, with network policies for fair, reproducible runs. bit.ly/4B8xf9Z
1
5
45
1,650
𝚞𝚟 𝚝𝚘𝚘𝚕 𝚒𝚗𝚜𝚝𝚊𝚕𝚕 "𝚑𝚊𝚛𝚋𝚘𝚛[𝚜𝚋𝚡]" 𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 -𝚎 𝚜𝚋𝚡 -𝚝 <𝚝𝚊𝚜𝚔> to get started!
1
130
Harbor Framework retweeted
Now, run agent evals with @harborframework + @Docker Sandboxes.
Agents that can edit files and run commands can also game their own evals. Harbor + Docker Sandboxes fix that: every eval task runs in an isolated microVM, with network policies for fair, reproducible runs. bit.ly/4B8xf9Z
1
2
216
𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 -𝚍 𝚛𝚎𝚏𝚛𝚎𝚜𝚑/𝚋𝚕𝚎𝚗𝚍𝚎𝚛𝚋𝚎𝚗𝚌𝚑-𝟸𝟶
Introducing Blender Bench v1 (BBv1): a benchmark for frontier LLMs on 20 realistic 3D production workflows including animation, modeling, rigging, cloth & surfacing. GPT-6 Astra leads overall; Claude Opus 5.5 wins modeling & cloth simulation.
1
2
459
results need receipts! 𝚑𝚊𝚛𝚋𝚘𝚛 𝚞𝚙𝚕𝚘𝚊𝚍 to share your trials on the hub
Replying to @unreallabsai
Full results, harbor traces and implementation details in our blog - unreallabs.ai/blog/long-hori…
5
721
𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 -𝚍 𝚕𝚘𝚘𝚗𝚒-𝚕𝚊𝚋/𝚜𝚖𝚍𝚍-𝚋𝚎𝚗𝚌𝚑
New blog: Is human taste overrated in harness engineering? An agent harness is the system around a model. It decides which tools the model can call, what evidence it sees, and what it remembers between steps. Most harnesses are still designed by human taste. Run the agent, read where it failed, change the harness, try again. That loop is increasingly being automated, with a strong model rewriting the harness itself. So we asked how far that can go. We ran Qwen on three drug design tasks from SMDD-Bench and compared a manually redesigned harness against one produced by automated harness search with Claude in the loop. No single approach won everywhere. The manual harness pulled clearly ahead when the fix was changing what the agent sees and remembers. Automated search did better once that was in place and the remaining problem was how to search the chemistry itself. The hard part was rarely implementing the fix. It was diagnosing what kind of failure we were looking at, and that is where human taste still mattered. Work with @KevinH1119568 , @aviral_kumar2 and @niloofar_mire Read more 👇
1
1
5
763
The Harbor team will be at Runtime tomorrow! @alexgshaw will be hosting a Harbor genius bar @ryan_marten will be presenting “Lessons from Terminal-Bench” Come with your questions and feature requests and say hi!
1
6
49
8,075
Harbor Framework retweeted
With hundreds of contributors and thousands of participants on Discord and GitHub, the @terminalbench team ships frequent updates in the open so everyone can learn alongside them. And with @harborframework, they've built the infrastructure for anyone to benchmark the use cases they care about. Great to host the community at Laude Lab last week and hear what's coming next! @alexgshaw @ryan_marten @StevenDillmann @Mike_A_Merrill @andykonwinski
1
4
11
1,591
Harbor Framework retweeted
hey @harborframework community i just ported these 2,698 tasks to harbor what model should i post-train with these?
xiaomi MiMo oss RL environments + training code github.com/XiaomiMiMo/verl huggingface.co/datasets/Xiao…
1
2
10
1,035
Harbor Framework retweeted
Terminal-Bench meetup tonight *with* swag. Register if you haven’t already! Benchmarks, RL environments, new features, and state of the union luma.com/tbench
17
10
116
12,063
Harbor Framework retweeted
The gold standard for science is sharing reproducible and auditable experimental data. When it comes to benchmarking agents, this means configs, trajectories, rewards, and logs for every single trial. We built harbor to make this easy. Unreal's analysis (related to @melissapan's recent work on comparing cost between harnesses while keeping the model fixed) sets a good example: they have uploaded and attached the raw harbor jobs for every result (each can be reproduced with a single harbor command). Good work!
Introducing Unreal Agent: An open-source harness with state-of-the-art cost efficiency 39% cheaper than Codex+Astra on Terminal-Bench 4.0 while maintaining performance
7
4
80
9,404
Harbor Framework retweeted
Half the set is public with every judge's full logs, a scorer for your own judge, and the audit trail. 35 items are held out. Every case is a runnable @harborframework task, so you can point your own judge at the set and score it with one command. Full writeup: heuristic.build/blog/arbiter…
1
8
364
Harbor Framework retweeted
Replying to @vercel
@vercel’s fx.sh harness is now available as an agent judge in Harbor Rewardkit. It makes judge setup time 7× faster than Codex and 16× faster than Claude Code. @harborframework
1
2
7
555
Jev-as-a-Judge in Harbor Rewardkit!
Jev-as-a-Judge in Harbor Rewardkit is pretty amazing. It agrees with Fable and GPT and is very fast. When we test it on a real eval that uses an LLM judge, such as the @harvey LAB benchmark, Jev achieves perfect agreement but is 50x faster.
1
1
35
3,495
Harbor Framework retweeted
We love @harborframework's new observability feature!
Got streaming working with @tensorlake sandboxes on Harbor too. So you can see the agent's trajectory and files while it runs. PR is on the way.
1
12
1,773
𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 -𝚍 𝚑𝚊𝚗𝚍𝚜𝚑𝚊𝚔𝚎-𝚊𝚒/𝚊𝚝𝚕𝚊𝚜-𝚏𝚒𝚗𝚊𝚗𝚌𝚎
Replying to @jomulr
In my view, ATLAS Finance is the closest benchmark yet to real end-to-end professional financial work. Code to run the benchmark as a @harborframework task suite: github.com/Handshake-AI-Rese… Data/environments: huggingface.co/datasets/hand…
8
36
4,957