NHacker Next on Rails
  • top
  • new
  • best
  • ask
  • show
  • jobs
login
GuidelinesFAQListsAPISecurityLegalApply to YCContact

Hacker News on Rails, powered by React on Rails Pro. Open the source on GitHub by ShakaCode.

Commit 06fc0df

Rendered at 12:45:27 GMT+0000 (Coordinated Universal Time) with Rails + React on Rails RSC.

Terminal Bench 4 Is Saturated (preferencemodel.com)

24 points by nharada 1d ago | 3 comments

Good4boothee 1d ago
I think ambiguously worded questions have their place, but you have to give up automatic scoring on such tasks.
vintermann 1d ago
You don't have to give it up entirely, you could grade on the interpretation they chose. Personally I'm a fan of models which seek clarification on critical ambiguities rather than barging ahead "autonomously" though.

But wow, they found a lot of issues with the benchmark.

scotty79 1d ago
I really like the benchmarks where frontier models score each other's solutions to a problem.