NHacker Next on Rails
  • top
  • new
  • best
  • ask
  • show
  • jobs
login
GuidelinesFAQListsAPISecurityLegalApply to YCContact

Hacker News on Rails, powered by React on Rails Pro. Open the source on GitHub by ShakaCode.

Commit 06fc0df

Rendered at 09:12:14 GMT+0000 (Coordinated Universal Time) with Rails + React on Rails RSC.

Benchmarking Bot Detection Systems Against Modern AI Agents in 2026 (research.poh.org)

45 points by mdahardy 16h ago | 7 comments

asksomeoneelse 1h ago
I don't want to say "that's an ad", but I can see an obvious conflict of interest there.

Also, the methodology just compare the true-negative detection rate for known negative, it seems?

> "Proof of Human identified 99% of automated sessions"

> "We measured detection rates for automated sessions but did not evaluate false-positive rates among human respondents."

Trivially easy measure to game if so. "isHuman(x) = return false" gets you a 100%.

vikrantrathore 4h ago
In your bot detection systems I do not see open source system like anubis, safeline or sibuna (I am the author of sibuna). Sibuna is built as self-hosted alternative to cloudflare with console UI providing OWASP CRS, JA4 TLS signature from ingress, these are still necessary besides bot detection and cloudflare offer it as enterprise offering.

I have a smaller benchmark for them in my book at https://insanai.github.io/sibuna/. If you can provide your benchmarking code as open source will incorporate this report into sibuna book.

aurareturn 1h ago
When they run these benchmarks, they should measure false positives - because it's annoying for a real human to solve a captcha.

If blocking 1 bot request is +1 then making a human solve a captcha should be something like -100. That's how we should grade.

conception 6h ago
Based on trying to buy anything retail that is scarce, bot detection systems are definitely not working. I wonder how their system would do against https://refractbot.com/ or other popular retail bots.
routerl 2h ago
To be fair, most bot detection can't and isn't trying to catch 100% of bots. Generally the idea is to prevent repeated abuse, rather than stop every single instance of it.

In the case of the scalper bot you've linked, the goal of bot protection would be to make the bot's transactions so computationally expensive/slow that they miss out on most of their intended purchases.

mimimile 1h ago
While building a project, I needed a crawler to read robots.txt. Out of 10,000 sites, just over 1,200 refused access outright.
AznHisoka 5h ago
Did they use the browsers against sites they owned or 3rd party ones? If the latter, then lots of sites have Cloudflare Bot Management on, without actual strict restrictions on bots (a lot of sites using CLoudflare as a CDN for instance)