NHacker Next on Rails
  • top
  • new
  • best
  • ask
  • show
  • jobs
login
GuidelinesFAQListsAPISecurityLegalApply to YCContact

Hacker News on Rails, powered by React on Rails Pro. Open the source on GitHub by ShakaCode.

Commit 06fc0df

Rendered at 11:17:55 GMT+0000 (Coordinated Universal Time) with Rails + React on Rails RSC.

Why Is Speculative Decoding Fast? (jbarrow.ai)

50 points by gmays 1d ago | 5 comments

jbarrow 19h ago
This is my post! Happy to answer any follow-up questions people have.

I wrote this to try and clear up what I think is a common misconception (speculative decoding is a free lunch), when the picture is more nuanced (and imo, interesting).

ArashEdalat 22h ago
we have action chunking in robotics for the same reason so VLAs can analyze faster and come to actions, the latency and inference are both slow so many steps need to come together for this to happen correctly
ArashEdalat 22h ago
has anyone tried different approaches to this?
RugnirViking 22h ago
this morning I tried a different approach to speculative decoding by not writing any code at all and in fact going out for a walk around the lake near my house? What do you mean specifically?
guipsp 22h ago
...yes? Can you put more effort into your question?