We made LLM inference a lot faster.
1/n
Speculative decoding throws away draft tokens when they don't match. What if you re-weighted your samples instead using importance sampling?
We introduce SMC-SD, sequential Monte Carlo speculative decoding, an approximate sampler that speeds up inference by 2.36× over SOTA speculative decoding, while producing a controllable Pareto frontier of speed and approximation accuracy.
ArXiv:
arxiv.org/abs/2604.15672
minimal repo release:
github.com/abdelfattah-lab/s…
Joint work with Mauricio Barba da Costa (@mauricio_b25181), Chi-Chih Chang (
@CCCCC1009CCC ), Cameron Freer, Tim Vieira (
@xtimv), Ryan Cotterell, and Mohamed Abdelfattah (
@mohsaied).