Get the weekly digestSubscribe

/ THE FEED /

Read what ships.

Deep, recent reads ranked by depth and freshness.

Machine Learningremove filterData Engineeringremove filterTestingremove filterPostgreSQLremove filterGitLabremove filterGitHubremove filterSpotifyremove filterClear all

  1. 01Spotify · scaling

    Advancing Generative Recommendation Inference: our collaboration with NVIDIA on TensorRT LLM

    Spotify migrated its generative recommendation inference to NVIDIA TensorRT-LLM to address performance bottlenecks with vLLM and feature gaps in other engines. Through a collaboration with NVIDIA, the team added support for beam search, multi-item scoring, and soft token injection into TensorRT-LLM. This integration resulted in significant cost reductions of up to 54 percent for beam search and 7 percent for scoring workloads.

    DEPTH ■■■■■7 min read9 days ago

    Machine Learning

    Sign in to save and rate

  2. 02GitHub · performance

    How we make AI coding more cost efficient without sacrificing task quality

    GitHub optimized GitHub Copilot to improve cost efficiency without sacrificing task quality by evaluating changes across the entire agentic workflow. The team implemented four targeted adjustments including selective output compression, removing unused line-number formatting, shortening prompts via meta-prompting with behavioral tests, and delivering background work completions directly. These modifications reduced token usage and model inference costs globally without causing task-success regressions.

    DEPTH ■■■■■9 min read7 weeks ago

    Machine Learning

    Sign in to save and rate