Read what ships.

01Meta · architecture

The Thin Orchestrator: How We Made Our AI Agents More Reliable by Giving the Router Less to Do

Meta engineers reduced hallucinations and improved reliability in their data pipeline agents by replacing a single heavyweight LLM orchestrator with a thin deterministic router and fourteen specialized subagents. The thin orchestrator handles routing and reliability checks through a fixed protocol, while isolated subagents execute domain reasoning. This architecture reduced configuration preparation time from a day to under an hour.

DEPTH ■■■■■7 min read6 days ago

Machine Learning

Sign in to save and rate

02Meta · architecture

Inside Prometheus: Meta’s Journey to a Gigawatt-Scale Supercluster

Meta built Prometheus, a gigawatt-scale AI supercluster designed to scale up to one million GPUs using multi-generation NVIDIA hardware and specialized backend Ethernet networks. The team achieved high goodput and significantly reduced system interruptions by implementing an in-rack sparing strategy, parallelism-aware fault tolerance, and close co-engineering with vendors. These infrastructure advancements allowed the team to successfully train frontier foundational models like Muse Wonder.

DEPTH ■■■■■17 min read6 days ago

InfrastructureMachine Learning

Sign in to save and rate

03Honeycomb · architecture

OpenTelemetry Collector Configuration for LLM Observability

LLM applications emit bulky telemetry containing sensitive user content and disparate attribute names. The team uses an OpenTelemetry Collector pipeline to normalize, enrich, redact, and export this data to Honeycomb. This centralizes control, protects data before it leaves the network, and maintains consistent observability.

DEPTH ■■■■■12 min read2 weeks ago

Sign in to save and rate

04Meta · practice

How Meta Built Forum With AI-Native Engineering

Meta engineers built the Forum app using a human-on-the-loop AI agent workflow that automated planning, building, testing, and verification. By shifting focus from code generation to automated verification, tooling improvements, and architectural legibility, the team reached feature parity in eight weeks while catching systemic memory issues early.

DEPTH ■■■■■8 min read10 days ago

FrontendMobile

Sign in to save and rate

05Meta · architecture

Meta’s Generative Ads Model (GEM): Scaling the Central Model and Compute Engine for Frontier Ads AI

Meta developed GEM, a unified recommendation foundation model using an LLM-style architecture called Kunlun to handle heterogeneous ads data across multiple surfaces. The team implemented block attention, custom low-precision kernels, and post-training intelligence transfer techniques like LoopFM and Solaris to scale training efficiency and bypass real-time serving latency bottlenecks. The resulting system delivered an 18.9% conversion improvement and a 12% click-through rate increase across Meta ads surfaces.

DEPTH ■■■■■15 min read12 days ago

Machine Learning

Sign in to save and rate

06Meta · architecture

OCP Summit 2026: Powering AI with Open Innovations in Scale-Up, Scale-Out, and Scale-Across Networks

Meta shared its next-generation open networking infrastructure for AI at the OCP Summit 2026, including liquid-cooled 102.4T switch platforms, multiplane scale-out fabrics, and Ethernet-based scale-up solutions. The team also highlighted collaborative industry standards like ESUN and OSPA to unify hardware and transport layers. These open innovations aim to support massive gigawatt-scale AI clusters across diverse data center environments.

DEPTH ■■■■■12 min read12 days ago

Machine Learning

Sign in to save and rate

07Honeycomb · practice

How Honeycomb Private Cloud Drinks From the Fire Hose

The Honeycomb Private Cloud team struggled to manually track and integrate rapid infrastructure and configuration changes produced by upstream teams. To solve this, they built custom scripts to generate automated drift reports and eventually integrated AI to help triage and track the necessary follow-up work. As a result, the team successfully scaled their integration process into a reliable daily workflow.

DEPTH ■■■■■10 min read2 weeks ago

Sign in to save and rate

08Meta · architecture

GenRecs Tokenization: Transforming Ads Recommendation at Meta with Stable Model Vocabulary

Meta implemented GenRecs Tokenization to replace dedicated item IDs with a stable semantic vocabulary for ads recommendation. Using a three-stage encoding and quantization pipeline, the system enables language models to reason about ad content and user preferences simultaneously. This approach resulted in a +0.35% lift in conversions and improved cold-start performance for new entities.

DEPTH ■■■■■6 min read6 days ago

Machine Learning

Sign in to save and rate

09Meta · scaling

2026 World Cup: Breaking Records, Spiky Charts and Reliability at Scale

Meta's infrastructure handled record-breaking traffic during the 2026 World Cup, including 30 million messages per second on WhatsApp, by relying on automated load balancing, capacity buffers, and targeted pre-builds. Traffic patterns revealed that surges occurred primarily during breaks rather than gameplay, requiring region-specific forecasting rather than cross-region load shifting. The systems maintained full reliability and allowed normal continuous deployment schedules to continue without manual intervention or system freezes.

DEPTH ■■■■■5 min read10 days ago

Infrastructure

Sign in to save and rate

10Meta · migration

Shipping Media Over QUIC at Meta Scale for Instagram Live Ingest

Meta deployed the emerging Media over QUIC (MoQ) protocol for Instagram Live ingest on iOS and Android to replace their legacy RUSH protocol. By using edge translation to avoid backend rewrites and a phased rollout strategy, the team achieved metric parity with the old protocol without regressions. This deployment establishes a foundation to extend MoQ across Meta's applications for lower-latency and more resilient video streaming.

DEPTH ■■■■■8 min read13 days ago

FrontendMobile

Sign in to save and rate

11Honeycomb · practice

Our Limits Are Not Our Imagination

Software development practices are shifting as artificial intelligence makes advanced rigor, such as formal verification and property-based testing, an affordable default for greenfield projects. Developers can leverage AI agents to thoroughly review code and enforce strict architectural boundaries rather than just writing code faster. However, successful outcomes require comprehensive development systems and thinking rather than relying on AI alone.

DEPTH ■■■■■3 min read5 days ago

Sign in to save and rate

12Julia Evans · practice

Replacing the old battery on rechargeable bike lights

An individual attempted to repair old rechargeable bike lights with degraded batteries by cutting open the casing, desoldering the dead cells, and soldering in replacements. Despite lacking electronics experience and losing some screws, the repair was successful and restored the functionality of the lights.

DEPTH ■■■■■5 min read4 weeks ago

Sign in to save and rate

13Honeycomb · practice

Dr. Cat Hicks on the Psychology of Software Teams

Engineering leaders must establish clear structures and cultural norms to reduce the psychological identity threat and anxiety developers experience when adopting AI tools. Research indicates that organizational trust regarding AI is currently fractured but repairable through realigned expectations and accountability. Furthermore, engineers are advised to build durable sources of professional identity rather than relying solely on mastery of specific languages or legacy codebases.

DEPTH ■■■■■3 min read3 weeks ago

Sign in to save and rate