LinkedIn Adopts Apache Celeborn to Transform Large-Scale Spark and Flink Infrastructure

LinkedIn has successfully deployed Apache Celeborn as a centralized, shared remote shuffle service across the overwhelming majority of its extensive Apache Spark and Apache Flink workloads. By completely separating intermediate data storage from underlying compute infrastructure, the professional networking platform has achieved unprecedented architectural efficiency. The strategic transition from its legacy Magnet implementation has yielded dramatic operational improvements, most notably reducing daily shuffle fetch failures from roughly 750 down to a mere 20. Furthermore, the migration has driven down overall compute costs by 10 percent while accelerating average application runtimes by 14 percent, reinforcing the platform’s capacity to process massive data volumes reliably.
The Scale and Engineering Challenges of LinkedIn’s Offline Data Platform
To understand the magnitude of this architectural shift, one must examine the staggering scale of LinkedIn’s offline data platform. On any given day, the infrastructure executes upwards of 300,000 distinct Spark jobs distributed across thousands of production nodes. Together, these demanding operations consume an estimated 200 petabyte-hours of compute annually.
As the organization continues its strategic migration toward a unified Flink architecture designed to handle both real-time streaming and historical batch processing, the scale of internal workloads has grown exponentially. Large-scale data backfills, in particular, place immense pressure on intermediate shuffle phases. The shuffle operation—the process of moving intermediate data between distinct processing stages—is historically one of the most volatile bottlenecks in distributed data processing. It places heavy concurrent stress on network interfaces, storage disks, and overall application runtimes.
Before adopting Apache Celeborn, LinkedIn sought to mitigate shuffle read inefficiencies through the creation of Magnet. Developed internally as a push-based shuffle implementation, Magnet was eventually contributed to the open-source community in Apache Spark 3.2. Magnet’s primary mechanism involved aggregating shuffle blocks by reducer partition into larger files, which successfully reduced the frequency of random disk reads.
However, Magnet suffered from a fundamental structural limitation: shuffle storage remained inextricably bound to the compute cluster. External Shuffle Services still shared critical physical resources—including CPU cycles, memory allocations, disk I/O bandwidth, and network pipelines—with active compute workloads. This resource contention frequently led to cascading performance degradation and node-level instabilities during peak processing hours.

Transitioning to Apache Celeborn and Decoupling Compute from Storage
The deployment of Apache Celeborn fundamentally alters this dynamic by establishing a decoupled architecture. In a Celeborn-powered environment, map tasks actively push intermediate data to a dedicated fleet of remote workers. These workers independently aggregate and store partitions entirely isolated from the compute executors.
When reducer tasks require data, they fetch it directly from the Celeborn workers. This separation allows shuffle storage to scale independently of compute clusters, drastically reducing the system’s reliance on executor lifecycles. Unlike Magnet’s merger External Shuffle Services, which operate directly within the Spark compute cluster, Celeborn functions as an independent service supported by a replicated control plane. This architectural decoupling effectively eliminates resource contention between compute-heavy transformations and shuffle data handling.
Engineering Reliability at Enterprise Scale
Operating a remote shuffle service reliably across thousands of enterprise nodes and hundreds of thousands of daily applications required substantial engineering innovation beyond the upstream open-source project. Aravind Patnam, a staff software engineer at LinkedIn, noted the complexity of the deployment, emphasizing that tailoring the platform for production-grade stability demanded rigorous internal development.
To achieve high availability, LinkedIn engineered a five-node Raft-based control plane distributed strategically across multiple availability zones. This control plane relies on Apache Ratis to handle metadata replication, automated leader election, and seamless failover mechanisms.
To safeguard against hardware degradation and infrastructure hiccups, the engineering team introduced network-aware replication. They also deeply integrated Celeborn with LinkedIn’s proprietary host health platform. This integration allows the system to automatically deprioritize workers that are approaching scheduled maintenance cycles and dynamically exclude degraded workers from active routing. According to internal metrics, these targeted reliability enhancements contributed to a twentyfold reduction in task failures during infrastructure disruptions when compared to the legacy Magnet framework.
/filters:no_upscale()/news/2026/11/linkedin-celeborn-remote-shuffle/en/resources/1Screenshot%202026-10-31%20at%203.44.38 PM-1793488412813.png)
Data Plane Optimization and Performance Gains
Beyond architectural decoupling, LinkedIn executed a series of aggressive data plane optimizations to maximize throughput and minimize latency. The organization migrated its shuffle storage infrastructure entirely to high-performance NVMe Solid State Drives (SSDs), cutting overall shuffle duration by 40 percent.
In a move to optimize storage density and network utilization, LinkedIn replaced the standard LZ4 compression algorithm with Zstandard (Zstd). This transition reduced the required shuffle worker fleet size by approximately 35 percent. Additionally, infrastructure teams provisioned workers with high-throughput 100 Gbps network interfaces and optimized Java Virtual Machine (JVM) configurations and data-plane logic, accelerating slot selection times by a factor of six.
Cumulatively, these optimizations drove shuffle costs down by nearly 90 percent on a per-task basis. Simultaneously, the proportion of executor runtime spent idling or waiting on shuffle operations plummeted by 10 to 15 percent from historical peak levels.
Implications for Unified Flink Workloads and Future Outlook
The successful rollout of Apache Celeborn has provided critical foundational support for LinkedIn’s broader architectural evolution. The company’s converged Flink platform now processes hundreds of millions of events per hour during historical data backfills, completing complex, multi-day workloads in mere single-digit hours.
Today, thousands of internal pipelines rely on unified Flink computations for both nearline processing and historical corrections, effectively replacing redundant parallel Spark implementations. Looking ahead, LinkedIn plans to explore advanced caching mechanisms and faster upstream project upgrades while maintaining its active commitment to contributing engineering enhancements back to the Apache Celeborn community.






