At the edge, the number that matters is memory – not throughput (specially in Ramageddon)
The landscape of industrial stream processing is currently undergoing a significant paradigm shift. As organizations transition from centralized cloud-based data centers to decentralized edge computing environments—such as industrial gateways, vehicle telematics, and smart-grid controllers—the metrics for evaluating performance are being fundamentally rethought. A team of engineers at I-Dacs Labs has introduced "rekuiper," a Rust-based stream-processing engine designed to prioritize deterministic memory usage and stability over raw, theoretical throughput. This development comes at a critical time when "bursty" traffic patterns—caused by synchronized device reconnections after network outages—often crash traditional, Go-based stream engines that lack rigorous backpressure mechanisms.
The Problem with Traditional Throughput Benchmarks
For years, the industry standard for benchmarking stream-processing engines has relied almost exclusively on peak throughput numbers generated on high-performance servers with virtually unlimited resources. However, this metric is increasingly viewed as a "vanity metric" that fails to represent the harsh realities of the edge. In an industrial gateway, hardware is constrained: typically limited to a single CPU core and less than 1 GB of RAM.
In these environments, throughput is not the primary bottleneck; memory stability is. When thousands of IoT devices simultaneously reconnect following a power or network failure, they flush massive amounts of buffered data at once. Engines that do not strictly manage memory will either exhaust their heap or fail under the pressure. The engineering team behind rekuiper posits that a system that processes 100,000 messages per second but crashes when memory spikes is significantly less valuable than a system that maintains a steady, predictable footprint under the same conditions.
Methodology: A Rigorous Approach to Benchmarking
The study conducted by I-Dacs Labs sought to eliminate the "vibes-based" reporting often found in software benchmarks. To ensure accuracy, the team implemented a "proof-based" testing harness. Each engine—rekuiper, LF Edge eKuiper, Telegraf, and Redpanda Connect—was confined to a rigid environment: a container with exactly one CPU core and 1 GB of memory, with no swap space allowed to prevent hidden performance masking.
The benchmark utilized five distinct workloads designed to mimic real-world deployment scenarios:
- Telemetry Filtering: High-volume data throughput with minimal transformation.
- Per-Device Windowing: Tracking metrics for thousands of individual sensors.
- ESPHome States: Processing complex, nested JSON objects common in smart home ecosystems.
- Vehicle Telemetry Windows: Time-series analysis across moving asset fleets.
- Charger Session Tracking: State-heavy operations involving long-running sessions.
Unlike typical benchmarks that count "ingest" as "success," this test mandated that the output be verified message-by-message. If an engine reported a throughput of 100,000 messages per second but failed to produce a correct, complete output file, the test marked that as a failure.
The Memory and Performance Gap
The findings, which were published alongside the release of the rekuiper engine, reveal a striking divergence in performance characteristics. While traditional engines frequently hit their 1 GB memory ceiling during windowed workloads, rekuiper maintained a lean footprint of between 5 and 10 MB.
This drastic efficiency is attributed to three core design principles implemented in the Rust-based engine:
- Bounded Queues with Backpressure: rekuiper utilizes a "reserve-then-commit" admission control strategy. If a subscriber queue is full, the system exerts backpressure on the source rather than attempting to buffer infinite messages. This ensures that the engine never enters an "out-of-memory" state, even under extreme load.
- Incremental Window Aggregation: By calculating aggregates (like sum, count, or average) incrementally per-group, rekuiper avoids the common pitfall of storing raw records. Memory usage thus scales with the number of devices, rather than the number of incoming messages.
- Efficient Disk Spilling: For scenarios with intermittent connectivity, the engine utilizes a persistent, disk-backed cache. This allows it to buffer data during network outages without silently dropping messages, a feature that ensures data integrity in remote deployments.
Comparative Analysis of Results
The comparative data highlights that language choice (Rust vs. Go) is a factor, but not the sole differentiator. Structural design—specifically how the engine manages state and memory—accounts for the two-order-of-magnitude difference in performance.
In the "Per-Device Windowing" workload, Redpanda Connect and the standard eKuiper implementation both hit the 1 GB memory limit at high throughput levels, leading to either crashes or significant data loss. Conversely, rekuiper successfully processed the full range of test traffic while maintaining a memory footprint that remained essentially flat. The CPU utilization for rekuiper was also notably lower, typically using half the processing power of its counterparts. This efficiency is critical for edge hardware, where CPU cycles are limited and shared with other mission-critical tasks.
The Role of Rigorous Testing in Debugging
One of the most revealing aspects of the I-Dacs Labs report was the admission of a prior failure in their own development cycle. An earlier iteration of rekuiper showed a 15% data loss rate on specific workloads. Because the team had implemented a strict output-verification protocol, they were able to identify that their windowing logic was flawed—it was incorrectly collapsing groups and omitting data.
The team noted that if they had relied on a "throughput-only" benchmark, this bug would have likely been masked as "fast" performance, as the engine was performing less work by skipping necessary grouping operations. This admission underscores the growing consensus in the engineering community: without verified correctness, speed is merely a metric of how quickly a system can produce incorrect results.
Implications for the IoT Industry
The success of this benchmark has profound implications for the Industrial Internet of Things (IIoT). Companies deploying edge gateways for EV charging, vehicle telematics, and smart factory monitoring are increasingly sensitive to the "predictability" of their infrastructure. When an engine’s memory consumption is non-linear—growing in tandem with message volume—the system is inherently fragile.
By demonstrating that memory consumption can be decoupled from message volume through better architectural design, the developers of rekuiper have set a new benchmark for what is expected of edge-native software. The decision to make the entire test harness open-source and reproducible serves as an invitation to the broader community to challenge these results.
Moving Forward: The Future of Edge Processing
The broader impact of this research is likely to be a re-evaluation of current standards for stream-processing engines in constrained environments. As the volume of data generated by edge devices continues to rise, the ability of a system to handle "bursty" traffic without compromising stability will become the primary differentiator in the market.
For organizations currently relying on traditional engines, the findings suggest that significant overhead might be reclaimed by adopting more memory-conscious architectures. While the transition from existing systems like LF Edge eKuiper to a new engine requires a commitment to migration, the potential for increased reliability in remote, difficult-to-maintain environments provides a compelling business case.
The rekuiper engine is currently available as a beta release, licensed under a dual MIT/Apache-2.0 model, encouraging enterprise adoption. As the industry moves toward more sophisticated edge-processing requirements, the ability to maintain predictable performance in the face of network instability will define the next generation of industrial software. The lesson from this research is clear: in the world of edge computing, the most important metric is not how fast you can run, but whether you can survive the storm.
