Scaling Edge-to-Cloud Telemetry: High-Throughput MQTT Ingestion & Time-Series Pipelines

As industrial IoT deployments scale from a few dozen lab prototypes to tens of thousands of distributed sensor nodes, the cloud ingest architecture faces unprecedented pressures. Bursty data streams, intermittent cellular connectivity, variable latency, and strict telemetry storage requirements can quickly overwhelm traditional REST APIs and relational databases.

To maintain real-time situational awareness and historical analytics without breaking cloud budgets, IoT architectures require purpose-built telemetry ingestion pipelines.

In this article, we explore the design choices behind KRAG’s high-throughput telemetry infrastructure, spanning the edge microcontroller to distributed time-series backends.

---

1. Serialization Matters: Protocol Buffers vs. Verbose JSON

In early prototyping phases, engineers almost universally choose JSON for telemetry:

{
  "device_id": "SN-849204-KRAG",
  "timestamp": 1736481029,
  "vibration_x": 0.0421,
  "vibration_y": -0.0193,
  "vibration_z": 0.9812,
  "temperature_c": 34.2,
  "battery_millivolts": 3280
}

While readable, this single payload consumes 192 bytes. Multiply that by 1,000 devices reporting at 10 Hz:

$$\text{Data Rate} = 1{,}000\,\text{devices} \times 10\,\text{Hz} \times 192\,\text{bytes} \approx 1.92\,\text{MB/s} \approx 5\,\text{TB/month}$$

On standard cellular telemetry SIMs, this overhead leads to catastrophic monthly data charges.

Switching to Compact Protocol Buffers (Protobuf)

By defining a schema with Protocol Buffers:

syntax = "proto3";

message TelemetrySample {
  uint64 timestamp_ms = 1;
  int32 vibration_x_mg = 2;
  int32 vibration_y_mg = 3;
  int32 vibration_z_mg = 4;
  int32 temp_centi_c = 5;
  uint32 battery_mv = 6;
}

Using variable-length integer encoding (varints) and eliminating repetitive string keys, the exact same payload compresses down to 24 bytes—an 87.5% reduction in bandwidth consumption.

Furthermore, embedded microcontrollers can serialize Protobuf in a few hundred microseconds using lightweight C libraries like nanopb, requiring zero heap allocation.

---

2. MQTT Quality of Service (QoS) in Real-World Networks

MQTT provides three Quality of Service levels:

• QoS 0 (At most once): Fire-and-forget, no acknowledgement.
• QoS 1 (At least once): Packet confirmed via PUBACK; retransmitted if unacknowledged.
• QoS 2 (Exactly once): Four-step handshake ensuring zero duplicates.

Many teams mistakenly assume QoS 2 is always best for mission-critical telemetry. In practice over cellular LTE-M or satellite links, QoS 2 introduces significant round-trip latency and battery drain, requiring up to four network round-trips per message.

Our Recommended Strategy:

1. High-Frequency Sensor Streams: Use QoS 0 with client-side ring buffering. If a packet drops during an antenna fade, the next sample arrives 100 ms later anyway.
2. State Changes & Alarms: Use QoS 1 with idempotent message IDs at the ingest layer so duplicate receipts can be deduplicated cleanly without QoS 2 overhead.
3. Keep-Alive Heartbeats: Configure adaptive MQTT PING intervals based on cellular operator NAT timeout rules (typically 60 to 120 seconds) to avoid unnecessary radio wakeups.

---

3. High-Throughput Edge Buffering (Local Flash Journaling)

What happens when an industrial asset (e.g. a locomotive or offshore wind turbine) passes through a cellular dead zone for four hours?

Devices without edge storage will drop incoming telemetry. To prevent telemetry loss, KRAG designs firmware with a non-volatile circular flash journal (using SPI NOR flash or SPI FRAM):

┌────────────────────────────────────────────────────────┐
│               Circular SPI Flash Buffer                │
├───────────────┬────────────────────────┬───────────────┤
│ Read Pointer  │ Telemetry Samples      │ Write Pointer │
│ (Transmitted) │ (Buffered during loss) │ (Incoming)    │
└───────────────┴────────────────────────┴───────────────┘

When network connectivity drops:

1. Samples are packed into flash memory pages.
2. When cellular connection restores, a low-priority background thread streams historical pages using batched packets.
3. Real-time alarm packets take priority over backfilled historical playback.

---

4. The Ingestion Pipeline Architecture

At the cloud edge, handling 100,000+ concurrent persistent TCP connections requires a horizontally scalable broker architecture:

 [Connected Devices]
          │
          ▼  (TLS 1.3 / mTLS on port 8883)
┌─────────────────────────────────┐
│ Distributed EMQX / VerneMQ      │
│ Cluster (Behind NLB)            │
└──────────────┬──────────────────┘
               │
               ▼
┌─────────────────────────────────┐
│ Apache Kafka / Redpanda Stream  │  ◄── Partitioned by Device ID
└──────────────┬──────────────────┘
               │
       ┌───────┴───────┐
       ▼               ▼
┌──────────────┐ ┌───────────────┐
│ ClickHouse / │ │ Real-Time     │
│ TimescaleDB  │ │ Alert Engine  │
│ (Historical) │ │ (Anomalies)   │
└──────────────┘ └───────────────┘

1. Mutual TLS (mTLS): Every hardware device carries a unique X.509 certificate baked into its hardware secure element (e.g. ATECC608B). The broker enforces client certificate authentication at the TLS layer.
2. Message Streaming: The broker forwards authenticated payloads directly to an Apache Kafka or Redpanda partitioned topic.
3. Time-Series Cold & Warm Storage: A consumer group ingests batched vectors into ClickHouse or TimescaleDB, allowing sub-second SQL queries across billions of raw sensor readings.

---

Summary

Building an industrial-grade telemetry pipeline requires treating the microcontroller, network transport, and cloud ingest as a single unified system. Prioritizing compact serialization (Protobuf), intelligent QoS selection, and resilient edge buffering ensures your fleet remains fast, reliable, and cost-effective as you scale.

Need help building custom telemetry firmware or high-scale cloud ingest systems? Reach out to our cloud and IoT engineers or get an estimate on your project.