IoT Infrastructure Monitoring — Smart Manhole System

Domain: IoT / Smart City / Infrastructure Type: Proprietary IP (built and presented to a municipal authority; project discontinued, IP retained) Team Size: Backend + IoT integration

---

The Problem

Urban infrastructure teams responsible for underground utilities — sewage, drainage, telecom ducts — had no way to monitor manhole conditions in real time. Flooding, unauthorized access, structural tilt, and extreme temperatures went undetected until they surfaced as failures, safety incidents, or service disruptions. Inspection rounds were manual, periodic, and reactive. The unit economics of a city-wide physical inspection program simply don't work at scale.

We built this as proprietary IP and presented it to the municipal authority responsible for monitoring approximately 50,000 manholes across the city. At that scale, "IoT project" stops being a hardware exercise and becomes an engineering problem where every design decision gets multiplied by 50,000. The system had to:

• Continuously monitor manhole conditions via battery-powered sensors installed in harsh underground environments
• Surface real-time anomalies through a prioritized alert pipeline — not a firehose of raw telemetry
• Work reliably over cellular networks that weren't designed with the ingest patterns of 50k devices in mind
• Present fleet state to operators through a dashboard that doesn't devolve into a sea of red pins
• Be cheap enough per node that the project was operationally defensible

The project was ultimately not deployed and was later discontinued; the IP remains with us.

---

What We Built

Sensor integration

• CNDINGTEK DC413 sensors for manhole monitoring, selected for their combination of metrics, battery life, and NB-IoT compatibility
• Metrics captured: water height, temperature, tilt angle, battery level, tamper detection
• NB-IoT (Narrowband IoT) as the primary link — chosen specifically for low-power wide-area characteristics that make battery-powered underground deployment viable
• 4G gateway fallback for locations where NB-IoT carrier coverage was thin or absent

Data ingestion API

• Python-based ingestion service receiving sensor telemetry
• Binary protocol parsing for the CNDINGTEK proprietary data format — byte-layout reversal, checksum validation, endianness handling, fragment reassembly
• Strict validation at the boundary so malformed frames from a flaky link never corrupt downstream state
• Idempotent writes — sensors re-transmit after reconnection, and a re-received frame must not double-count events
• Out-of-order tolerance — samples arriving stale (hours after the fact, post-reconnect) are written to their correct temporal position, not appended as if they were live

NB-IoT gateway

• Gateway configuration and provisioning workflows
• Sensor registration flow — bind a device EUI to a manhole ID, a physical install location, and an owning operations team
• Connectivity and signal-quality monitoring, because a "missing" device is sometimes offline and sometimes vandalized
• Remote configuration updates so sample rates, thresholds, and reporting cadence can be tuned fleet-wide without a truck roll

Monitoring & alerting

• Threshold-based rules per metric (water level, temperature, tilt)
• Battery health trend detection for preventive maintenance scheduling
• Tamper detection with prioritized routing
• Historical time-series for trend analysis and anomaly baselining

---

Architecture Deep Dives

Sensors (DC413) → NB-IoT / 4G → Gateway → Ingest API → Time-series store → Alert pipeline / Dashboard

Scale math — why the architecture matters at 50k devices

At a conservative one-sample-per-hour reporting cadence, 50,000 devices generate ~14 messages per second steady-state. Outage-recovery bursts push that an order of magnitude higher as the fleet catches up after a carrier incident. Any architecture that doesn't acknowledge that asymmetry — normal load is easy, recovery load is the design point — falls over the first time a carrier has a bad night.

The ingest path is built around three principles:

1. Stateless ingest workers behind a load balancer so capacity scales linearly for recovery bursts
2. Durable intake buffer decoupling ingest from downstream processing so a slow consumer never backs pressure up to devices
3. Idempotent, out-of-order-tolerant writes so buffered messages can be replayed without data duplication

Device duty cycle & data economics

Battery-powered NB-IoT devices live or die by their duty cycle. Every byte on the wire costs battery and cellular-plan budget. The protocol reports compact binary frames at configurable intervals; thresholds and sample rates are tunable from the backend via downlink config so a device's behavior can be changed without physical access. Dead-zone reporting (silence when nothing is changing) is preferred over periodic heartbeats where possible, with a watchdog heartbeat on a long interval to distinguish "quiet" from "failed".

Time-series storage

Manhole telemetry is a classic time-series workload: heavy writes, mostly-recent reads, occasional historical scans for trend analysis. The schema is keyed on (device_id, metric, timestamp) with server-side timestamps preserved alongside device-reported timestamps so clock skew and delayed delivery are both visible. Retention is tiered — high-resolution recent data, downsampled longer-term data — so analytics queries stay fast as the fleet accumulates years of history.

Alert pipeline — making 50k devices operable

A raw threshold-alert system at 50k devices produces a stream of alerts no human can act on. The pipeline is designed around what an operations team can actually process:

• Deduplication — repeated firings of the same condition are coalesced into a single open incident, not a notification stream
• Flapping detection — metrics that oscillate around a threshold don't generate an alert per oscillation
• Severity classification — tamper and flooding route to immediate operator channels; battery-trend and tilt drift route to a scheduled-maintenance queue
• Per-location context — every alert carries the physical install record, owning team, and a trail of recent readings so the operator has what they need before they open the dashboard
• Silence windows — scheduled maintenance on a manhole suppresses alerts for that device without disabling the rule globally

Sensor protocol engineering

The CNDINGTEK data format was documented sparsely and behaviorally. Getting to a production-ready parser meant reverse-engineering the binary layout, validating our understanding against live sensor captures across multiple devices, and building a test harness of captured frames so protocol work could be verified without a device in the loop. The parser is defensive: unknown fields are preserved, malformed frames are rejected with enough diagnostic context that a flaky device can be identified and tagged, not just logged.

Fleet operations

At city scale, individual devices are invisible — what matters is fleet health. The operator view emphasizes aggregates: percentage of devices reporting in the last N hours by neighborhood, battery-distribution histograms, signal-quality heatmaps, and a drill-down from aggregate anomalies to specific devices. A 50k-pin map is a theater prop; a searchable, filterable table with saved views is an operations tool.

Field reality

Infrastructure IoT lives or dies on install discipline. The system treats device-to-manhole binding as a first-class provisioning event — GPS tag at install time, photographic record, owning team assignment, and a reconciliation pass that flags devices reporting from coordinates materially off their registered location (likely mis-bound or stolen).

---

Challenges Solved

• Low-power wide-area communication — NB-IoT link characteristics dictated the duty cycle, payload size, and retry semantics of the entire stack
• Harsh environments — intermittent connectivity and data gaps treated as a steady-state condition, not an exception
• Proprietary sensor protocol — binary format reverse-engineered and validated against live-device captures with a replay harness for regression
• City-scale architecture — ingest, storage, and alerting designed around recovery-burst load, not steady-state, because recovery bursts are what break these systems
• Operator ergonomics at scale — alert deduplication, flap suppression, and aggregate-first dashboards so 50k devices become operable, not overwhelming

---

Tech Stack

Layer	Technology
Sensors	CNDINGTEK DC413
Connectivity	NB-IoT (primary), 4G (fallback)
Backend API	Python
Gateway	NB-IoT Gateway (custom provisioning and config)
Protocol	Proprietary binary format — custom parser with replay-test harness
Storage	Time-series schema with tiered retention (high-res recent, downsampled historical)
Alerts	Deduplicated, severity-classified, flap-suppressed pipeline

---

Outcome

• Working end-to-end prototype demonstrated to municipal authorities for 50,000+ manhole monitoring
• Real-time monitoring of underground infrastructure conditions across a representative test deployment
• Automated, deduplicated alerting replacing periodic manual inspection
• Fleet health tracking (battery, signal, uptime) for proactive maintenance
• Architecture sized for city-wide scale, with the asymmetric recovery-burst load as the explicit design point
• Project was not deployed and was later discontinued — IP retained by Kiwimesh for re-use on future IoT engagements

---

Key Takeaway

IoT projects expose a specific kind of engineering maturity: they span hardware protocols, low-power networking, intermittent connectivity, time-series scale, and operator ergonomics at fleet size, and any one of those dimensions can sink the project. Getting this system to work meant reverse-engineering a proprietary sensor protocol, respecting the duty cycle of battery-powered devices, designing the backend around recovery-burst load rather than steady-state, and — critically — designing an alert pipeline a real operations team could actually live with. Even though the project didn't reach production, the design decisions and the IP we retained are directly applicable to the class of problem again.