Prompt: Design a secure OTA firmware update pipeline for a fleet of 500 ESP32-based devices deployed across SA. Include signing, rollback, staged rollouts, and handling devices on intermittent cellular connections.
1 – High‑level OTA Architecture
``
┌───────────────────────┐ ┌───────────────────────┐
│ Cloud / Edge Layer │ │ Device Fleet (500×) │
│ ----------------- │ │ ----------------- │
│ • Firmware repo │◄────────►│ • OTA‑client firmware │
│ • Signing service │ │ • Dual‑bank OTA flash │
│ • Manifest server │ │ • Cellular (NB‑IoT/LTE‑M) │
│ • Revocation list │ │ • TLS/DTLS │
└───────────────────────┘ └───────────────────────┘
`
- Cloud / Edge Layer
- A small HTTPS/WS server (AWS S3 + Cloudflare Workers, or an on‑premises reverse proxy) hosts the firmware blobs, a per‑region manifest (manifest.json
) and a public‑key revocation list.
- The server is protected by IAM‑role / certificate pinning; all traffic to devices uses TLS 1.2+ (or DTLS over UDP for low‑latency MQTT).
- Device Layer
- Each ESP32 runs the official Espressif OTA SDK, backed by two writable partitions (ota_0
, ota_1).
- A lightweight watchdog on boot will automatically roll back to the previous partition if a health‑check fails.
--------------------------------------------------------------------
2 – Firmware Signing & Verification
| Step | What happens | Why it matters |
|------|--------------|----------------|
| Key Pair | – One private key resides only in the build machine.<br>– One public key is baked into every device at provisioning time (see §6). | Prevents tampering; signature guarantees authenticity. |
| Signing Process | sha256(firmware.bin) → ECDSA‑P-256(private) → produces a 64 byte DER signature file (firmware.sig). | ESP32 can verify quickly with mbedTLS/mbedCrypto. |
| Manifest File | JSON per region/group: <br>`{ "version": "v1.2.3", "url":"https://cdn/.../fw.bin", "size":123456, "sha256":"…", "sig":"…" }` | Provides atomic metadata; devices only download the exact image referenced by the manifest. |
| Verification on Device | 1️⃣ Download firmware.sig. <br>2️⃣ Compute SHA‑256 of received fw.bin while streaming.<br>3️⃣ Verify signature with embedded public key. <br>4️⃣ If any step fails → abort, keep old firmware. | Guarantees that only the authorized vendor can push code. |
Key Rotation & Revocation
- Every build cycle a roll‑over key pair is generated; the new public key is added to the device provisioning bundle (OTA client).
- The server exposes revoked_keys.json
containing SHA‑256 fingerprints of expired keys. Device checks it at startup and refuses updates signed with a revoked key.
--------------------------------------------------------------------
3 – Dual‑Bank OTA on ESP32
`
┌───────────────┐
│ ota_0 ← current │ (bootloader selects this)
└────┬──────────────┘
▼ ▲
───┤ ├───
│ │ ota_1 │ new firmware downloaded here
└───┴─────────────┘
`
- Partition Table – Two user partitions (ota_0
, ota_1) and the bootloader.
- Update Flow (pseudo‑code):
`c
esp_http_client_config_t cfg = {...};
httpd_handle = esp_http_client_init(&cfg);
size_t total = get_manifest_size();
uint8_t *buf = malloc(CHUNK_SIZE);
for (offset=0; offset<total; offset+=CHUNK_SIZE) {
// range‑request ensures resumable downloads
resp = esp_http_client_perform(httpd_handle, "Range: bytes=%lu-%lu", offset, next-1);
if (resp->status_code != 206) return OTA_FAIL;
uint32_t written = esp_ota_write(ota_handle, buf, resp->data_len);
// check written == resp->data_len
}
esp_ota_end(ota_handle); // verify checksum, signature inside the loop
esp_ota_set_boot_partition(ota_info.partition);
esp_restart(); // new boot partition becomes active
`
- Rollback – If a post‑boot health‑check (e.g., device publishes heartbeat
within N seconds) fails, the bootloader automatically boots to the previous partition.
- The ESP32 OTA SDK keeps an ota_status
flag that is set to GOOD only after successful esp_ota_mark_app_valid().
--------------------------------------------------------------------
4 – Staged Rollouts
| Layer | Technique | Example |
|-------|-----------|---------|
| Device Grouping | • Device ID prefixes (e.g., SA‑00100…) <br>• Geo‑tags in the provisioning blob | “Phase‑1: 10 % of devices in Gauteng” |
| Manifest Per Group | Separate manifest URLs per group; same version tag but different download window. | https://cdn/manifest/gauteng.json |
| Control Plane | • A small REST API (GET /schedule) that returns the next wave and expiry timestamp.<br>• The device checks every 24 h, or when a new OTA client config is pushed via MQTT. | wave: 1 of 5, expires: 2026‑09‑30 |
| Safety Switch | If any device in the current wave reports a failure >10% of the time, the server automatically pauses that wave and falls back to previous firmware (via OTA rollback). | Real‑time KPI dashboard. |
Implementation Tip: Use AWS IoT Device Management or a custom MQTT topic (/ota/wave) for each group. Devices subscribe once; the broker publishes only when a new wave is ready.
--------------------------------------------------------------------
5 – Handling Intermittent Cellular Connections
| Challenge | Solution |
|-----------|----------|
| Low Duty‑cycle links (e.g., NB‑IoT) | Chunked download + range requests. The ESP32 stores the partially downloaded image in flash and continues from where it left off when connectivity returns. |
| Unreliable TCP over 2G/3G | Use MQTT‑over‑WebSocket with keep‑alive=30 s and clean session=false. The device will queue outgoing telemetry, so a reconnection simply re‑establishes the session without data loss. |
| Power‐constrained windows | Schedule OTA download during times when the device is idle or powered by solar/UPS; keep the cellular radio off otherwise to conserve energy. A simple sleep_mode() loop that wakes every X minutes checks for new manifest. |
| Large firmware blobs on slow link | Compression: use ZSTD or LZ4; store compressed image in OTA partition and decompress at boot (ESP32 can handle ~200 KB/s on LTE‑M). Alternatively, use delta updates with bspatch; only 5–10 % of the binary changes for a new version. |
| Connectivity loss during download | The device keeps track of offset in non‑volatile storage; on next boot it resumes from that offset (Range: bytes=offset-). If offset stays static >30 min, it aborts and reports to cloud (“download stuck”). |
--------------------------------------------------------------------
6 – Security Hardening
| Layer | Measure |
|-------|---------|
| Transport | TLS 1.2+ (or DTLS for MQTT) with certificate pinning on device; server presents a short‑lived client certificate if mutual auth is required. |
| Firmware Verification | ECDSA P‑256 signature + SHA‑256 hash. Public key stored in OTP or protected flash area (e.g., nvs_flash partition flagged as “secure”). |
| Key Protection | Hardware Crypto – ESP32 has a secure element for AES/Hash; use it to derive keys if needed. Store the private key only on the build machine, never in source control. |
| Revocation | Periodic fetch of revoked_keys.json; device refuses any update signed with a revoked key. |
| Bootloader Integrity | Sign bootloader too (or use the ESP32 secure boot feature). If the bootloader is tampered, the device refuses to boot. |
| Rollback Protection | Only a trusted OTA client can modify ota_info.partition. The OTA SDK’s esp_ota_mark_app_valid() guarantees that an unverified image cannot be marked as “good”. |
--------------------------------------------------------------------
7 – Operational Monitoring & Alerting
- Telemetry Topics
- /device/<id>/heartbeat
– sent every 30 s.
- /device/<id>/ota/status
– {state:"success|fail", version, bytes_downloaded}.
- Cloud Dashboard (AWS CloudWatch + Grafana)
- % of devices in each OTA phase.
- Failure rate per wave.
- Mean time to first byte (TTFB) for cellular downloads.
- Automatic Roll‑back Trigger
- If 5 consecutive heartbeats after a new OTA fail, the OTA service marks the firmware as “bad” and automatically publishes a rollback manifest (rollback.json
).
- Devices pull this and revert within minutes.
- Logging – Every OTA step logs an event to CloudWatch; if something fails on a device it reports ota_error` with details (e.g., "signature mismatch", "checksum error").
--------------------------------------------------------------------
8 – Step‑by‑Step Deployment for 500 Devices in SA
| Stage | What you do