Gateway Firmware — P0 Production-Readiness Design

Date: 2026-05-14
Branch: feature/thingsboard-integration
Status: Design — awaiting review
Target scope: Fleet 100+ devices, greenfield (no field devices yet)

---

1. Goals

Lift gateway_firmware/ from Beta to fleet-deployable by closing 5 P0 gaps:

1. P1 OTA + SHA256 checksum + dual-OTA partition + auto-rollback
2. P2 FreeRTOS task watchdog for all tasks + reboot recovery
3. P3 MQTT TLS + per-device credentials in NVS
4. P4 WiFi credential provisioning (factory-flash default + AP mode fallback)
5. P5 Async Modbus state machine (unblock RPC during scans)

Non-goals: signed firmware with PKI (defer N10), full secure boot v2 / flash encryption, per-device runtime configuration of Modbus slave map (S1), command-history persistence (S3).

---

2. Constraints & assumptions

• Deployment target: fleet 100+ devices at heterogeneous customer sites.
• Greenfield — free to change partition table.
• OTA distribution: ThingsBoard shared-attribute pushes fw_url + fw_checksum; binary hosted on an HTTPS server we operate.
• Image authenticity model: SHA256 + HTTPS chain-of-trust. No PKI/eFuse signing in this scope.
• MQTT credentials and WiFi defaults are flashed per-device at factory via an esptool.py NVS-image script. Single firmware image, per-device differs only in NVS.
• WiFi AP-mode portal is the field-installer override path.
• Task WDT trigger → reboot, with resetReason + hung-task-name surfaced in first telemetry after boot.
• Async Modbus target: full scan duration may remain ~10 s, but RPC p99 latency under load must be < 200 ms.

---

3. Foundation — Partition table & NVS layout

This must land before any device is flashed for real. Single-app partitions cannot OTA themselves to dual-OTA.

3.1 New file: gateway_firmware/partitions_two_ota.csv

# Name,   Type, SubType, Offset,   Size,     Flags
nvs,      data, nvs,     0x9000,   0x6000,
otadata,  data, ota,     0xf000,   0x2000,
phy_init, data, phy,     0x11000,  0x1000,
factory,  app,  factory, 0x20000,  0x180000,
ota_0,    app,  ota_0,   0x1A0000, 0x180000,
ota_1,    app,  ota_1,   0x320000, 0x180000,
nvs_keys, data, nvs_keys,0x4A0000, 0x1000,
creds,    data, nvs,     0x4A1000, 0x5000,
spiffs,   data, spiffs,  0x4A6000, 0x15A000,

Assumes 4 MB flash (esp32doit-devkit-v1 default). If a 16 MB SKU is procured later, expand ota_* slots.

3.2 platformio.ini changes

[env:esp32doit-devkit-v1]
board_build.partitions = partitions_two_ota.csv
board_upload.flash_size = 4MB

3.3 NVS namespace layout

creds is intentionally separate from wifi so an installer using the AP portal cannot accidentally overwrite the MQTT token.

3.4 Factory-flash workflow

Python script tools/factory_flash.py (new):

1. Take --token, --device-id, --default-ssid, --default-psk from CLI or CSV batch file.
2. Build a per-device NVS partition image with nvs_partition_gen.py.
3. Flash firmware (firmware.bin) + per-device NVS image via esptool.py write_flash.
4. Boot test → confirm device joins WiFi + TB MQTT within 60 s, then ship.

---

4. P3 — MQTT TLS + per-device credentials

4.1 Modules

• New: src/provisioning/nvs_creds.{h,cpp} — typed accessors for creds namespace.
• New: src/mqtt/tb_ca.h — PROGMEM constant with ThingsBoard CA chain.
• Changed: src/mqtt/mqtt_client.{h,cpp} — replace WiFiClient with WiFiClientSecure, port 8883, setCACert(TB_CA_PEM).
• Changed: src/mqtt_manager.cpp + src/main.cpp — load credentials from NvsCreds instead of compile-time secrets.h.

4.2 Behavior

• TB MQTT auth: access token as username, empty password (TB convention).
• If creds namespace is empty at boot → log fatal + redirect into AP mode (entry point shared with P4).
• setInsecure() is not allowed in any build profile, including pilot. Forces correct CA from day one.

4.3 Threat model coverage

4.4 Tests

• test/native_mqtt (existing) extended: mock WiFiClientSecure::setCACert/connect, assert TLS path used.
• New: test/native_nvs_creds — round-trip read/write of creds namespace via mocked Preferences.

---

5. P4 — WiFi provisioning + AP mode

5.1 Modules

• New: src/provisioning/wifi_provisioner.{h,cpp} — wraps existing IotWebConf dependency, exposes loadFromNvs(), startApPortal(timeoutMin), saveAndReboot().

5.2 Boot decision flow

boot
 ├─ load wifi NVS
 ├─ exists?
 │   ├─ yes → try connect (timeout 30s, 3 retries)
 │   │        ├─ success → proceed to MQTT (Section 4)
 │   │        └─ 3 fails → enter AP mode
 │   └─ no  → enter AP mode

AP mode
 ├─ SSID = "HMP-GW-<last4MAC>"
 ├─ Captive portal at http://192.168.4.1/
 ├─ Form fields: WiFi SSID, WiFi PSK
 │   (TB token field hidden when `creds` already populated)
 ├─ Save → write `wifi` NVS + reboot
 └─ 10-min idle timeout → reboot to retry station mode

5.3 Triggers for AP mode (runtime)

• Boot with empty wifi NVS.
• 3 consecutive station-mode connect failures.
• factoryReset RPC (erases wifi namespace, then reboots).
• Hardware button GPIO0 long-press ≥ 5 s (deferred — depends on hardware revision).

5.4 Tests

• New: test/native_provisioning — mock NVS, drive boot state machine through:
  • empty wifi → AP entered
  • valid wifi → station attempted
  • 3 failures → AP entered
  • portal save → NVS written, reboot scheduled

---

6. P1 — OTA + rollback

6.1 Modules

• New: src/ota/ota_updater.{h,cpp} — owns esp_https_ota lifecycle.
• Changed: src/mqtt_manager.cpp — subscribe v1/devices/me/attributes and .../attributes/response/+, dispatch OTA-related keys.
• Changed: src/state/health_monitor.cpp — publish fwVersion, resetReason, OTA state transitions.

6.2 Flow

1. After MQTT connect, request shared attributes ["fw_title","fw_version","fw_url","fw_checksum","fw_size"].
2. On attribute update where fw_version != FW_VERSION:
   • Telemetry: fw_state = "DOWNLOADING".
   • Start esp_https_ota_begin(fw_url) with CA bundle (firmware HTTPS server CA may differ from TB CA — bundled separately).
   • Stream chunks, accumulate SHA256.
3. Compare computed SHA256 with fw_checksum:
   • mismatch → esp_ota_abort, telemetry fw_state = "FAILED: checksum".
   • match → esp_ota_end, esp_ota_set_boot_partition, telemetry fw_state = "UPDATING", reboot.
4. On next boot: pending-verify window of 120 s waiting for WiFi + MQTT + one successful telemetry round-trip.
   • success → esp_ota_mark_app_valid_cancel_rollback(), telemetry fw_state = "UPDATED".
   • failure → ESP-IDF auto-boots prior partition on next reset; first telemetry from prior partition reports fw_state = "FAILED: rollback".

6.3 Concurrency

• OTA runs in its own task (stack 8 KB, priority above loopTask).
• Before download begins: set pauseScan = true flag observed by Modbus scan SM, drain command worker queue, suspend display task.
• Resume only on OTA abort path. On success path → reboot, so resume is unreachable.

6.4 Resilience

6.5 Tests

• New: test/native_ota — mock HTTPClient + esp_ota_* API, cover checksum mismatch abort, chunk failure abort, success path.
• Manual integration: bench device served a known-bad-checksum URL, observe rollback marker telemetry on next boot.

---

7. P2 — Task WDT for all tasks + recovery

7.1 Registration

In src/main.cpp after each xTaskCreate*:

esp_task_wdt_init(TASK_WDT_TIMEOUT_S, true);  // panic on timeout
esp_task_wdt_add(loopTaskHandle);
esp_task_wdt_add(modbusScanTaskHandle);
esp_task_wdt_add(displayUpdateTaskHandle);
esp_task_wdt_add(commandWorkerTaskHandle);
esp_task_wdt_add(otaTaskHandle);  // only while OTA active

7.2 Feeding cadence (per task)

7.3 Hung-task surfacing

ESP-IDF prints the hung task name to UART on panic, but ops can’t see UART. Capture into RTC RAM so it survives the soft reset:

1. Install esp_task_wdt_isr_user_handler → write hung task name to RTC RAM slot.

2. After boot, health_monitor reads esp_reset_reason() + RTC RAM, emits first telemetry:

   {"resetReason": "task_wdt", "hungTask": "ModbusScan", "fwVersion": "1.2.3"}
   

3. Clear RTC RAM slot after publish confirmed (avoid repeating same hung-task report).

7.4 OTA + WDT interaction

If WDT fires during OTA flash write: esp_ota_end was not yet called → otadata uncommitted → boot rolls back. Safe.

7.5 Tests

• test/native_health_monitor extended — mock esp_reset_reason() + RTC RAM accessor, verify mapping into telemetry fields.

---

8. P5 — Async Modbus state machine

8.1 Problem

Current modbus_sensor::backgroundScanTask already runs in its own task, but uses blocking delay() between reads while holding the modbus_bus mutex. RPCs that need the bus (readModbusRegister, writeI2C-to-Modbus, etc.) wait the full inter-register / inter-slave delay every time. p99 RPC latency observed under load: up to 10 s.

8.2 Refactor

Replace blocking-delay loop with a state machine that holds the mutex only for one I/O step per tick:

enum class ScanState {
  IDLE,
  SLAVE_BEGIN,
  REG_REQUEST,
  REG_WAIT,
  REG_DONE,
  SLAVE_DONE,
  PUBLISH,
};

Per tick:

1. xSemaphoreTake(modbus_bus_mutex, 50ms) — if fail, yield, leave state as-is.
2. Perform one short step (issue request, or read response, or advance index).
3. xSemaphoreGive.
4. vTaskDelay(inter_register_delay_ms / portTICK_PERIOD_MS).

8.3 Constraints kept

• Full scan duration roughly unchanged (~10 s) because inter-register/inter-slave delays are sensor requirements, not software artifacts.
• Existing abort/scan_request_id semantics retained — set aborted = true, SM returns to IDLE on next tick.
• ModbusMaster library unchanged.

8.4 New telemetry

• modbusScanDurationMs — wall time of last full scan.
• modbusErrorCountPerSlave — count of CRC / timeout fails per slave per scan.

8.5 Tests

• test/native_modbus_sensor extended — simulate an RPC arriving mid-scan; assert RPC completes within 200 ms while scan progresses.

---

9. Sequencing

F   Partition table + NVS layout (Section 3)          [BLOCKER for all flash work]
        │
        ├──► nvs_creds + wifi_provisioner modules
        │       ├──► P4a factory_flash.py script
        │       ├──► P4b AP mode portal
        │       └──► P3  MQTT TLS + load cred from NVS
        │
        └──► P1a OTA module (esp_https_ota + SHA256 + rollback)
                └──► P1b Shared-attribute subscribe + fw lifecycle

P2  Task WDT for all tasks            [parallel, no dependency]
P5  Async Modbus state machine        [parallel, no dependency, sequenced last]

Recommended 8-week plan

Critical path: F → P3 → P4 → P1 (~5 weeks).
Parallel track: P2 + P5 (additional dev).

Verification gates

• End S1: Device flashed via factory script boots into AP mode without panic; WDT registered for every task and observable on UART under induced hang.
• End S2: One pilot device runs 24 h on TLS + WDT, no degradation, no fatal reconnects.
• End S3: Three pilot devices OTA from FW 1.0.0 → 1.0.1; one device served deliberate bad-checksum URL → auto-rollback verified.
• End S4: Five devices soak 7 days; no heap leak (freeHeap drift < 10 % of baseline); no unattributed reboots.

---

10. Out of scope

Tracked but not addressed in this design:

• M3 RPC doc/impl drift (covered by separate in-flight plan).
• M4 MQTT QoS1 on feedback path.
• M5 remote reboot/factoryReset RPCs beyond what P4 needs.
• S1 runtime Modbus slave map.
• S3 cross-reboot command-history persistence.
• N10 signed firmware with secure boot v2.

These remain in gateway_firmware/docs/production-readiness-roadmap.md and should be picked up in the next planning cycle after P0 lands.