ThingsBoard Connector Plugin - Design

• Date: 2026-07-11
• Status: Approved (design), pending implementation plan
• Scope: Turn ThingsBoard from a core dependency into a pluggable connector, make the HMP platform its own source-of-truth for telemetry, and let each organization connect its own ThingsBoard instance. MQTT connector is explicitly out of scope for this pass but the framework is built to accept it.

1. Background & Problem

Today ThingsBoard (TB) is woven into the core, not pluggable:

• Two independent, uncoordinated TB clients share the same tenant service-account env vars but no code:
  • backend/src/thingsboard/thingsboard-client.service.ts (REST: provisioning, RPC, telemetry read, activity, health).
  • backend/src/monitoring/services/thingsboard-integration-adapter.service.ts (WebSocket + REST: live monitoring stream).
• Device entity carries thingsboard_device_id (unique) and access_token as first-class columns.
• TelemetryService reads latest/history/export directly from TB, so TB is currently the source of truth for time-series. This is the core dependency to break.
• Ingest already exists: POST /telemetry/tb-ingest, authed by a single global x-ingest-token, mapping deviceName == serialNumber.
• Configuration is env-var only and global single-tenant (one TB tenant account for all orgs). No per-org integration config.
• No plugin/adapter/registry abstraction exists; every consumer injects the concrete TB class.
• EMQX/MQTT has been fully removed; only stale cruft remains (Device.mqtt_topic, Device.secret, dead .env.production vars, stale backend/CLAUDE.md doc).

The goal is to support customers already running ThingsBoard to migrate onto HMP: they connect their TB, HMP imports/maps devices, telemetry flows into HMP, and HMP stops depending on TB for reads.

2. Goals & Non-Goals

Goals

• ThingsBoard becomes a pluggable connector behind a provider-neutral interface + registry.
• The HMP platform is the source of truth for telemetry: ingest writes to TimescaleDB; all reads (latest/history/export and the live monitoring stream) come from internal storage.
• Per-organization connection: each org configures its own TB URL + username/password; credentials stored encrypted.
• Auto-discovery: on connect, HMP fetches TB devices, maps existing ones and auto-creates missing ones, then reaches a connected state.
• Bidirectional device sync: import from TB, and continue pushing HMP-created devices to TB via the connector layer.
• The webhook ingests telemetry + device attributes/metadata; alerts/events are generated internally by HMP’s existing automation/alert engine (not pulled from TB).

Non-Goals (this pass)

• MQTT connector (framework must accept it later, but no MQTT implementation now).
• Backfilling historical telemetry from TB on first connect (future enhancement).
• Pulling TB alarms/events into HMP.
• Dynamic/runtime plugin loading or separate plugin packages.

3. Architecture

New module backend/src/integrations/.

3.1 Connector abstraction (interface + lightweight registry)

• DeviceConnector interface (provider-neutral):
  • verifyConnection(config): Promise<void> - authenticate against the provider.
  • discoverDevices(): Promise<RemoteDevice[]> - list remote devices (name, externalId).
  • pushDevice(device): Promise<RemoteDeviceRef> - create a device on the provider.
  • deleteRemoteDevice(externalId): Promise<void>.
  • getRemoteDeviceCredentials(externalId): Promise<RemoteCredentials>.
  • sendRpc(externalId, payload): Promise<RpcResult> - outbound command to the physical device.
• ConnectorRegistry: maps provider string to a connector factory. Registers only 'thingsboard' now; 'mqtt' slots in later without touching consumers.
• ConnectorConnectionManager: keyed by organizationId. Loads the org’s encrypted config, instantiates and caches a config-bound connector instance, and exposes it to consumers. Handles connection/JWT caching and refresh per org.

3.2 ThingsBoard implementation

backend/src/integrations/thingsboard/ contains the TB DeviceConnector implementation. The existing REST logic in thingsboard/thingsboard-client.service.ts is relocated here and adapted to accept per-org runtime config instead of env vars. The old thingsboard/ module is removed once all consumers move to the connector layer.

3.3 Consumer wiring

devices, telemetry, and command dispatch stop importing the concrete TB class. They obtain a connector for the current org via ConnectorConnectionManager. Injection is by interface/token, not concrete type.

4. Data Model

4.1 integration_connection (new, per-org)

Unique constraint on (organization_id, provider).

4.2 device_integration (new, provider-neutral mapping)

Replaces Device.thingsboard_device_id and Device.access_token.

Unique constraint on (provider, external_id) and on (device_id, provider).

4.3 Encryption

Credentials (org login + device access tokens) are encrypted at the application layer using AES with a key from env (INTEGRATION_ENCRYPTION_KEY). During planning, check whether the repo already has a secrets/crypto utility to reuse before adding one.

4.4 Cleanup

• Drop Device.mqtt_topic and Device.secret.
• Remove dead MQTT/EMQX vars from .env.production.
• Fix the stale mqtt/ documentation in backend/CLAUDE.md.

5. Connection Lifecycle (state machine)

States: disconnected → connecting → syncing → connected, with error reachable from connecting/syncing.

• connecting: validate credentials via POST /api/auth/login. Failure → error with last_error.
• syncing: run device discovery & mapping (Section 6). Failure → error.
• connected: setup complete; ingest is flowing.
• Actions: resync (re-run discovery), disconnect (tear down, back to disconnected).
• Background reconnect/JWT refresh while credentials remain valid.

6. Device Discovery & Mapping

On syncing:

1. discoverDevices() fetches the org’s TB device list.
2. For each remote device, match by name == device.serialNumber within the org:
   • Match found → create a device_integration row linking it; store external_id + encrypted credentials.
   • No match → create a new HMP device, then link it.
3. Persist summary counts (device_linked_count, device_created_count) and last_synced_at on integration_connection, then transition to connected.

Idempotent: re-running resync links newly-appeared devices and does not duplicate existing mappings.

7. Ingest Webhook (per-org, generalized)

• New route POST /integrations/thingsboard/ingest. The legacy POST /telemetry/tb-ingest route is kept working (backward compat with rule chains already deployed on customer TB instances).
• Auth: per-org ingest_token (via x-ingest-token), resolving the owning integration_connection and thus the organizationId. Replaces the global token.
• Payload types accepted:
  • Telemetry { deviceName, ts?, values } → resolve device by serialNumber within the org → write to TimescaleDB, update Redis latest-cache, update lastSeenAt/status, emit websocket, run internal automation/alert engine.
  • Device attributes/metadata (e.g. lastActivity, firmware) → sync onto the device.
• Unknown device names are logged and ignored.

8. Telemetry Read Decoupling (largest, highest-risk change)

• getLatest, getHistory, getPondTelemetry, exportTelemetry read from TimescaleDB + Redis instead of calling TB.
• Monitoring live stream: today the monitoring module streams from TB over WebSocket. It moves to the internal realtime pipeline - TelemetryService.save() already emits over the app websocket on ingest, so live monitoring subscribes to that. The TB WebSocket adapter is no longer used for reads.
• The TB connector retains only outbound responsibilities: sendRpc (control commands still travel through TB to the physical device), pushDevice, deleteRemoteDevice.

9. Device Activity / Offline Detection

Remove the 5-minute TB polling cron (DeviceActivityMonitorService.getDeviceActivity). Offline detection is derived from ingest: a device with no telemetry within N minutes is marked offline, complemented by attribute sync from the webhook.

10. Frontend

• Per-org Integration Settings page:
  • Connect form: TB URL, username, password.
  • Status card showing the 5-state progress and, once synced, the summary (X linked / Y created).
  • Resync and Disconnect actions.
• RBAC: new resource integration with integration:read and integration:manage permissions; page gated by integration:manage.
• Follows existing frontend conventions (TanStack Query hooks, Zod schemas, Ant Design, @/ imports).

11. Backward Compatibility & Migration

• Migrate the current env-var TB config into one integration_connection row for the org already using TB.
• Migrate Device.thingsboard_device_id / Device.access_token into device_integration, then drop those columns.
• Keep the legacy tb-ingest route functional so deployed rule chains keep delivering.

12. Testing Strategy (TDD)

• Unit: ConnectorRegistry, ConnectorConnectionManager, device mapping logic, ingest service (per-org token resolution), telemetry reads from internal storage, connection state machine transitions.
• E2E: connect flow (connecting → syncing → connected), per-org webhook ingest and rejection of bad tokens, decoupled reads returning internal data, legacy tb-ingest route still accepted.
• Backend Jest, frontend Vitest. Run via bun run test / Makefile targets, never bun test.

13. Open Details for Planning

• Confirm/choose the app-level crypto utility for credentials_encrypted.
• Offline-detection threshold N (default proposal: align with the current 5-minute activity window).
• Exact monitoring live-stream subscription contract on the app websocket.