kota-technician-tool/docs/firmware-diagnostics-proposals.md

Firmware diagnostics improvement proposals

Status: draft proposal, AI-generated (Claude) for review by the firmware team. Nothing here is implemented. The firmware source was only read for analysis; no firmware was changed. Every item links to an observation from bench testing box A4:CB:8F:C9:46:F8 (firmware v1.0, build 45fa9d5b9) or from the sample logs, so the value of each change can be judged.

Why

Troubleshooting currently depends on reverse-engineering free-text log lines and on a status LED whose states overlap. Observed problems:

#	Observation	Effect for technicians
O1	A new, uncommissioned box shows green for about 60 s, then purple	Looks healthy during installation
O2	Three different states share white (SD error, factory reset, setup portal)	Needs serial or guesswork
O3	Blue means "RTC did not answer", but the RTC is just the first I2C device	A loose I2C cable is misreported as an RTC fault
O4	Every build reports App version: 1.0	Builds are told apart only by ELF SHA
O5	Restarts after 150 s in an error state happen without any log line	A self-restart looks like a crash
O6	With the RTC not running and no Internet, sensor tasks loop forever on RTC is uninitialized	An offline box records nothing, and it isn't obvious why
O7	The OTA token is printed in plain text (OTA token loaded from NVS: %s, received token: %s)	Secrets end up in technician logs
O8	Native USB loses output while no host is reading; a stale FIFO fragment is glued to the next line	The early boot after a power cycle is lost, and lines get corrupted
O9	The microphone reports "shutdown state" (GPIO3) with no hint of the cause	A technician can't tell a switch from a fault
O10	The deployed build logs some messages under different tags than the source (e.g. SD mount under INFLUX_BUFFER)	Parsers must ignore tags

Serial output

Priority: P1 = large gain for little effort, P2 = valuable, P3 = nice to have.

P1

1. One structured boot line with a fixed prefix, printed right after app_init:

   KOTA: boot fw=1.0.15 build=45fa9d5b hw=2 mac=A4CB8FC946F8 reset=USB_UART_CHIP_RESET mode=matter commissioned=0 last_restart=status_watchdog
   

   The tool can then identify firmware, hardware and mode from a single line (fixes O4 and O10). Fill PROJECT_VER from FIRMWARE_VERSION (currently 15 in pinout.hpp) so every build has a unique version.
2. Log every status change inside Status::set_status(), with a reason argument:

   KOTA: status OK -> CONNECTION_ERROR reason=influx_flush_failed
   

   This is the exact LED colour. Today the tool has to infer it from about 15 different messages.
3. Log the reason before every esp_restart() and store it (RTC memory or NVS) so it can be printed at the next boot as last_restart=. Values: status_watchdog_150s, out_of_resources_30s, button_factory_reset, setup_portal_toggle, ota_done, init_failed:<step> (fixes O5).
4. Stop printing secrets: print only the first and last 4 characters of the OTA token, or token=present (fixes O7).
5. Warn once and continue when the time is unset, instead of the while (now == 946684800) loops in device_status.cpp and the sensors. Record data with a time_valid=false field or buffer it with uptime, then correct it after SNTP (fixes O6). This is a behaviour change, but the biggest data-loss risk found.

P2

6. I2C bus scan at boot, before any sensor init:

   KOTA: i2c SDA=1 SCL=1 0x39=AS7341 0x68=DS1307 0x6B=SEN66(missing) 0x76=BME688
   

   Reading the SDA/SCL levels first detects a stuck-low bus (shorted or broken cable). The scan then separates "whole cable" from "one sensor" (fixes O3).
7. RTC detail: at init, print the DS1307 clock-halt (CH) bit and the raw time (KOTA: rtc running=0 time=2000-01-01 (coin cell?)). After SNTP sets the time, re-read it 2 s later to verify it's ticking.
8. SD card detail: card type, size, FAT type, free space, which ENV.TXT keys are present (names only, never values) and the buffer size in lines. Example: KOTA: sd type=SDHC size=7.4GB fs=FAT32 free=7.3GB env_keys=INFLUX_URL,INFLUX_TOKEN buffer_lines=42.
9. Periodic health line every 60 s. DeviceStatus already computes almost everything:

   KOTA: health up=312s status=CONNECTION_ERROR wifi=disconnected rssi=0 time_valid=0 sd=ok buffer=42 heap=81234 psram=6.1M last_upload=never ota=unregistered
   

10. Matter commissioning status at boot: KOTA: matter fabrics=0 window=open plus the QR payload and manual code, so the tool can show the QR code on screen when the sticker is missing.
11. Microphone enable reason: KOTA: mic enabled=0 (GPIO3=1, expected 0) once at boot, instead of a warning every 10 s (fixes O9).
12. Stable error IDs in messages, e.g. [E-SD-01] Failed to initialize the card, documented in one table shared by the firmware, the technician tool and the service docs.

P3

13. Console commands over USB (the firmware already runs a console task): diag (full health dump), i2cscan, sdinfo, ledtest (cycles all colours so the technician can confirm the LED hardware), version, bootlog. The tool could then run checks without a reboot.
14. Boot-log ring buffer in RAM, dumped with bootlog or on first host connect. This recovers the early boot lost on native USB after a power-switch cycle (fixes O8).
15. Button events: log the press start and release with duration (KOTA: button press 1.2s -> setup_portal). This allows a safe button test and shows accidental factory resets.

RGB LED

1. Separate "not connected yet" from "OK" (fixes O1): cyan breathing while the box is uncommissioned / advertising over BLE, fast cyan while a phone is connected, and a green flash when commissioning completes.
2. Split the white states with patterns (fixes O2):
   • SD card / ENV.TXT error: white, fast blink (2 Hz)
   • factory reset in progress: white, solid
   • Wi-Fi setup portal: white, slow breathing
3. Blink codes for red and blue (fixes O3): N short pulses followed by a pause identify the part.
   • Red: 1 = BME688, 2 = SEN66, 3 = AS7341, 4 = microphone, 5 = flicker, 6 = self-check failed
   • Blue: 1 = RTC not responding, 2 = I2C bus stuck low
4. Split purple: solid = no network/DNS; blinking = Internet works but the server rejected the box (InfluxDB 401/404, OTA 401). These need different fixes (network vs SD card settings).
5. Show the cause before a self-restart: flash the error colour 3 times before esp_restart(). After boot, show the previous boot's error colour for 3 s (from RTC memory), so a reboot loop has a visible cause.
6. Update progress: yellow blinking while downloading (faster as progress grows); orange stays solid for the self-check.
7. Brightness and distinguishability: colours are set at 10 % (255/10). Yellow and orange, and white and cyan, are hard to tell apart in daylight. Prefer fewer colours plus patterns, and raise brightness for error states.
8. Heartbeat LED (GPIO6): keep 1 Hz for "running", add a double-blink when offline, so the heartbeat alone gives a rough state even when the RGB LED is faulty.

What changes in the technician tool

• The structured KOTA: lines (items 1, 2, 3, 9) would replace most regex rules in rules.toml and make the LED model exact rather than inferred.
• The console commands (item 13) would let guided checks test hardware directly (I2C scan, LED test) instead of asking the technician to power-cycle.
• The new LED patterns would add answers to the Quick check (e.g. "white, blinking fast") and remove the follow-up questions currently needed to separate the white states.