← incident filesTKT-60009 · MT Jasper · T-1032 · opened 14 Oct 2026, 03:40 UTC

Firmware bug

RMTFix remotelyS1Firmware cohort90% confidence
Customer report
MT Jasper

“Internet cuts out completely for a few minutes every couple of hours, then comes back on its own. Has been happening since yesterday.”

Diagnosis

Firmware 4.2.1 has a memory leak. T-1032 memory climbs from about 36% to about 90% over roughly 1h50m and then the unit does a watchdog reset (reason 7). Each reset drops rx lock for about 5 minutes, which is the customer's outage. The same pattern, with 11-14 reboots in 24h, appears on every 4.2.1 terminal across beams B1-B4. No 4.1.3 terminal shows it.

T-1032T-1004T-1009T-1010T-1012T-1017T-1026T-1034T-1036
Evidence
  1. platform.log: mem 37%→47→54→65→71→80→90% (00:00–01:31), then 01:41:30 [ERR] sys: reset (reason 7) and boot firmware=4.2.1 reset_reason=7
  2. Same pattern again: mem 89% at 03:15, reset at 03:29:30, mem back to 35% at 03:30. Also resets at 22:05:30 and 23:53:30 on 10-13 (mem 93% and 92% just before)
  3. demod.log: 01:45:36 and 03:30:36 'rx lock lost' with tx=off, re-acquired about 5 min later. Pedestal shows state=INIT at the same times. BUC, GPS and pointing are normal.
  4. Fleet: 4.2.1 terminals T-1004/1009/1010/1012/1017/1026/1032/1034/1036 have 11-14 reboots_24h across B1-B4, and T-1010/1012/1036 carry MGMTD_MEM_HIGH. All 4.1.3 terminals have 0 reboots.
  5. show version on T-1032: Firmware 4.2.1, Last reset 7, uptime 0d00h15m
Other findings
T-1033Antenna mispointThe ACU restarted after an ODU power loss at 02:38 and settled at 1.65° pointing error (limit 0.5°) with signal 7%. TX is inhibited and the terminal is offline. Re-acquire or re-peak the antenna remotely, and dispatch if that fails.RMTFix remotely
T-1011Not enough evidence yetT-1011 is offline and unreachable. Its only log is stale (ends 2026-10-11) and it has no modem, antenna or network logs, so the cause can't be determined. Get console or out-of-band data.DATCollect more data
Evidence gaps
  1. The T-1032 router.log was not read in detail. The reset/lock-loss correlation with the platform and demod logs is sufficient, so this is a minor gap.
  2. No mgmtd process-level memory log, so the leaking process is not confirmed. The MGMTD_MEM_HIGH alarms on other 4.2.1 units point to mgmtd.
  3. Logs carry no timezone (UTC assumed). Event ordering was cross-checked across the platform, demod and pedestal logs.
Next steps
  1. Roll back the 4.2.1 cohort to 4.1.3, or push a fixed build. Start with the most affected units and T-1032.
  2. As a stop-gap, schedule a controlled reboot or restart of mgmtd on 4.2.1 terminals about every 90 minutes, before memory reaches roughly 90%, until the rollback is done.
  3. Raise a defect with the firmware vendor or team: mgmtd memory leak in 4.2.1 leading to watchdog reset reason 7.
  4. Tell the customer the outages are a known software issue and a fix is in progress. No technician visit is needed for T-1032.
  5. Re-acquire the T-1033 antenna remotely and dispatch if it does not recover.
Investigation

The investigation trail is sealed. Design partners get the full walkthrough.

Request a walkthrough →