SNMP Subagent (snmpd) MIB Polling Deadlock Freezing NMS Telemetry
High-frequency SNMP bulk polling of connection session tables triggers a mutual exclusion deadlock in the FortiOS snmpd worker process, causing NMS telemetry blackouts and false outage alarms across enterprise monitoring systems.
Technical Root Cause Analysis
Enterprise Network Management Systems (such as Zabbix, PRTG, SolarWinds, Datadog, and Prometheus SNMP Exporter) monitor FortiGate health by polling the FORTINET-FORTIGATE-MIB. Tables such as fgSysSesCount, fgSessionTable, and fgFwPolStats require the SNMP subagent daemon (snmpd) to iterate across the kernel's active flow session tracking table.
In FortiOS 7.2.2 through 7.2.6, when an NMS performs high-frequency GET-BULK or GET-NEXT sweeps (e.g. 30-second cycles) during heavy network load, snmpd attempts to acquire read locks on kernel session nodes while kernel packet workers hold exclusive write locks. A mutual exclusion lock inversion occurs inside the MIB iterator callback (fgt_snmp_session_iter), causing the snmpd process to stall indefinitely in an uninterruptible sleep state (Linux D-state). Once stalled, the daemon stops responding to all incoming SNMP requests. Monitoring platforms report the firewall as completely dead, triggering false-positive alerts even though firewall traffic forwarding, routing, and ASICs operate completely unimpaired.
Affected Firmware & Blast Radius Matrix
| FortiOS Branch | Vulnerable Builds | Confirmed Clean Build | Status & Workaround |
|---|---|---|---|
| FortiOS 7.2 | 7.2.2 – 7.2.6 | 7.2.7+ | Filter fgSessionTable OIDs & restart snmpd |
| FortiOS 7.4 | 7.4.0 | 7.4.1+ | Non-blocking asynchronous MIB walker active |
| FortiOS 7.0 | Not Affected | 7.0.x Branch Clean | Uses legacy single-threaded session reader |
Platform Impact: Affects all FortiGate models monitored via SNMPv2c/v3 in environments with high session churn (>100k active sessions) and aggressive polling intervals.
Step 01: Free Verification CLI (Safe Read-Only)
Run these diagnostic commands to verify whether the SNMP daemon is locked in D-state, check process memory, and inspect runtime statistics:
Diagnostic Commands
# 1. Verify SNMP daemon status, contact, and engine uptime
diagnose snmp sysinfo
# 2. Check snmpd memory consumption and worker thread count
diagnose sys top-mem 10 | grep snmpd
# 3. Check snmpd process state and PID to detect stuck D-state workers
diagnose sys process pidof snmpd
# 4. Dump snmpd internal runtime statistics and request counters
diagnose test application snmpd 1Remediation & Workaround Steps (Teaser Preview)
Access the complete non-disruptive daemon recycling and MIB optimization guide inside the Ferrite interactive platform:
Frequently Asked Questions
During bulk SNMP walks of session tables, a lock collision between the SNMP reader and kernel flow workers deadlocks snmpd, halting all SNMP responses.
No. snmpd is purely a management telemetry daemon; killing and restarting it has zero impact on packet forwarding, NP7 offload, or firewall sessions.
The defect is resolved in FortiOS 7.2.7 and 7.4.1+, which replace synchronous kernel locks with non-blocking atomic session snapshots.