FortiGate 50G SP5 Hardware VXLAN Decapsulation Drops on Inner VLAN Frames
We have all been there at 2 AM: you deploy a shiny new FortiGate 50G desktop unit to terminate a multi-tenant VXLAN tunnel from your datacenter spine, ping over the bare overlay works cleanly, but the moment customer traffic carrying an 802.1Q tagged VLAN crosses the tunnel, packets disappear into thin air with zero firewall policy log hits. Here is what is happening inside the SP5 silicon.
Technical Root Cause Analysis
The FortiGate 50G and 51G represent the debut of Fortinet's 7nm Security Processing Unit 5 (SP5 / SoC5). The SP5 integrates multi-core ARM Neoverse compute engines with a unified hardware packet processing pipeline designed to handle line-rate VXLAN encapsulation and decapsulation without host CPU intervention.
When a VXLAN UDP packet arrives on port 4789, the SP5 ingress parser peels the outer Ethernet (14 bytes), outer IPv4 (20 bytes), outer UDP (8 bytes), and the 8-byte VXLAN Network Identifier (VNI) header. In early SP5 microcode released with FortiOS 7.4.2 NPI, the hardware state machine assumed all tenant payloads were standard untagged Ethernet frames, fixing the inner L3 IP header offset at exactly byte 50 from the start of the frame.
However, when enterprise overlays transport 802.1Q tagged tenant traffic, the inner Ethernet frame contains a 4-byte TPID 0x8100 tag. The SP5 parser failed to detect the tag insertion, causing the inner IP header offset to remain at byte 50 rather than adjusting to byte 54. Consequently, the hardware parser read the 0x8100 VLAN tag bytes as the inner IPv4 Version and IHL fields. Because 0x8 and 0x1 do not represent valid IP version/header lengths, the SP5 ingress sanity filter marked the frame corrupt and silently dropped it in hardware before it could reach the internal routing table or kernel bridge.
Firmware Trajectory & Blast Radius Matrix
Understanding the firmware trajectory is crucial for NPI (New Product Introduction) hardware platforms:
| Firmware Build | Release Lifecycle Stage | SP5 Parser Behavior | Production Status |
|---|---|---|---|
| 7.4.2 NPI Build 0231 | Initial 50G Factory Release | Inner 802.1Q frames dropped in ASIC | Critical VXLAN data loss; no logs generated |
| 7.4.3 NPI Build 0418 | Interim NPI Maintenance | Tagged frames punted to CPU (kernel bridge) | Restores ping, but causes 100% CPU spikes under load |
| 7.4.4 GA Build 2456+ | Unified Mainline GA | Full multi-tag hardware parser offload | Permanent resolution; full line-rate wire speed |
| 7.4.5 GA Build 2510+ | Consolidated Long-Term GA | Unified SP5 driver & offload microcode | Recommended baseline for all SP5 enterprise deployments |
Affected Platforms: FortiGate 50G, FortiGate 51G (and early 70G NPI revisions utilizing the initial SP5 microcode rev 1.0.4). Older NP6XLite and NP7 platforms (e.g. FG-60F, FG-100F, FG-200F) use distinct microcode branches and are completely unaffected.
Step 01: Free Verification CLI (Safe Read-Only)
Run these diagnostic commands directly from the FortiGate 50G console or SSH session to safely confirm if your drops are caused by SP5 VXLAN inner parser failure:
Diagnostic Commands
# 1. Query the SP5 NPU hardware drop counters
diagnose npu sp5 drop
# 2. Check active hardware offloaded VXLAN session status
diagnose npu sp5 session-stat
# 3. Inspect the Linux kernel bridge bindings and packet counters
diagnose netlink brctl list
# 4. Check if interface counters report dropped frames on the VXLAN tunnel
diagnose hardware deviceinfo nic vxlan.0 | grep -E "drop|rx_packets|tx_packets"
What to look for: If vxlan_inner_err or ipv4_ver_mismatch increments in diagnose npu sp5 drop whenever customer traffic traverses the tunnel, your unit is running uncalibrated SP5 parser microcode.
Remediation & Workaround Steps (Teaser Preview)
When upgrading immediately to FortiOS 7.4.4+ is not feasible inside a change freeze window, apply these verified mitigations:
Frequently Asked Questions
The Fortinet SP5 (SoC5) network processor contains an ultra-fast hardware parser pipeline. In early 7.4.2 NPI firmware, the microcode assumed all VXLAN tenant payloads were raw untagged Ethernet frames. When an 802.1Q header was present inside the VXLAN payload, the parser offset was off by 4 bytes, reading the 0x8100 EtherType as an invalid IP version nibble, triggering immediate hardware drop counter vxlan_inner_err.
No. This defect is strictly isolated to the SP5 (Security Processing Unit 5 / SoC5) hardware parser microcode introduced on desktop and entry enterprise models like the FortiGate 50G and 51G in the FortiOS 7.4 NPI branch.
You can force the FortiGate to punt VXLAN tunnel termination to the kernel software bridge by setting 'auto-asic-offload disable' inside the firewall policy matching the VXLAN overlay traffic, or disabling npu-vlan-offload on the underlying physical interface.