ZTNA HTTPS Access Proxy Multiplexing Deadlock on Heavy HTTP/2 Browser Sessions
We have all been there at 2 AM: remote staff complain that internal web apps like Jira, GitLab, or Salesforce dashboards spin endlessly on page load, yet ping works and the TLS handshake succeeds. Under high concurrency HTTP/2 multiplexing, FortiOS wad reverse proxy workers enter a circular mutex deadlock, freezing user streams without dropping the outer TCP connection.
Technical Root Cause Analysis
When modern web browsers connect to enterprise applications published through a FortiGate Zero Trust Network Access (ZTNA) HTTPS Access Proxy, they negotiate HTTP/2 via ALPN. Because HTTP/2 multiplexes dozens of concurrent data streams (JavaScript bundles, CSS, REST API calls, WebSockets) across a single underlying TCP transport, the proxy must dynamically schedule frame transfers to back-end real servers.
In affected builds, the WAD reverse proxy engine maintains a per-session stream context queue alongside a shared pool of TCP worker connections to internal application servers. Under heavy concurrent page requests, if the browser issues an HTTP/2 RST_STREAM frame (for instance, when a user cancels a query or navigates away) at the exact millisecond the back-end worker generates a WINDOW_UPDATE frame, a mutex lock inversion occurs between h2_stream_lock and backend_conn_lock.
The thread enters an unrecoverable spin-wait inside wad_ztna_h2_stream_schedule. Because the master process considers the worker healthy (it is not crashing or receiving SIGSEGV), no automatic restart triggers. Meanwhile, every client stream sharing that WAD worker stalls indefinitely until the browser encounters an HTTP 504 Gateway Timeout.
Firmware Trajectory: Discovered in firmware 7.2.2, an initial fix was attempted in firmware 7.2.4 using a stream reset patch designed to forcibly terminate frozen streams. However, the 7.2.4 patch left orphaned backend sockets, causing socket descriptor leaks and premature conserve mode. Permanent verified resolution was achieved in firmware 7.2.7+ and 7.4.2+ through a complete redesign of stream synchronization using lockless circular ring queues.
Affected Firmware & Blast Radius Matrix
| FortiOS Branch | Vulnerable Builds | Confirmed Clean Build | Status & Workaround |
|---|---|---|---|
| FortiOS 7.2 | 7.2.2 – 7.2.6 | 7.2.7+ | Disable HTTP/2 in profile-protocol-options |
| FortiOS 7.4 | 7.4.0 – 7.4.1 | 7.4.2+ | Upgrade to 7.4.2+ or force HTTP/1.1 fallback |
| FortiOS 7.0 | 7.0.x Branch Clean | Not Affected | 7.0 ZTNA uses legacy single-stream proxy architecture |
Platform Impact: All FortiGate physical appliances (FG-60F up to FG-3000F) and FortiGate-VM instances configured with ZTNA HTTPS Access Proxies (Virtual IP server-type ztna-access-proxy) handling modern browser traffic with HTTP/2 enabled.
Step 01: Free Verification CLI (Safe Read-Only)
Execute these read-only diagnostic commands on your active FortiGate CLI to check for stuck WAD workers, examine ZTNA client sessions, and identify stalled stream queues without affecting user traffic:
Diagnostic Commands
# 1. Query WAD worker thread status and stalled session counters
diagnose test application wad 1000
# 2. Inspect active ZTNA tags and connected endpoint proxy sessions
diagnose wad ztna-tag list
# 3. Check memory and CPU consumption across active WAD worker processes
diagnose sys top-mem 10
# 4. View active WAD process IDs to identify runaway worker threads
diagnose sys process pidof wad
# 5. Check firewall session statistics for TCP proxy backlog spikes
diagnose sys session statRemediation & Workaround Steps (Teaser Preview)
The Ferrite interactive platform provides step-by-step automated remediation for live environments:
Frequently Asked Questions
Bug 0887102 occurs when high-concurrency browsers issue multiplexed HTTP/2 streams through a ZTNA HTTPS reverse proxy. Concurrent WINDOW_UPDATE and RST_STREAM events cause a mutex lock inversion between the stream queue and back-end socket thread, stalling workers.
Discovered in firmware 7.2.2, an initial fix was attempted in firmware 7.2.4 using an aggressive stream reset patch that unfortunately caused orphaned backend sockets. Permanent verified resolution arrived in firmware 7.2.7+ and 7.4.2+ with lockless ring queue stream synchronization.
Yes. Disabling HTTP/2 in profile-protocol-options forces clients to negotiate HTTP/1.1 via ALPN on next request. This completely avoids the deadlock path without dropping existing state or requiring a reboot.