URL Exemption Table Hash Collision Forcing Proxy Worker Heap Growth
Maintaining extensive URL filter exemption tables with over 500 wildcard rules causes degenerate hash collisions in urlfilter, driving proxy worker processes into exponential CPU thrashing and memory conserve mode.
Technical Root Cause Analysis
The FortiOS Web Filter engine relies on the urlfilter daemon to match requested URLs against category rules, blacklists, and exemption tables. When URLs are added as exemptions (action=exempt), the daemon indexes patterns into an internal hash table (hash_tbl) to achieve fast O(1) matching during live proxy traffic.
In FortiOS versions 7.2.1 through 7.2.5, this hash table had a static allocation of only 256 buckets with a simplistic modulus hashing algorithm. In enterprise deployments where security administrators populated the table with over 500 wildcard domain rules (e.g. *.cdn.*.internal or *vendor-api*.com), the hash algorithm produced severe clustering collisions. Over 80% of configured rules fell into a handful of overloaded buckets, degrading lookups to an O(n) linear search. Under high web concurrency, worker threads spent excessive CPU cycles traversing linked list pointers, allocating temporary heap buffers without releasing them, eventually consuming available RAM and triggering FortiOS conserve mode.
Affected Firmware & Blast Radius Matrix
| FortiOS Branch | Vulnerable Builds | Confirmed Clean Build | Status & Workaround |
|---|---|---|---|
| FortiOS 7.2 | 7.2.1 – 7.2.5 | 7.2.6+ | Consolidate wildcard rules / Upgrade |
| FortiOS 7.4 | 7.4.0 | 7.4.1+ | Murmur3 dynamic hash table added |
| FortiOS 7.0 | 7.0.4 – 7.0.12 | 7.0.13+ | Prune exemption table below 300 rules |
Platform Impact: Mid-range enterprise firewalls (FG-100F, FG-200F, FG-400F) supporting large active user populations with extensive URL filtering exemption lists.
Step 01: Free Verification CLI (Safe Read-Only)
Run these diagnostic commands to inspect the urlfilter process memory, check conserve mode status, and review active exemption rules:
Diagnostic Commands
# 1. Query urlfilter daemon memory, hash table stats, and worker health
diagnose test application urlfilter 1
# 2. View top memory consumers to identify urlfilter heap bloat
diagnose sys top-mem 10
# 3. Check current system conserve mode status and thresholds
diagnose hardware sysinfo conserve
# 4. Count the total number of URL filter exemption rules configured
show webfilter urlfilter | grep -c "set url"Remediation & Workaround Steps (Teaser Preview)
Recover available RAM and exit conserve mode without losing web security controls:
Frequently Asked Questions
In FortiOS 7.2.1–7.2.5, the urlfilter subsystem allocated a static 256-bucket hash table for exempt URLs. When configurations exceeded 500 wildcard entries, hash collisions forced hundreds of rules into the same linked list buckets. High traffic volumes forced repetitive O(n) linear traversals, leading to memory fragmentation and unbounded heap expansion in the WAD worker pool.
Executing 'diagnose sys top-mem' will reveal urlfilter and wad worker processes consuming more than 60-70% of total physical RAM, accompanied by Log ID 0100032001 (system entered conserve mode).
Running 'diagnose test application urlfilter 2' flushes the active URL rating and exemption caches, momentarily freeing memory without requiring a full unit reboot.