Ferritaas Incident Intelligence
Defect Intelligence Report Bug 0883102 Severity: Critical (P1)

FortiLink MCLAG Split-Brain & Dual-Active Core on FortiSwitch 400/500

During transient Inter-Chassis Link (ICL) flaps, fortilinkd and the mclag_sync daemon fail to enforce active-standby tiebreakers, causing both FortiSwitch core units to enter dual-active forwarding and triggering severe campus-wide Layer 2 loops and MAC address thrashing.

Technical Root Cause Analysis

In multi-chassis link aggregation (MCLAG) deployments with FortiLink, two FortiSwitch units (such as FS-424E, FS-448E, or FS-548D) form an active-active Layer 2 cluster interconnected by an Inter-Chassis Link (ICL) and an Inter-Chassis Control Link (ICCL). Coordination between peer switches is governed by fortilinkd on the FortiGate and the local mclag_sync daemon on each switch.

When optical transceiver jitter, cable reseating, or high interface utilization causes a micro-outage on the ICL, heartbeat keepalive frames are dropped. Under FortiOS 7.2.3, the failure detection state machine instantly promotes the standby core switch to active forwarder without verifying whether peer traffic is still reaching the managing FortiGate. Both switches simultaneously broadcast gratuitous ARPs for downstream gateway interfaces and un-block local LAG member ports, creating a catastrophic Layer 2 forwarding loop.

[mclag_sync 1402] ERROR: icl_heartbeat_timeout: peer switch FS548D4619000182 silent for 3000ms [mclag_sync 1402] STATE CHANGE: peer state PEER_DOWN -> transition to DUAL_ACTIVE_DETECTED [mclag_sync 1402] CRITICAL: peer tiebreaker unresolved; keeping local MCLAG trunks UN-BLOCKED [switch_mgr 849] WARN: mac flapping detected: 00:50:56:b2:44:a1 between port24 (ICL) and port1 (Trunk-Agg1) [kernel] switch fabric packet drop: buffer ring saturation on port24 tx queue

The 3-Stage Firmware Evolution:

  • Discovered in FortiOS 7.2.3 / FortiSwitchOS 7.2.3: Rapid ICL drops immediately triggered dual-active split-brain due to an aggressive 3-second hard timeout without out-of-band arbitration.
  • Initial Fix Attempted in FortiOS 7.2.4: Fortinet extended the peer-sync timeout to 9 seconds. While this mitigated false triggers from micro-bursts, rapid link bouncing still caused packet loss in the ring buffer, leading to split-brain during recovery.
  • Permanent Verified Resolution in FortiOS 7.2.6+ & 7.4.1+: Introduced an atomic dual-active detection protocol with FortiGate out-of-band arbitration over the native FortiLink control plane, immediately isolating the standby switch downlinks during true ICL isolation.

Affected Firmware & Blast Radius Matrix

Branch Firmware Build MCLAG Split-Brain Status Recommended Action
FortiOS 7.2 7.2.3 (Build 1262) Vulnerable — Instant Dual Active on ICL Jitter Upgrade to 7.2.6+ or apply manual trunk standby lock
FortiOS 7.2 7.2.4 – 7.2.5 Partial Fix — Extended Timeout, Flap Race Condition Apply FortiLink arbitration patch; upgrade to 7.2.6+
FortiOS 7.2 7.2.6+ GA (Build 1575) Clean — Atomic Arbitration Protocol Active Stable release for campus core deployments
FortiOS 7.4 7.4.1+ GA (Build 2463) Clean — Native OOB Arbitration Engine Verified clean across 400/500 series FortiSwitch

Hardware Blast Radius: FortiGate 100F, 200F, 400F, 600F managing pairs of FortiSwitch 424E, 448E, 524D, 548D, and 1024D/1048E switches operating in multi-chassis LAG topologies.

Step 01: Free Verification CLI (Safe Read-Only Diagnostic)

Run these diagnostic commands directly from the FortiGate management CLI or the FortiSwitch console to determine if MCLAG split-brain or peer synchronization divergence is actively threatening your campus core:

# 1. Check MCLAG peer status, tiebreaker election, and ICL health diagnose switch-controller dump mclag peer # 2. Inspect active multi-chassis LAG member port forwarding states diagnose switch-controller dump mclag trunk # 3. Verify FortiLink switch synchronization status across the fabric execute switch-controller get-sync-status # 4. Check for MAC address flapping across the ICL and local access ports diagnose switch-controller switch-info mac-table | grep -i "flap"

Diagnostic Tip: If diagnose switch-controller dump mclag peer displays peer-state: down while both physical switches show all local downlink ports in the Forwarding state, your network is in an active split-brain state.

Interactive Runbook & TAC Remediation Suite

Automate your campus core recovery with Ferrite's zero-downtime runbook scripts, pre-flight firmware validators, and automated TAC escalation dossiers.

Live MCLAG Auto-Healing Temporarily isolates standby ports without dropping management sessions.
Pre-Upgrade Quorum Validator Audits ICL SerDes error rates before firmware reloads.

Frequently Asked Questions

What causes FortiLink MCLAG cores to enter a dual-active split-brain state?

When the Inter-Chassis Link (ICL) trunk suffers transient packet loss or SerDes flapping, the mclag_sync daemon times out peer heartbeats. Under FortiOS 7.2.3–7.2.5, the secondary core fails to honor the active tiebreaker and un-shuts its downstream member ports, resulting in both cores actively forwarding traffic for identical virtual MAC addresses.

Why did the initial firmware patch in FortiOS 7.2.4 fail to fully resolve Bug 0883102?

The 7.2.4 build increased peer-sync keepalive timeouts from 3 seconds to 9 seconds, which reduced false positives on high-utilization uplinks but did not introduce an atomic quorum or out-of-band arbitration mechanism. If the ICL bounced rapidly, peer-sync packets were dropped in the ring buffer, triggering split-brain upon link recovery. FortiOS 7.2.6+ resolved this with atomic split-brain detection over FortiLink control packets.

Can I resolve an MCLAG split-brain without rebooting the campus core switches?

Yes. By administratively shutting down downstream MCLAG member trunks on the designated secondary core, Layer 2 broadcast storms are instantly stopped. Once the ICL stabilizes, re-enabling peer synchronization restores seamless multi-chassis LAG operation without packet loss.