Note: this issue originated from AI-assisted triage of githedgehog/fabricator#1482 (Claude Code), which led to reading through this code path while investigating that CI failure. The conclusions below should be treated as a starting point to verify, not a confirmed finding.
Noticed this while reading through the RIB-to-FIB path. Wanted to flag it and check whether it is already known or already being addressed.
When a route has multiple ECMP nexthops and one of them has a VXLAN encapsulation whose remote MAC (RmacStore lookup) is not yet known, that nexthop's FibGroup is built with a single Drop-action FibEntry rather than being excluded from the route.
VxlanEncapsulation::resolve returns false when the RMAC isn't known (routing/src/rib/rib2fib.rs:189-200). build_pkt_instructions treats that as force-drop rather than skip: instructions = vec![PktInstruction::Drop] (rib2fib.rs:52-69). That instruction becomes a valid FibEntry and is added to the nexthop's FibGroup, since is_valid() accepts a Drop-only entry (rib2fib.rs:107-142).
A route's forwarding set is the concatenation of its members' FibGroups (routing/src/fib/fibgroupstore.rs, FibRoute::from_nhopkeys / get_fibentry), and the ECMP hash picks uniformly across that concatenated set. So a 2-way ECMP route with one resolved member and one member whose RMAC has not arrived yet becomes a 2-entry set where one entry unconditionally drops, meaning roughly half of flows on that route would be dropped rather than the route falling back to its working path.
This would self-heal once the RMAC RPC arrives, since Rmac::add triggers refresh_fibs_by_vni (routing/src/router/cpi.rs:260-274), so any impact would be a brief window rather than a persistent one.
Questions for the dataplane team: is this a known behavior or limitation? Is there a design change already planned or landed after v0.26.0 that touches this ECMP-resolution path? Is the reading above correct, or is there something about the resolution/refresh flow that we are missing that would prevent this from actually happening in practice?
Note: this issue originated from AI-assisted triage of githedgehog/fabricator#1482 (Claude Code), which led to reading through this code path while investigating that CI failure. The conclusions below should be treated as a starting point to verify, not a confirmed finding.
Noticed this while reading through the RIB-to-FIB path. Wanted to flag it and check whether it is already known or already being addressed.
When a route has multiple ECMP nexthops and one of them has a VXLAN encapsulation whose remote MAC (RmacStore lookup) is not yet known, that nexthop's FibGroup is built with a single Drop-action FibEntry rather than being excluded from the route.
VxlanEncapsulation::resolve returns false when the RMAC isn't known (routing/src/rib/rib2fib.rs:189-200). build_pkt_instructions treats that as force-drop rather than skip: instructions = vec![PktInstruction::Drop] (rib2fib.rs:52-69). That instruction becomes a valid FibEntry and is added to the nexthop's FibGroup, since is_valid() accepts a Drop-only entry (rib2fib.rs:107-142).
A route's forwarding set is the concatenation of its members' FibGroups (routing/src/fib/fibgroupstore.rs, FibRoute::from_nhopkeys / get_fibentry), and the ECMP hash picks uniformly across that concatenated set. So a 2-way ECMP route with one resolved member and one member whose RMAC has not arrived yet becomes a 2-entry set where one entry unconditionally drops, meaning roughly half of flows on that route would be dropped rather than the route falling back to its working path.
This would self-heal once the RMAC RPC arrives, since Rmac::add triggers refresh_fibs_by_vni (routing/src/router/cpi.rs:260-274), so any impact would be a brief window rather than a persistent one.
Questions for the dataplane team: is this a known behavior or limitation? Is there a design change already planned or landed after v0.26.0 that touches this ECMP-resolution path? Is the reading above correct, or is there something about the resolution/refresh flow that we are missing that would prevent this from actually happening in practice?