NM handling of multiple ICMPv6 RA RIOs for the same network does not maintain multiple routes, loses potential redundancy
We're running into this with Thread networks and OTBR, but it might be applicable more generally.
The issue occurs on this machine: OS: Home Assistant OS (Buildroot/Docker/others): https://github.com/home-assistant/operating-system uname -a: Linux a0d7b954-ssh 5.15.90 #1 (closed) SMP Tue Jan 24 18:48:18 UTC 2023 x86_64 Linux NM: version 1.40.10
It does not occur on this machine, though the behavior observed might be undesirable: OS: Debian Bookworm uname -a: Linux devm 6.1.0-3-amd64 #1 (closed) SMP PREEMPT_DYNAMIC Debian 6.1.8-1 (2023-01-29) x86_64 GNU/Linux NM: version 1.42.0
What's happening: Thread border routers advertise their Thread network's IP space via ICMPv6 RIOs. Those routes become invalid surprisingly frequently, whether because the Thread network's IPv6 prefix changes, or the border routers lose the link local IPv6 addresses they had previously advertised for those routes. We've been able to use scapy to replicate the following behavior. Our main issue is that NM seems to "lose track" of multiple routes to the same network. This is the general format of the scapy invocation being used:
xIP = <link local IPv6>
xDEV = <eth0/ensXX/etc.>
send(IPv6(src=xIP, dst='FF02::1')/ICMPv6ND_RA(routerlifetime=0,reachabletime=0)/ICMPv6NDOptSrcLLAddr(lladdr=get_if_hwaddr(xDEV))/ICMPv6NDOptRouteInfo(plen=64, rtlifetime=10, prefix='dead::'))
HAOS Machine Behavior
One route being advertised and then allowed to expire behaves as expected.
If a second route is advertised (all scapy parameters the same except for the source IP, and therefore the gateway advertised for the route) the first route is deleted and the new route is inserted.
Something causes NM to lose track of that route, and the new route will never time out.
Re-sending the first route again causes both gateways to be recorded in the kernel's routing table. ip -6 route
shows the following:
dead::/64 proto ra metric 100 pref medium
nexthop via fe80::7089:c1ff:fe0f:5eef dev enp0s18 weight 1
nexthop via fe80::7089:c1ff:fe0f:5ee dev enp0s18 weight 1
Neither will ever time out.
Debian Machine Behavior The Debian machine simply deletes the old route when the new advertisement arrives, and NM never seems to lose track. This is great from the perspective of preventing stale routes, but would mean that all potential redundancy is lost.
Linux itself
I should also note that neither behavior mirrors that of the Linux kernel, which creates and maintains separate routing table entries for each RIO received. Presumably because the "nexthop via" approach used by NM in HAOS can't have an in-kernel lifetime for each nexthop independent of one another. Only the route itself can have that expiry. Linux does have to be configured to even allow/process these RIOs via net.ipv6.conf.XXX.accept_ra_rt_info_max_plen=64
.
An actual RA from an Apple TBR:
NDP payload len 40, from addr: fe80::857:14bc:ddbb:6abc, iface: ens18
Type: RA
Hop limit: 0
Managed address configuration: no
Other configuration: no
Default router preference: medium
Router lifetime: 0s
Reachable time: unspecified
Retransmit time: unspecified
Source linkaddr: 94:ea:32:8c:6b:b6
Route: fd88:fad1:1e70::/64, lifetime: 1800s, preference: medium
One of our scapy RAs:
NDP payload len 48, from addr: fe80::7089:c1ff:fe0f:5eef, iface: ens18
Type: RA
Hop limit: 0
Managed address configuration: no
Other configuration: no
Default router preference: medium
Router lifetime: 0s
Reachable time: unspecified
Retransmit time: unspecified
Source linkaddr: 72:89:c1:0f:05:ee
Route: dead::/64, lifetime: 5s, preference: medium
NM debug/trace logs TBD