Route policy applied by nm-cloud-setup results in unexpected behaviour when using NACL's on AWS
When using NACL's within an AWS VPC, it is expected that the NACL will only enforce its rules at subnet boundaries. https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html
The OS route table acquired via DHCP has a local scope of the VPC subnet eg:
default via 10.10.10.1 dev eth0 proto dhcp metric 100
10.10.10.0/24 dev eth0 proto kernel scope link src 10.10.10.181 metric 100
However the route policy applied by nm-cloud-setup and explained in the man page, overrides this with a policy eg:
[ec2-user@ip-10-10-10-181 ~]$ ip rule
0: from all lookup local
30400: from 10.10.10.181 lookup 30400
32766: from all lookup main
32767: from all lookup default
The result of this policy is that all traffic between instances within the same subnet is passed via the default gw and therefore the NACL rules. This will lead to unexpected behavior for anyone who has applied NACL rules with an assumption that traffic flow within the subnet isn't processed by the rule set.
Observed on Red Hat Enterprise Linux release 8.4 (Ootpa), preventing nm-cloud-setup from running or removing the route policy restores expected behavior.