I recently needed to have containers egress through a different public IP due to some issues with some archiving of YouTube videos (They don’t seem to want you to do that for some reason).
I’m always looking for an excuse to put on my network administrator hat, so I grabbed some coffee and got to work. What resulted was perhaps the craziest Kubernetes and networking rabbit hole of have been down in a while. I won’t spoil the actual solution (that will be for a future post), but I did learn a neat trick with the NMState operator after some searching and some reverse engineering.
The NMState operator configures networking using NetworkManager on Kubernetes clusters. It is declarative, and allows the configuration of interfaces, sub-interfaces and bridges. This cuts down on management of the actual Linux endpoints, and is all but required for immutable distros like CoreOS (which was my first exposure to the operator).
For example, if I want to configure a VLAN sub-interface and create a bridge interface that is suitable for virtual machines, I can apply the following:
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: example-config
spec:
desiredState:
interfaces:
- name: enp72s0f3.10
description: VLAN 10 on enp72s0f3
type: vlan
state: up
vlan:
id: 10
base-iface: enp72s0f3
- name: br10
description: Linux bridge with enp72s0f3.10 as a port
type: linux-bridge
state: up
ipv4:
enabled: true
dhcp: false
address:
- ip: 10.0.10.10
prefix-length: 24
bridge:
options:
stp:
enabled: false
port:
- name: enp72s0f3.10The above creates a VLAN sub-interface on enp72s0f3 and created a new bridge interface called br10. What results is exactly what we would expect if we created a VLAN interface and bridge interface without all of the pesky nmtui, ansible, or tool-of-choice.
server ~ > ip addr show enp72s0f3.10
444: enp72s0f3.10@enp72s0f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br10 state UP group default qlen 1000
link/ether f4:ce:46:a5:c0:d3 brd ff:ff:ff:ff:ff:ff
Policy-Based Routing
Now, for my initial problem, I needed to send packets down a different route, not based on their destination, but based on their source (or other criteria). This, in essence, is what policy-based routing is for. Linux allows us to create separate routing tables and assign interfaces to them.
Keep in mind that these are separate routing tables, so all of the things one takes for granted when it comes to having them populated on a Linux system does not exist, just take a look at what I need to do on my firewall to create and assign a routing table:
ip route add default via ${WAN1_GW} dev ${WAN1_IF} table lab
ip route add ${CLIENT_NET} dev ${CLIENT_IF} table lab
ip route add ${STORAGE_NET} dev ${STORAGE_IF} table lab
ip route add ${K8SPROD_NET} dev ${K8SPROD_IF} table lab
ip route add ${VMPROD_NET} dev ${VMPROD_IF} table lab
ip route add ${VMLAB_NET} dev ${VMLAB_IF} table lab
ip route add ${CLIENT2_NET} dev ${CLIENT2_IF} table lab
ip route add ${IOT_NET} dev ${IOT_IF} table lab
ip route add ${GUEST_NET} dev ${GUEST_IF} table lab
ip route add ${ADM_NET} dev ${ADM_IF} table lab
ip route add ${VPN_NET} dev ${VPN_IF} table lab
ip route add ${VMLAB2_NET} dev ${VMLAB2_IF} table lab
ip route add ${SECURE_CLIENT_NET} dev ${SECURE_CLIENT_IF} table lab
ip rule add from ${VMLAB_NET} dev ${VMLAB_IF} lookup lab priority 100
ip rule add from ${VMLAB2_NET} dev ${VMLAB2_IF} lookup lab priority 100A routing table is a series of routes, and a set of rules that match what will use the above routing table. The the case of the above, any traffic from the ${VMLAB_NET} and ${VMLAB2_NET} will use the lab routing interface. This allows the lab traffic to exit a different default gateway than the rest of my traffic. With the proper NAT rules, my lab traffic will now leave using a different interface and gateway on the firewall.
Note in the above example from my firewall, I needed to add all of the routes, even if they were directly connected.
NMState Operator and Policy-based routes
We can achieve the same result using the NMstate operator. Admittedly, this has less value on a Kubernetes node, but when combined with Cilium’s egress policies, we can really do some cool stuff.
Here is the example from my NodeNetworkConfigurationPolicy:
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: example-config
spec:
desiredState:
...
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 10.0.10.1
next-hop-interface: br10
table-id: 101
- destination: 10.0.10.0/24
next-hop-interface: br10
table-id: 101
route-rules:
config:
- ip-from: 10.0.10.10/32
route-table: 101The above creates a new route table with an ID of 101 and adds a couple of routes. It also creates a rule matching any traffic that originates from the 10.0.10.10 IP address, which is assigned to an interface.
The net result is the following:
server ~ > ip route show table 101 default via 10.0.10.1 dev br10 proto static 10.0.10.0/24 dev br10 proto static scope link server ~ > ip rule show ... 30000: from 10.0.10.10 iif br10 lookup 101 proto static ... server ~ > curl ifconfig.me 131.191.104.138 server ~ > curl --interface br10 ifconfig.me 131.191.55.228
The path my packets take depends on the source interface they were assigned.
The other valid fields from the api are:
- family
- state
- ip-from
- ip-to
- priority
- route-table
- fwmark
- fwmask
- action
- iif
- suppress-prefix-length
- suppress_prefixlength
I have been fighting with the egress policy, but I figured a quick post to document how to create route rules would not hurt, as I didn’t find much in the way of documentation.
I’m still investigating, so drop me a line if you have an addition or correction.
