Update netns post to account for systemd v254

_posts/2020-12-03-poking-pi-ii.md

@@ -3,7 +3,7 @@ layout: post
 title: 'Poking around a Pi: Part II'
 subtitle: 'VPNs and Network Namespaces'
 tags: [tech, linux]
-description: Using a VPN only for a select group of applications
+description: Using a VPN only for a select group of applications (updated August 2023)
 
 ---
 
@@ -94,7 +94,7 @@ What do these commands do? Let's examine them block by block.
     ip link set veth1 up
     ip netns exec default ip link set veth2 up
     ```
-4. Route to the external network via your proper network interface (here, I'm assuming it's IP is 192.168.1.2.) and
+4. Route to the external network via your proper network interface (here, I'm assuming its IP is 192.168.1.2) and
    route back:
 
     ```
@@ -110,7 +110,7 @@ What do these commands do? Let's examine them block by block.
     ip netns exec default iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE
     ```
 
-6. Enable IPv4 forwarding using `sysctl`
+6. Enable IPv4 forwarding using `sysctl`:
 
     ```
     sysctl -w net.ipv4.ip_forward=1
@@ -128,7 +128,7 @@ namespace:
 # /etc/systemd/system/netns-vpn.service
 [Unit]
 Description=VPN network namespace
-StopWhenUnneeded=true
+ConditionPathExists=!/tmp/netns-vpn-concluded
 
 [Service]
 Type=oneshot
@@ -137,22 +137,21 @@ RemainAfterExit=yes
 # Ask systemd to create a network namespace
 PrivateNetwork=yes
 
-ExecStartPre=-/usr/sbin/ip netns delete vpn
-ExecStartPre=/usr/sbin/ip netns add vpn
-ExecStartPre=-/usr/bin/ln -s /proc/1/ns/net /var/run/netns/default
-ExecStartPre=/usr/bin/umount /var/run/netns/vpn
-ExecStartPre=/usr/bin/mount --bind /proc/self/ns/net /var/run/netns/vpn
-ExecStartPre=/usr/sbin/ip link add dev veth1 mtu 1500 type veth peer name veth2 mtu 1500
+ExecStartPre=-/usr/bin/ln -sf /proc/1/ns/net /var/run/netns/default
+ExecStartPre=/usr/bin/ln -sf /proc/self/ns/net /var/run/netns/vpn
+ExecStartPre=/usr/sbin/ip link add dev veth1 mtu 1500 type veth peer name veth2 mtu
 ExecStartPre=/usr/sbin/ip link set dev veth2 netns default
 ExecStartPre=/usr/sbin/ip addr add dev veth1 10.0.0.1/24
 ExecStartPre=/usr/sbin/ip link set veth1 up
-ExecStartPre=/usr/sbin/ip netns exec default ip link set veth2 up
+ExecStartPre=/usr/sbin/ip netns exec default /usr/sbin/ip link set veth2 up
 ExecStartPre=/usr/sbin/ip route add 192.168.1.2/32 dev veth1
 ExecStartPre=/usr/sbin/ip route add default via 192.168.1.2
-ExecStartPre=/usr/sbin/ip netns exec default ip route add 10.0.0.0/24 dev veth2
-ExecStartPre=/usr/sbin/ip netns exec default /usr/sbin/iptables -A FORWARD -i veth2 -o eth0 -j ACCEPT
-ExecStartPre=/usr/sbin/ip netns exec default /usr/sbin/iptables -A FORWARD -o veth2 -i eth0 -j ACCEPT
-ExecStart=/usr/sbin/ip netns exec default /usr/sbin/iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE
+ExecStartPre=/usr/sbin/ip netns exec default /usr/sbin/ip route add 10.0.0.0/24 dev
+ExecStartPre=/usr/sbin/ip netns exec default /usr/sbin/iptables -A FORWARD -i veth2
+ExecStartPre=/usr/sbin/ip netns exec default /usr/sbin/iptables -A FORWARD -o veth2
+ExecStartPre=/usr/sbin/ip netns exec default /usr/sbin/iptables -t nat -A POSTROUTIN
+
+ExecStart=/usr/bin/touch /tmp/netns-vpn-concluded
 {% endhighlight %}
 
 It differs a bit from the code above, because this uses the network namespace that systemd can set up for a service. The
@@ -161,16 +160,17 @@ It differs a bit from the code above, because this uses the network namespace th
 Then use a drop-in to modify the OpenVPN service to use the same namespace:
 
 {% highlight shell linenos %}
-# /etc/systemd/system/openvpn-client@.service.d/override.conf
+==> /etc/systemd/system/openvpn-client@/override.conf <==
 [Service]
-NetworkNamespacePath=/var/run/netns/vpn
+PrivateNetwork=yes
 
 [Unit]
+JoinsNamespaceOf=netns-vpn.service
 Requires=netns-vpn.service
 After=netns-vpn.service
 {% endhighlight %}
 
-This runs all instances of the `openvpn-client` template service in the namespace created by the one-shot service.
+This runs all instances of the `openvpn-client` template service in the namespace created for the one-shot service.
 You can, of course, make a template of the network namespace setup service, and have one for each instance of the
 OpenVPN client.
 
@@ -182,6 +182,52 @@ altogether (and correspondingly have an `ExecStartPre` set up the default route
 service).
 {: .danger}
 
+<!-- section -->
+
+## Update (August 2023): systemd 254 and `PrivateMounts`
+
+An earlier version of this post used bind mounts (`mount --bind /proc/self/ns/net /var/run/netns/vpn`) instead of
+symbolic links (`ln -sf /proc/self/ns/net /var/run/netns/vpn`) to name the private namespace. The `mount` method created
+two complications:
+
+1. The `/var/run/netns/vpn` had to exist (the old unit ran `ip netns add vpn` to create yet another namespace).
+2. Mounts are problematic with `PrivateMounts=yes`.
+
+The latter only became a problem with [systemd's v254 release][systemd-v254]:
+
+>     * PrivateNetwork=yes and NetworkNamespacePath= now imply
+>       PrivateMounts=yes unless PrivateMounts=no is explicitly specified.
+
+Since I run Arch Linux, I get the latest versions of systemd, and one day this setup just started failing, because:
+
+> File system namespaces are set up individually for each process forked off by the service manager. Mounts established
+> in the namespace of the process created by `ExecStartPre=` will hence be cleaned up automatically as soon as that
+> process exits and will not be available to subsequent processes forked off for `ExecStart=` (and similar applies to
+> the various other commands configured for units).
+>
+>  &mdash; _[`man 5 systemd.exec`][systemd.exec]_, `PrivateMounts=`
+
+This meant that while I had a `vpn` netns created, the commands that I was running assuming that it was in the `vpn`
+netns were actually being run in the unit's private namespace. So, when the VPN interface started up, there was nothing
+useful in the netns. No `veth` devices, no route to the internet.
+
+I solved this problem by:
+
+1. Using symbolic links to name the namespace, and
+2. Using `JoinsNamespaceOf` instead of using name for the namespace in the other units.
+
+This actually improved and simplified the setup, in addition to not having to create the `vpn` netns unnecessarily:
+
+1. `JoinsNamespaceOf` directly expresses the relationship between the VPN service and the netns service.
+2. Any other services that should be in the namespace of the VPN service can now use
+   `JoinsNamespaceOf=openvpn-client@<whatever>`, instead of having to join some arbitrary namespace.
+
+This creates a nice tree of namespace relationships. The `vpn` name for the netns now exists only for convenience, for
+when (if) I need to run `ip netns exec` to do something in it.
+
+(Note that each unit that uses `JoinsNamespaceOf` must also have `PrivateNetwork` enabled.)
 
 [tb-linux]: https://www.tunnelbear.com/blog/linux_support/
 [`veth`]: https://man7.org/linux/man-pages/man4/veth.4.html
+[systemd-v254]: https://github.com/systemd/systemd/releases/tag/v254
+[systemd.exec]: https://www.freedesktop.org/software/systemd/man/systemd.exec.html#PrivateMounts=