Multihoming with IPv6… It’s still complicated

I use OpenWrt with mwan3 with a multiple WAN failover policy that works for both IPv4 and IPv6. I’ve had IPv6 connectivity for years through tunnels and native IPv6 options and I found out interesting bits of information about various the aspects around OpenWrt, IPv6 and mwan3 and thought I’d document them to help others who may be looking at doing something similar.

mwan3 and IPv6

There are a few key areas related to mwan3 and IPv6 that you need to be aware of.

1. You need to split your IPv4 and IPv6 interfaces as mwan3 currently doesn’t support dual stack interfaces. This is more verbose and increases the amount of network interfaces and configuration but can still be reasonably manageable by using alias interfaces. For your WAN interfaces, you would want to have something like WAN and WAN6 (which is the default setup). For your second WAN, you’d want WANB and WANB6 and so on. You could also use the naming convention of WAN_6 WANB_6 however you could potentially trigger a known issue with this naming convention, explained here: https://github.com/openwrt/packages/issues/13133 so be careful.
2. Due to important kernels patches that were backported and merged into the OpenWrt source tree, running a build of OpenWrt 19.07.5 and above is recommended given older OpenWrt builds before this potentially can cause problems for mwan3track related to tracking IPv6 interfaces and how it selects source addresses. mwan3 still tries to handle this scenario for older kernels but you are better off with the newer kernel.
3. You are likely going to find yourself needing to use some form of masquerading/NAT in IPv6. This could be NETMAP, NPTv6 or NAT6. Depending on your viewpoint on the whole IPv6 and “should you use NAT or not” debate, I’m going to tell you right now, you are likely going to need to with mwan3 as multihoming with IPv6 has several challenges which make is nearly impossible from a consumer sense without implementing some form of NAT. There I said it.
4. IPv6 support is still a work in progress. There have been great improvements in more recent versions of mwan3 by the package maintainers (particularly the 2.10 branch), but because of the slow IPv6 adoption generally and the relatively low mwan3 usage with IPv6 overall, there are likely various edge cases and scenarios that mwan3 probably doesn’t handle currently.

NAT6 on OpenWrt

NAT6 is one implementation and supported on OpenWrt in an experimental form, because of this a few areas cannot be configured via LuCI, so you must manually edit some of the configuration. The OpenWrt wiki entry on NAT6 explains the steps required.

There is also a workaround for the firewall side, because IPv6 and NAT is currently broken in fw3. This however has been addressed in fw4 but it is not yet widely used.

https://bugs.openwrt.org/index.php?do=details&task_id=500

NAT6 and IPv6 masquerading

NAT6 will essentially route all your outgoing IPv6 traffic from the LAN through the WAN IPv6 address of that interface. You can still have one or more prefixes announced on your LAN, but outbound traffic on the WAN will be seen on the WAN interface not the client address.

Why is NAT needed for IPv6 multihoming right now?

While NAT and IPv6 is a little bit of a controversial talking point, it is something that makes multihoming easier and more workable for general consumer hardware. One method without using NAT is using Provider Independent address space and BGP. The issue with this, is it is likely way out of reach for most consumers and Internet Service Providers that might offer this will likely be only doing it for enterprise or businesses only.

So as a consumer, we have to turn to using NAT within IPv6 to implement a workable multihoming configuration. Having anything NAT with IPv6 is controversial, but in my opinion it does have a place in the IPv6 world when used correctly and for the right reasons.

There is an important distinction with the IPv6 NAT discussion. NAT shouldn’t be implemented in IPv6 to fix or workaround providers that don’t follow RFC standards. Example, if you have a provider that doesn’t provide at least a /64 or a routable prefix, consider changing providers. They are part of the problem by being able to get away with deploying IPv6 like this. Equally, providers that have dynamic prefixes that change regularly, enquire about getting a static prefix. Don’t implement NAT to workaround this, challenge providers that do this and start a conversation.

For the average home user without multihoming, you probably shouldn’t need to use NAT with IPv6 at all. Even with dynamic prefixes you can still implement firewall rules that can handle this without needing NAT. The general design of IPv6 negates having NAT at all in a most cases. I do however disagree with those that NAT should be banished from IPv6 entirely. It shouldn’t. However using NAT with IPv6 needs to be carefully considered and not to fall into the trap of thinking about the IPv4 ways with NAT44 e.g. being seen as security feature (it isn’t).

Why NAT6

Based on my own experiences and testing, NAT6 is one of the few ways to have OpenWrt and mwan3 work with IPv6 traffic in the same way as IPv4 policies, which ultimately is what most users need. Other methods are available like NETMAP, but NAT6 is one of the more easier deployment options. One of the key issues with mwan3 and IPv6 is depending on which IP address is selected by the originator of the traffic, the traffic can only traverse the WAN associated with that prefix. As mentioned earlier, in enterprise there is a solution to this called Provider Independent (PI) Address Space, which is IPv6 address space that isn’t tied to a single ISP, however this is only going to be available on business type contracts and requires BGP, something the average consumer will likely not have the ability to control or implement.

You could technically ignore IPv6 in mwan3 and simply route IPv4 traffic only, but that would be inconsistent and if you want to have a true load balanced or failover, mwan3 needs to be able to control both IPv4 and IPv6 traffic. If you don’t implement some form of NAT, you let the IPv6 routing table and metrics handle what IPv6 route is used for clients, but again you won’t have the same control and many clients are known to be bad at selecting the right source address a lot of the time. Likewise, if a prefix from one ISP is down how would the client know this without something steering the routing?

Inbound services over IPv6

If you happen to be hosting services over IPv6, you can still do this with IPv6 NAT configuration. If you have a stable prefix you can have this advertised across your LAN as normal and implement a standard IPv6 firewall rule.

config rule
    option name 'Allow-HTTP-HTTPS-IPv6'
    list proto 'tcp'
    option dest 'lan'
    option dest_port '80 443'
    option target 'ACCEPT'
    list dest_ip '2001:470:xxxx::x'
    option src 'wan'
    option family 'ipv6'

If you are using ULA addressing only, you would then need to manually add firewall rules that NAT your ULA address to your external WAN IPv6 address with ip6tables. An example could be:

ip6tables -t nat -A prerouting_wan_rule -p tcp -m tcp --dport 51413 -j DNAT --to-destination [fdf1:e8a1:8d3f::23]:51413
ip6tables -t filter -A forwarding_wan_rule -m conntrack --ctstate DNAT -j ACCEPT

This example is some pretend service on TCP 51413. For this to work you need to use Static DHCPv6 leases.

https://openwrt.org/docs/guide-user/network/ipv6/ipv6.nat6#port_forwarding

IPv6 aware mwan3 configuration

Recent versions of mwan3 include a default configuration which supports IPv6, in fact mwan3 will assume rules apply to both IPv4 and IPv6 unless option family 'ipv4' or option family 'ipv6' is explicitly specified on a rule.

This example configuration assumes you have the interfaces wan, wan6, wanb and wanb6. It includes various policies which you might use.

config globals 'globals'
	option mmx_mask '0x3F00'
 
config interface 'wan'
	option enabled '1'
	list track_ip '8.8.4.4'
	list track_ip '8.8.8.8'
	list track_ip '208.67.222.222'
	list track_ip '208.67.220.220'
	option reliability '2'
	option family 'ipv4'
 
config interface 'wan6'
	option enabled '0'
	list track_ip '2001:4860:4860::8844'
	list track_ip '2001:4860:4860::8888'
	list track_ip '2620:0:ccd::2'
	list track_ip '2620:0:ccc::2'
	option reliability '2'
	option family 'ipv6'
 
config interface 'wanb'
	option enabled '0'
	list track_ip '8.8.4.4'
	list track_ip '8.8.8.8'
	list track_ip '208.67.222.222'
	list track_ip '208.67.220.220'
        option reliability '1'
	option family 'ipv4'
 
config interface 'wanb6'
	option enabled '0'
	list track_ip '2001:4860:4860::8844'
	list track_ip '2001:4860:4860::8888'
	list track_ip '2620:0:ccd::2'
	list track_ip '2620:0:ccc::2'
        option reliability '1'
	option family 'ipv6'
 
config member 'wan_m1_w3'
	option interface 'wan'
	option metric '1'
	option weight '3'
 
config member 'wan_m2_w3'
	option interface 'wan'
	option metric '2'
	option weight '3'
 
config member 'wanb_m1_w2'
	option interface 'wanb'
	option metric '1'
	option weight '2'
 
config member 'wanb_m2_w2'
	option interface 'wanb'
	option metric '2'
	option weight '2'
 
config member 'wan6_m1_w3'
	option interface 'wan6'
	option metric '1'
	option weight '3'
 
config member 'wan6_m2_w3'
	option interface 'wan6'
	option metric '2'
	option weight '3'
 
config member 'wanb6_m1_w2'
	option interface 'wanb6'
	option metric '1'
	option weight '2'
 
config member 'wanb6_m2_w2'
	option interface 'wanb6'
	option metric '2'
	option weight '2'
 
config policy 'wan_only'
	list use_member 'wan_m1_w3'
	list use_member 'wan6_m1_w3'
 
config policy 'wanb_only'
	list use_member 'wanb_m1_w2'
	list use_member 'wanb6_m1_w2'
 
config policy 'balanced'
	list use_member 'wan_m1_w3'
	list use_member 'wanb_m1_w2'
	list use_member 'wan6_m1_w3'
	list use_member 'wanb6_m1_w2'
 
config policy 'wan_wanb'
	list use_member 'wan_m1_w3'
	list use_member 'wanb_m2_w2'
	list use_member 'wan6_m1_w3'
	list use_member 'wanb6_m2_w2'
 
config policy 'wanb_wan'
	list use_member 'wan_m2_w3'
	list use_member 'wanb_m1_w2'
	list use_member 'wan6_m2_w3'
	list use_member 'wanb6_m1_w2'
 
config rule 'https'
	option sticky '1'
	option dest_port '443'
	option proto 'tcp'
	option use_policy 'balanced'
 
config rule 'default_rule_v4'
	option dest_ip '0.0.0.0/0'
        option family 'ipv4'
	option use_policy 'balanced'
 
config rule 'default_rule_v6'
	option dest_ip '::/0'
	option family 'ipv6'
        option use_policy 'balanced'

Alternatives to NAT6

NAT6 is potentially breaking for some services (I can think of SIP being one them and there will be many others). One more widely accepted solution is called NPTv6, which allows for IPv6 multihoming without NAT (RFC7157).

RFC 7157 - IPv6 Multihoming without Network Address Translation

There is an interesting video from the UK IPv6 Council a few years ago by Eric Vyncke (Cisco), where the topic is discussed in more detail about IPv6 multihoming.

There’s also been an interesting debate around the IPv6 NAT conversation in another UK IPv6 council video related to IPv6 security, where the topic of NAT in IPv6 and multihoming did come up.

https://youtu.be/a8zefJ_wAbQ?t=2317

Many of the proposals and methodologies discussed are also potentially limited to enterprise networking equipment which means general consumers won’t really have much benefit currently.

Multihoming is still a very widely debated topic in IPv6, there have been various drafts/informational discussions by the IETF but event today the picture isn’t really clear and is likely to evolve over time as the global IPv6 coverage increases.