All I wanted was to update some SSH keys…
But for reasons that I’ll spare you, it turned out to be a tad more complicated.
Also, in what follows, the context will be simplified for clarity.

As to make the very basic context of this post clear, this is on FreeBSD, and it’s all about IPv4 vs IPv6 address selection. What address should I use to reach xyz.com? Chances are that if you are just mainly using your web browser, that never appeared to be a problem. See, browsers use what is called the Happy Eyeballs algorithm, basically whoever IPv4/IPv6 responds first gets used.

But it’s generally not implemented outside browsers. So that’s why your package manager decides to use either IPv6 or IPv4, but that selection is unreachable, so it’s just stuck there. That’s also why sometimes web apps seem to work perfectly fine in your browser, but on your phone, they seem very sluggish at best, and most of the time just clicking on any button or link just makes the app hang in there forever.

Chances are the app didn’t implement the aforementioned algorithm; it had a route to reach the destination, but for some reason it isn’t reachable. So it tries IPv6 and just has to wait for some timeout. Hence why I guess most people just disable IPv6. But they really shouldn’t. They should slam their sysadmin/netadmin/ISP for doing a bad job. Those are the ones slowing down IPv6 adoption, not the users.

So back to the problem at hand.

At some point, I had to reach over the same destination (read the same fqdn) via SSH from two different hosts. That destination DNS resolution presented both a A (IPv4) record, and a AAAA (IPv6) record. However on the first host (let’s call him host A), IPv6 was preferred to reach the destination (good). However, on the second host (let’s call him host B), IPv4 was preferred instead (bad).

Both hosts received the exact same DNS resolution (with IPv4+IPv6 resolve), both have a globally routable IPv6 and an IPv6 route to the destination. Both hosts can ping6 to the actual destination. In other words, the destination is perfectly reachable via IPv6. Why then would A prefer IPv6 and B prefer IPv4?

See in FreeBSD system configuration file (/etc/rc.conf), there is ip6addrctl_policy that can be configured to ipv6_prefer. As the name suggests, with this configuration and when presented with both a IPv4 and IPv6 to reach the destination, the network should choose the later by default. And it was configured to this value on both host A and host B, so in both case it should have chosen IPv6, but it didn’t.

Under the ip6addrctl_policy setting is the ip6addrctl command that actually configures the address selection policy. Let’s see what it has to say:

$ ip6addrctl show
Prefix                          Prec Label      Use
::1/128                           50     0        0
::/0                              40     1     6131
::ffff:0.0.0.0/96                 35     4        0
2002::/16                         30     2        0
2001::/32                          5     5        0
fc00::/7                           3    13        0
::/96                              1     3        0
fec0::/10                          1    11        0
3ffe::/16                          1    12        0

That is the policy that is configured with ipv6_prefer. The selection goes as follows. Suppose the DNS resolution gave you two candidate IP addresses, and for the sake of our example, let’s suppose it’s 1.2.3.4 and 2001:aaaa:bbbb::1.

For each IP, as with a routing table, the longest prefix (i.e. the most specific match) wins. So for 1.2.3.4, it translates to the IPv4 mapped address ::ffff:1.2.3.4. Hence it selects the line ::ffff:0.0.0.0/96 with precedence 35.

Similarly for 2001:aaaa:bbbb::1, the longest matching prefix is ::/0 hence it is selected with precedence 40. Note that it’s not 2001::/32, which with zero expansion in the network prefix is really 2001:0000::/32.

Between those two:

::ffff:0.0.0.0/96 with precedence 35
::/0              with precedence 40

it will choose the candidate that matched the line with higher precedence, 40 > 35 so the address that matched ::/0 will be retained. So 2001:aaaa:bbbb::1 is selected to reach the destination.

That still doesn’t explain why IPv4 is preferred on host B.

Notice that there are several other lines below ::ffff:0.0.0.0/96. What are those?

• 2002::/16: 6to4 (RFC 3056, deprecated by RFC 7526 in 2015). Embeds a public IPv4 address into an IPv6 prefix to tunnel IPv6 over IPv4 without explicit configuration. Seldom used nowadays.
• 2001::/32: Teredo (RFC 4380). IPv6 tunnelling over IPv4 NAT via UDP encapsulation. Transitional and seldom used nowadays.
• fc00::/7: Unique Local Addresses (ULA) (RFC 4193). The IPv6 counterpart to RFC 1918 private address space. In practice, only fd00::/8 is used. ULA addresses are not globally routable, but are perfectly valid for local routing.
• ::/96: IPv4-compatible addresses (deprecated by RFC 4291). Obsolete dual-stack mechanism embedding an IPv4 address in the low 32 bits. Deprecated in 2006.
• fec0::/10: Site-local addresses (deprecated by RFC 3879 in 2004).
• 3ffe::/16: 6bone test addresses (RFC 2471, deprecated by RFC 3701 in 2006). Address space used by the now-defunct IPv6 testing network. Any traffic on this prefix is misconfigured or relic traffic.

In our cross-site networks, we use local addresses (fd00::/8) a lot. So if one of the candidates is a ULA, it must be preferred over IPv4. This is not the case if you look at the precedence for IPv4 mapped addresses (35) vs ULA addresses (3).

A quick read in /etc/rc.d/ip6addrctl shows that it will load a custom IP selection policy from /etc/ip6addrctl.conf when in /etc/rc.conf you have ip6addrctl_policy=AUTO and the configuration file is present, readable, and non-empty.

Hence the new configuration (note: ::ffff:0.0.0.0/96 and ::ffff:0:0/96 are equivalent notations for the same prefix):

::1/128		 50	 0
::/0		 40	 1
::ffff:0:0/96	 35	 4
2002::/16	 30	 2
2001::/32	 5	 5
fc00::/7	 37	13
::/96		 1	 3
fec0::/10	 1	11
3ffe::/16	 1	12

But that would only allow preferring ULA over IPv4 mapped addresses. In our case the destination address, 2001:aaaa:bbbb::1 is globally routable and totally not a ULA, so the default configuration should work. Or did we miss something?

The astute reader might have noticed there is also a Label column. Also, the more knowledgeable reader might point out that all this IPv6 selection mechanism is described by RFC 6724. Here is what it has to say about the policy table, precedence, label, and selection algorithm:

   The policy table is a longest-matching-prefix lookup table, much like
   a routing table.  Given an address A, a lookup in the policy table
   produces two values: a precedence value denoted Precedence(A) and a
   classification or label denoted Label(A).

   The precedence value Precedence(A) is used for sorting destination
   addresses.  If Precedence(A) > Precedence(B), we say that address A
   has higher precedence than address B, meaning that our algorithm will
   prefer to sort destination address A before destination address B.

   The label value Label(A) allows for policies that prefer a particular
   source address prefix for use with a destination address prefix.  The
   algorithms prefer to use a source address S with a destination
   address D if Label(S) = Label(D).

and again with more details in Section 6. Destination Address Selection:

   Rule 5: Prefer matching label.
   If Label(Source(DA)) = Label(DA) and Label(Source(DB)) <> Label(DB),
   then prefer DA.  Similarly, if Label(Source(DA)) <> Label(DA) and
   Label(Source(DB)) = Label(DB), then prefer DB.

   Rule 6: Prefer higher precedence.
   If Precedence(DA) > Precedence(DB), then prefer DA.  Similarly, if
   Precedence(DA) < Precedence(DB), then prefer DB.

In our case, it's Rule 5 that was causing IPv4 to be preferred. See, on host B, the route table states that in order to reach 2001:aaaa:bbbb::1, you must go via a specific interface. That interface only has one ULA configured, so it is selected as the source address to reach that destination.

At the input of the destination address selection algorithm, we have those two candidates:

# source-address     destination-address

## IPv4 mapped candidate (from the A record)
::ffff:192.168.1.1   ::ffff:1.2.3.4

## IPv6 candidate (from the AAAA record)
fd08:aaaa:bbbb::1    2001:aaaa:bbbb::1

For the IPv4 mapped candidate, the source and destination label match (4). For the IPv6 candidate, the source and destination label don't match (13 != 1). Hence, by rule 5, the first candidate is preferred.

So the solution is to ensure both the ULA and globally routable lines of the policy share the same label:

::1/128		 50	 0
::/0		 40	 1
::ffff:0:0/96	 35	 4
2002::/16	 30	 2
2001::/32	 5	 5
fc00::/7	 37	 1
::/96		 1	 3
fec0::/10	 1	11
3ffe::/16	 1	12

Note that the problem of ULA being disfavored is explicitly acknowledged in Section 10.6. Configuring ULA Preference of the RFC. I quote:

   [...] By default, global IPv6 destinations are preferred over
   ULA destinations, since an arbitrary ULA is not necessarily
   reachable.

   [...]

   However, a site-specific policy entry can be used to cause ULAs
   within a site to be preferred over global addresses [...].

When you work with ULAs and globally routable addresses in cross-site networks, the prefixes used are generally known in advance and static. The recommended way is to add dedicated policies for those prefixes with higher precedence and ensure that the label matches if those ULAs can also be used as source addresses to reach your own globally routable IPs:

# custom policies for our network
fd08:aaaa:bbbb::/48  42	 14
2001:aaaa:bbbb::/64  41  14
2001:aaaa:cccc::/64  41  14

# default automatic policy with IPv6 prefer
::1/128		     50	 0
::/0		     40	 1
::ffff:0:0/96	     35	 4
2002::/16	     30	 2
2001::/32	     5	 5
fc00::/7	     3	13
::/96		     1	 3
fec0::/10	     1	11
3ffe::/16	     1	12

Here, our ULA prefix and our globally routable prefixes are all assigned the same label (14), ensuring that rule 5 (prefer matching label) never penalizes a ULA source address when reaching one of our own globally routable destinations, and vice versa. They are also given higher precedence than the default ::/0 and ::ffff:0.0.0.0/96 entries, so our known prefixes are always preferred over the generic fallback behavior. For any address outside these explicitly listed prefixes, the default policy applies unchanged.

This was for FreeBSD, though. What about Linux, I hear you ask? There, this is configured in /etc/gai.conf. The syntax changes slightly, but surely with the explanation above you will figure it out.