The excitement never starts

  • Post by JuliaRez
  • Sep 02, 2016

One of the things that I’m doing at work is hacking on our VM-build rig so that is it less terrible to use. [1] Coincidentally, there’s some network-fondling going on that will make it ‘trivial’(tm) to do IPv6 on the servers. So I have this medium-sized chain of VMs and config running on the Xen cluster in the basement and fucking hell has it been a right old laugh to make any of it work, hoo boy let me tell you.

So in order, from a notional left where the will to build a new virtual machine that will do Server! Things! resides in a representative Carbon Unit, to the far right hand side of the process where there is a Server! Doing Things! this is what we find.

Generating VMs using the Xen API is… Not actively unpleasant. It is a mild inconvenience that all the Xen docco assumes that you’re using Python. (Is there a Config Management tool based around Python? I mean, there probably is. Fabric?) However, it’s just an API and the Ruby wrapper insulates one from the worst of the XMLRPC hatefulness.

However. Generating a VM is only about 10% of the job. After that you must get the thing to boot enough of an OS to install something, and fairly obviously automate that install completely so your interaction with the process is some command-line fun, followed a mug of tea later by the monitoring system telling you your new box is live. Beardian have a thing they call ‘preseeding’, which is more or less scripting the sequence of install menus that you get on every Beard-derived distro. No, I do not wish to take part in the popularity contest. I don’t know anyone who does.

We had a working preseed rig, but it was generated from the Squeeze docs, massaged for Wheezy and mostly broke for Jessie. Along with the Xen heads and a set of other customs and practices that were only there because they were there.

I like throwing code away it is great.

The IPv6 bits for preseeding are, er, entirely undocumented. Thus I spent about a week, in between rebuilding obsolete Xen servers, discovering that though the manuals allege that DHCPv6 is supported, it doesn’t work in any useful sort of way.

For instance. When you configure a Beardian box to do DHCPv4, you’ll find a line in its network config that looks like this: ‘iface eth0 inet dhcp’. If you copy that and append ‘6’ to the ‘inet’ bit, then it will allegedly do DHCPv6 as well and you will get a dual-stacked interface. Only not, because the preseed doc is entirely lacking details of the knob you twiddle to enable DHCPv6 such that the installer will write that ‘inet6’ bit into the network config.

DHCPv4 is a fairly ancient protocol that is mostly filled with things the designers thought would be jolly useful at the time. Like this: ‘The Impress server option specifies a list of Imagen Impress servers available to the client.’ There are probably others for specifying the size and shape of the horseless carriage that will be required to route the packets should the primary link fail. The original plan (from what I remember) for IPv6 was that you wouldn’t need any of that sort of rubbish. You’d tell the routers what the first half of the IPv6 allocation was, and the machines connected to same would use their MAC address as the bottom half. Sorted!

Yeah, so about that notion of machines just turning up at random points in an IP allocation larger than the observable universe. How are you going to manage that, and given that most people can’t even be arsed typing ‘’ into a browser, how d’you think typing ‘https://2001:0470:1f15:0fa1:76d4:35ff:fef1:4acd’ is going to work for you?

Thus DHCPv6 and some bit-twiddling. Given you’ve already had to set up Router Advertisment to tell your machines what the first half of their IPv6 allocation looks like, you might as well also use that to advertise the fact that this netblock is Managed. Yes, you set the ‘M’ bit and all the good children will start emitting DHCPv6 queries.

Which is obviously not mentioned in the preseed documentation because, well, why would you even need to know that was a thing? If you were doing a proper v6 rollout you’d have the network infrastructure already working and you wouldn’t need to spend half a day working out what code did the network config, finding same on a Beardian archive server and then tracing through someone else’s wonky C to find the one line that makes you think dark thoughts before going out for quite a long walk to calm down again.

Of course it’s entirely irrelevant because even when you do get RAs working with the M bit, nothing will write the ‘inet6’ bit into the network config.

Meanwhile. A thing you can do with a small subset of server daemons is have them update the DNS when a new machine pitches up. Ordinarily you would never ever do this inside a server netblock ever because that way lies madness. However, doing reverse DNS for v6 involves an awful lot of typing. If you are a bit hazy on how DNS works, it’s an awful lot of typing, followed by a bunch of cargo-culted procedures, kicking the DNS server twice and then going ‘Oh you’ll have to wait 24 hours for it to propagate’

If anyone says that to you about the DNS, they have no idea what they’re talking about and you should ask to speak to a grownup without delay.

Thus having some computers to manage all of that would probably be a really good idea. After all, the dynamic DNS standard has been about for a while. About a decade ago, when I was still at Labs, I ended up running the DHCP rig there. I became very familiar with the contents of Droms & Lemon, which was (is?) The Book About DHCP. Back then, the standard servers were either Windows DHCP or ISC ditto. In the docs for ISC-DHCP, it mentioned that the DDNS update style in use was ‘interim’ because the standard had not been ratified when they were writing that code. Since the BOOTP servers at Labs had only just been turned off, DHCP was viewed with a deal of suspicion since these modern protocols were everywhere and would probably amount to nothing and do you have an up-to-date copy of the hosts table?

I was digging through the doc for the latest version of ISC-DHCP at the beginning of the week. In there I found the DDNS section, where it mentioned that new rollouts should use the ‘standard’ style. It was nice to know that there was now a proper standard RFC way of doing DDNS. Hurrah for a decade of progress!

It didn’t work. I had a look at the client config on one of the Jessie nodes I was using. ‘interim’ style. Well, that was a bit tiresome. Still, it’s just a config change and it is a new standard and everything.

i) Beardian hard codes the DHCP client config into the ‘ifup’ command. ii) I went and looked at the date on the DDNS RFC. Nineteen ninety fucking seven.

I am fact coming to the conclusion that no-one’s really tried any of this yet. Or, if they have, they’ve facebooked it and written the entire stack themselves in a fit of pique.[1]

[1] There’s an Internet Law (or probably a devops/automation law) that goes along the lines of ‘All private cloud automation attempts will tend toward replicating Heroku, only badly’. So we carefully bear that in mind, should the mood to turn it up to eleven strike.