Moving my Server with less than a second downtime

As I’ve written in my previous post, I moved to a new Hetzner Box. This is how I setup my virtual Host and how I moved my server, including all virtual machines, to the new data center with a downtime of a second or less. This setup is only possible if hetzner allows you to move you network via datacenter.

TL;DR

Use host routing to avoid wasting IPs
install centos-release-qemu-ev to install kvm/qemu from SigVirt/OVirt/RHEV
With the newer kvm/qemu you can use virsh migrate --live --copy-storage-all

Don’t waste IP addresses

Because fixed IP addresses are expensive wasting two of the eight IPs to have a broadcast network is not an option.

Virtual Machine setup

We create virtual Interfaces vif0 to vif7 and configure the routing to which we connect the virtual machines on eth0. We let libvirt call a little script on startup of the virtual machines. To do this we add the configurations to the libvirt/quemu xml config of the VMs with virsh edit virt0:

<interface type='ethernet'>
      <mac address='XX:XX:XX:XX:XX:XX'/>
      <script path='/etc/libvirt/scripts/vif-route'/>
      <target dev='vif0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

which calls:

#!/bin/bash
#/etc/libvirt/scripts/vif-route

# $1 is interface

case "$1" in
      vif0)
      ip r a xx.xx.xx.x0 dev $1 &> /dev/null
      ;;
      vif1)
      ip r a xx.xx.xx.x1 dev $1 &> /dev/null
      ;;
      vif2)
      ip r a xx.xx.xx.x2 dev $1 &> /dev/null
      ;;
      vif3)
      ip r a xx.xx.xx.x3 dev $1 &> /dev/null
      ;;
      vif4)
      ip r a xx.xx.xx.x4 dev $1 &> /dev/null
      ;;
      vif5)
      ip r a xx.xx.xx.x5 dev $1 &> /dev/null
      ;;
      vif6)
      ip r a xx.xx.xx.x6 dev $1 &> /dev/null
      ;;
      vif7)
      ip r a xx.xx.xx.x7 dev $1 &> /dev/null
      ;;
esac
echo 1 > /proc/sys/net/ipv4/conf/$1/proxy_arp

exit 0

Setup inside the virtual Machine

Now we configure the network and routing for the virtual machines in /etc/sysconfig/network-scripts/ It should be noted, that the IP address is arbitrary. We need it becouse we cannot add a default route directly to a device, in this case eth0.

/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE="eth0"
BOOTPROTO="static"
HWADDR=XX:XX:XX:XX:XX:XX
IPV6INIT="no"
IPV6_AUTOCONF="no"
NM_CONTROLLED="no"
ONBOOT="yes"
IPADDR=XX.XX.XX.YY
NETMASK=255.255.255.255
TYPE=Ethernet

/etc/sysconfig/network-scripts/route-eth0

XX.XX.XX.YY/ZZ dev eth0
default via XX.XX.XX.YY dev eth0

Moving

Now we want to move the VM to our new server.

The old server, let’s call it Hetzner-Old, is a CentOS 7 with a bridged net/29 network and several virtual machines running on it (e.g. this Blog). The VMs are running on LVMs which themself are running on crypto devices. Each VM has its own LVM group.

The new server, Hetzner-New is also a CentOS 7 machine. The LVM devices have the same name and are of the same size or bigger. They have to have the same names. This is also a good opportunity to make an LVM device bigger without the need to shut down the according VM.

Now we add the repo for SIG Virtualisation on both machines.

yum install centos-release-qemu-ev

and update qemu/kvm etc

Next we open a tunnel between the two machines with

ssh -w 2:2 tun2-IP

and add ipaddesess on both sides (I will use ifconfig because othwise you have to use two commands)

ifconfig tun2 192.168.0.X/24

This way we have encrytion and don’t have to worry encryting it with TLS on qemu.

Server Diagramm

On Hetzner old we have to route through the tunnel tun2

ip r a net/29 via tun2-IP

Next we do

echo 0 | tee /proc/sys/net/ip4/conf/*/rp_filter

That way, when one of the network interfaces of a VM (vif+) gets removed, when the VM is moved, the according route is removed. Otherwise it would try to go through the default GW of Hetzner-New, wich is not allowed by Hetzner yet. Now, when the VM starts on Hetzner-New it has no interface to this IP-Address and it’s routed through the tunnel, through the GW of Hetzner-Old.

On Hetzner-New we add a new routing table with:

echo 201 vm-out >> /etc/iproute2/rt_tables

The number 201 is only internally used, therefore arbitrary.

Next we set a rule for every package where fwmark 1 is set use table vm-out:

ip rul a fwmark 1 table vm-out

Now we set a new default GATEWAY

ip r a default via IP-Hetzner-Old table vm-out

and set a firewall rule (vif+ are the interfaces for the VMs and iptables uses + instead of the commonly used *

iptables -t mangle -s net/29 -A PREROUTING -j MARK --set-mark 1 -i vif+

Now we are ready for moving the VMs from Hetzner-Old to Hetzner-New

virsh migrate --live VIRTUALMACHINE qemu+ssh://IP-HETZNER-Old/system --copy-storage-all --verbose --persistent

Finally we wait for Hetzner to swich the net/29 network over to Hetzner-New and throw away the previously set iptables rule. Since the new machine is in a different data center, Hetzner has to allow this moving. Because for this to work the net/29 network has to be in Hetzners core routers. With this command we wait for the first tcp package to arrive at Hetzner-New and flush our iptables rule.

tcpdump -i br0 net net/29 -s 1 && iptables -F PREROUTING -t mangle