You may have seen an earlier guide: Workaround for instances with networking bug on Ubuntu 18.04 in relation to a networking bug on 18.04 causing intermittent connectivity issues. This guide is similar but an updated version that allows the use of the true private networking by disabling Netplan altogether. This guide walks you through the steps required in order to replace Netplan with the older ifupdown package on Ubuntu Bionic Beaver (18.04).

Before you start

If you’re not already a Civo user and want to try this out safely in the cloud first. Civo offers $70 free credit to new users if accepted into the KUBE100 beta: Apply here. This will be enough to run a Medium-sized 3 node cluster.

Process

IMPORTANT: IF YOU HAVE CONFIGURED THE VM TO BE ACCESSIBLE VIA SSH KEY ONLY (NO PASSWORD ACCESS), IF YOU MAKE A MISTAKE IN THE CONFIGURATION THAT CAUSES AN ISSUE WITH THE NETWORKING, WE WILL BE UNABLE TO ACCESS THE MACHINE... PERMANENTLY. WHILST SETTING PASSWORDS FOR SSH IS OFTEN INADVISABLE, IN THIS CASE IT IS HIGHLY RECOMMENDED IN ORDER TO ENSURE ACCESS TO THE VM VIA THE CONSOLE, SHOULD A MISCONFIGURATION OCCUR. PLEASE ENSURE YOU FOLLOW THE STEPS CAREFULLY AND CHECK / TEST WHERE INDICATED TO MAKE SURE YOU DO NOT LOSE YOUR DATA.

If you would like us to do this for you, please get in touch and we will run through these steps. We will however need access to the machine in order to do so.

If at all possible, we would recommend one or more of the following before applying the workaround:

  • Backup your data.
  • Migrate data onto another Civo instance (not using 18.04).
  • Mount a volume to the instance following this guide and move your data onto it.
  • Snapshot the VM.

Step 1 - Recovering your machine

If you have not rebooted your machine and / or still have SSH access, you can skip this step and move onto Step 2.

If you have already rebooted and are no longer able to access the VM continue reading Step 1 (this step).

If you set up your machine with a password, you should be good to move onto Step 4. If you only have SSH key access to the machine, get in touch with us and we will try and update your NAT address to point to the other interface therefore allowing access. Once you have done this, you can move onto Step 3.

Step 2 - Find whether you created your machine using a key or a password

There are two paths you can take, depending on whether your instance was configured to use an SSH key at creation or a password. If you created your instance with an SSH key, move onto Step 3. If you created your instance with a password move onto Step 4.

Step 3 - Create a temporary user and password

YOU DO NOT HAVE TO PERFORM THIS STEP. HOWEVER, IF YOU MAKE A MISTAKE IN THE NETWORK CONFIGURATION THAT PREVENTS SSH ACCESS, YOU WILL NOT BE ABLE TO RECOVER THE MACHINE. EVER.

Log into your instance via SSH. Then create a new user

sudo adduser civorescue

When prompted, set a password for the user. Ensure you set a strong password:

  • Use a combination of letters (upper and lower case), numbers and symbols.
  • It should be at least 8 characters long (if not longer).
  • Avoid using a single dictionary word like "workaround", even if you replace letters with numbers or symbols such as "W0rk4r0und".
  • Ensure that you keep it somewhere safe. A password manager like 1Password, LastPass, KeePassX etc is ideal.
  • Ensure you use a password that you have not used elsewhere in case another site is compromised and malicious users gain access to your password.
  • Even if you do not wish to keep the password after setting it in this guide, you must still ALWAYS use strong passwords.
Enter new UNIX password:
Retype new UNIX password:

For each of the following, hit return:

Enter the new value, or press ENTER for the default
    Full Name []:
    Room Number []:
    Work Phone []:
    Home Phone []:
    Other []:

Finally select Y when asked if the information is correct:

Is the information correct? [Y/n] Y

Add the user to the sudo group

sudo usermod -aG sudo civorescue

Now move onto Step 4.

Step 4 - Ensure you have console access

This step is important. Do not assume console access works. It is well worth the effort to confirm you have console access now, rather than be sorry later.

Navigate to the instance in the Civo UI and locate the console button (computer screen icon) from the action buttons on the top right of the instance. This will launch a spice terminal that will prompt you for a username and password.

If successful, also confirm that your user has sudo access by running:

sudo su

This should prompt you for your password and then switch to the root user. Exit the root user with:

exit

If this all worked, then we are good to move onto Step 5.

Step 5 - Find the address of the private network interface

Find the IP address of the ens3 interface:

ifconfig ens3

This should output something similar to:

ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.0.0.2  netmask 255.255.255.0  broadcast 10.0.0.255
        ether 00:00:00:00:00:00  txqueuelen 1000  (Ethernet)
        RX packets 5  bytes 1224 (1.2 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14  bytes 1564 (1.5 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The part you are interested in is inet 10.0.0.2, which is the address you need for the next step. In this case it is 10.0.0.2. Yours will likely be different but will be an address from the 10.0.0.0/8 range.

Move onto Step 6

Step 6 - Remove the incorrect gateway and restore SSH access

Remove the incorrect gateway, replacing <GATEWAY_IP> with the address from the previous step:

sudo route del -net 0.0.0.0 gw <GATEWAY_IP> netmask 0.0.0.0 dev ens3

Using the address from our example, this should look something like this:

sudo route del -net 0.0.0.0 gw 10.0.0.2 netmask 0.0.0.0 dev ens3

If you are accessing the VM via the console, then this should have resumed SSH connectivity which will make things easier for you as you can now use normal terminal functionality such as copy and paste. Close the console and SSH into the VM if you haven't already. Then move onto Step 7.

Step 7 - Install ifupdown

Next install the ifupdown package

sudo apt-get update
sudo apt-get install -y ifupdown

Step 8 - Replace configuration files

Delete all of the Netplan configuration files:

sudo rm -rf /etc/netplan/*.yml

Open the file /etc/network/interfaces and copy and paste the following:

auto lo
iface lo inet loopback

auto ens3
iface ens3 inet dhcp

auto ens4
iface ens4 inet dhcp
up ip route change default via 172.31.255.254

source /etc/network/interfaces.d/*.cfg

Step 9 - Reboot the instance

You can now reboot the instance.

reboot

Now move onto Step 10.

Step 10 - Confirm configuration

Rebooting the instance should hopefully have worked and you should still have SSH access to the machine. You can confirm this by SSHing into the instance. Once in the instance, running:

ip r

Should now only show one default route:

default via 172.31.255.254 dev ens4 proto dhcp metric 100
...

Move onto Step 11 only if you normally access the VM with an SSH key and you created a username and password in Step 3 that you wish to disable.

Step 11 - Remove user password access

IF YOU REMOVE THE ABILITY TO LOG IN WITH A PASSWORD AND DO NOT HAVE AN SSH KEY OR OTHERWISE RELY ON PASSWORD ACCESS VIA SSH, YOU WILL NOT BE ABLE TO ACCESS THE VM AND ANY ACCESS SSH, CONSOLE OR OTHERWISE WILL BE LOST AND UNRECOVERABLE AS WILL ANY DATA ON THE VM. DO NOT RUN THIS STEP ON ANY USER THAT YOU NEED TO LOG IN WITH USING A PASSWORD.

From the VM run:

passwd civorescue --lock

This will prepend a ! to the encrypted password in /etc/shadow meaning it will result in a value that will match no possible encrypted value.