Dev Ups

Published 2022-03-06 in ci-cd

Cleaning up after uninstalling, Docker in Ubuntu

In A first Docker microservice I referred to the official uninstall instructions from Docker. Having recently had an installation fail evaluation on a resource constrained (aren't they all?) system I needed to make the uninstall as clean as possible, and the official instructions from Docker weren't much use. It is worth knowing a more generalised approach to cleaning up resources in Ubuntu in general. No one wants to spend time re-installing their OS because of something left behind my previous installations causes a shared tool to not work on their machine as it does on everyone else's. Docker is about the weight of tool I have in mind and will illustrate this example.

Docker's solution

Understanding package managers is a key driver for productivity. We don't want to be taking custom uninstall instructions for every installation we want undone. That is an advantage of package managers like apt-get, and something the OS maintainers put a lot of time and energy into ensuring. I know Docker would have me do:

sudo apt-get remove docker docker-engine docker.io containerd runc

Space used (from df /, in 1K-blocks):

  • pre installation: 1623876
  • post installation (after running hello-world): 2050544
  • post removal: 1699020
  • peak: 351524
  • residual: 75144

What the package manager tells us we are getting:

  • bridge-utils containerd dns-root-data dnsmasq-base docker.io libidn11 pigz runc ubuntu-fan
  • 361 MB of additional disk space will be used

What the package manager tells us we are removing:

  • containerd docker.io runc
  • 359 MB

Those are some big discrepancies. sudo apt-get remove docker docker-engine docker.io containerd runc doesn't work because docker-engine is not installed. My disk is left 72 MB smaller.

Better solution

Things didn't work out and you need to rollback? Let's reclaim the space and sanity we lent Docker:

# this is shorthand for "apt-get remove --purge xxx":
sudo apt-get purge docker.io
sudo apt-get autoremove --purge

Space used (from df /, in 1K-blocks):

  • pre installation: 1623848
  • post installation (after running hello-world): 2050512
  • post removal: 1697020
  • peak: 353492
  • residual: 73172

What the package manager tells us we are getting:

  • bridge-utils containerd dns-root-data dnsmasq-base docker.io libidn11 pigz runc ubuntu-fan
  • 361 MB of additional disk space will be used

What the package manager tells us we are removing:

  • docker.io* + bridge-utils* containerd* dns-root-data* dnsmasq-base* libidn11* pigz* runc* ubuntu-fan*
  • 193 + 168 MB = 361 MB

The discrepancies are less. The residual space used on the disk is still a mystery. Maybe some massive logs which in time will pale into insignificance? Counter to this is that only 337 MB were installed, out of a claimed 361 MB, so why would uninstalling create such a large log? More on this later.

The one thing both solutions appeared to miss, the network device, got purged upon rebooting:

user432@managed-node:~$ ip -4 addr show docker0
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

A reason to roll back

Let's say you bought a cheap OpenVZ VPS and got this error (it looked a lot redder on my screen):

user810@cloudvps1:~$ df /
Filesystem        1K-blocks    Used Available Use% Mounted on
/dev/ploop23404p1  82436504 4869860  74090296   7% /
user810@cloudvps1:~$ sudo apt-get install docker.io
...
The following additional packages will be installed:
  apparmor bridge-utils containerd dnsmasq-base git git-man libcurl3-gnutls liberror-perl netcat
  netcat-openbsd pigz runc ubuntu-fan
...
After this operation, 402 MB of additional disk space will be used.
...
Done.
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service  /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket  /lib/systemd/system/docker.socket.
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
invoke-rc.d: initscript docker, action "start" failed.
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Fri 2022-03-04 23:24:50 GMT; 9ms ago
TriggeredBy:  docker.socket
       Docs: https://docs.docker.com
    Process: 597456 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
   Main PID: 597456 (code=exited, status=1/FAILURE)

Mar 04 23:24:50 cloudvps1 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Mar 04 23:24:50 cloudvps1 systemd[1]: docker.service: Failed with result 'exit-code'.
Mar 04 23:24:50 cloudvps1 systemd[1]: Failed to start Docker Application Container Engine.
Setting up git (1:2.25.1-1ubuntu3.2) ...
Processing triggers for systemd (245.4-4ubuntu3.2) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for dbus (1.12.16-2ubuntu2.1) ...
Processing triggers for libc-bin (2.31-0ubuntu9) ...
user810@cloudvps1:~$ $?
-bash: 0: command not found

At this point the docker daemon was not running and I didn't have the docker network adaptor. I had used 468 MB (5349516k-4869860k).

I investigated with sudo journalctl -u docker.service:

Mar 04 23:24:50 cloudvps1 dockerd[597456]: time="2022-03-04T23:24:50.162031580Z" level=warning msg="Running iptables --wait -t nat -L -n failed with message: `iptables v1.8.4 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)\nPerhaps iptables or your kernel needs to be upgraded.`, error: exit status 3"
Mar 04 23:24:50 cloudvps1 dockerd[597456]: failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables --wait -t nat -N DOCKER: iptables v1.8.4 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Mar 04 23:24:50 cloudvps1 dockerd[597456]: Perhaps iptables or your kernel needs to be upgraded.
Mar 04 23:24:50 cloudvps1 dockerd[597456]:  (exit status 3)
Mar 04 23:24:50 cloudvps1 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Mar 04 23:24:50 cloudvps1 systemd[1]: docker.service: Failed with result 'exit-code'.
Mar 04 23:24:50 cloudvps1 systemd[1]: Failed to start Docker Application Container Engine.

I still had discrepancies:

operation quoted MB
install 402 MB
purge 193 MB
autoremove 207 MB
residual 2 MB
Filesystem        1K-blocks    Used Available Use% Mounted on
/dev/ploop23404p1  82436504 4869860  74090296   7% /  # before
/dev/ploop23404p1  82436504 5349516  73610640   7% /  # during
/dev/ploop23404p1  82436504 4959728  74000428   7% /  # after

That's 89868 1K blocks unaccounted for, 86 MB. The docker log is still there, but tiny. Luckily my RAM isn't affected.

My guess where 86 MB went was:

user810@cloudvps1:~$ sudo journalctl --disk-usage
Archived and active journals take up 2.7G in the file system.

Conclusion

Ubuntu has a well defined procedure for fully removing and adding packages. It outperforms Docker's own instructions in this example but considerable space is left unaccounted for after uninstalling a considerable package, Docker.

I went through the process once more, checking sudo journalctl --disk-usage at the same stages as the other measurements were taken. It never exceeded 16MB. My next exercise would be learning about the file hierarchy of journalctl and better disk space tracking and management programs.