During the system upgrade (apt-get upgrade) which updates also the docker-engine package, the docker service is restarted, which means stopping all containers.
Theoretically, containers created with the restart policy set to „always” should restart themselves automatically, but some time ago we had a problem with restarting one of such containers. The container was part of the multi-container application:
~:# docker-compose up -d ... ERROR: for app oci runtime error: container with id exists: b10b023395 ...
Manual interventions and attempts to remove the container did not give anything:
~:# docker-compose restart app Restarting app ... error ERROR: for app Cannot restart container b10b023395: oci runtime error: container with id exists: b10b023395 ~:# docker rm app Error response from daemon: Driver btrfs failed to remove root filesystem b10b023395: Failed to destroy btrfs snapshot /var/lib/docker/btrfs/subvolumes for 94f08a2322: device or resource busy
Also, attempts to directly remove the indicated snapshot using
btrfs subvolume delete failed with the message „device or resource busy”. A quick search for a solution to the problem on the Internet only led to quite old errors in the docker itself, which were not related to our case.
lsof command used on the path of the problematic snapshot revealed that the processes of our container were still working (!!!) and were blocking the file system. Manual killing those processes solved the problem:
~:# lsof -Fp /var/lib/docker/btrfs/subvolumes/94f08a2322 | tr -d 'p' | xargs kill -TERM