Docker compose up build fails to update container

The Problem

if you try the following command to update your service after changes and it works for you with some project but not with all project I can help you to fix the problem and understand the reason behind this behaviour:

Copy
1docker compose -f docker-compose.prod.yml up --build --no-deps --detach your_service

Let's explain what this command is requested to do:

• up start the service your_service
• --build build the image if it does not exist or has changed. Use cache for parts which have not changed.
• --no-deps do not start any linked services
• --detach run container in background (official documentation docker-compose up)

What we expect after we changed files which are part of the image:

• rebuild of the image
• using the cache to avoid rebuild of parts which have not changed
• restart of the container based on the new image

There are many places on the web which suggest this command and for it only worked with some of my projects. Digging into the problem I found out that the reason behind are anonymous volumes.

Solution

To fix the problem with anonymous volumes you only need to add --renew-anon-volumes:

Copy
1docker compose -f docker-compose.prod.yml up --build --no-deps --renew-anon-volumes --detach your_service

The additional flag will allway recreate all anonymous volumes on the restart for this service. This will make any changes overlapping with these volumes to become visible.

What's behind anonymous volumes

Check for existing anon volumes

Use this command and replace <your container id> with your container ID:

Copy
1docker inspect --type container -f '{{range $i, $v := .Mounts }}{{printf "%v\n" $v}}{{end}}' <your container id>

all anonymous volumes will have a long internal hash directly after {volume :

Copy
1{volume 277654df19e38eeb10f92be90c8df76558033bcd3c7b871e75abdc14174a46d8 /var/lib/docker/volumes/277654df19e38eeb10f92be90c8df76558033bcd3c7b871e75abdc14174a46d8/_data /opt/app/etc local  true }

How do I create such a volume?

A) docker-compose.yml

Copy
1services:
2  your_service:
3     build: .
4     volumes:
5       - /my_data # anonymous volume

B) Dockerfile

Copy
1VOLUME  "/var/logs" "/data"

Let's asume that we normally only use named volumes and no anonymous volumes in our docker-compose.yml files and focus on option B).

If we look at the official documentation for VOLUME we will not find any clues:

To explain the details we need an example which simulates the problem.

Copy
1src
2  config
3    name.txt
4  hello-world.sh
5  Dockerfile
6docker-compose.yml

You can clone these files from this repository: https://github.com/aheissenberger/docker-anon-volumes

src/hello-world.sh - our service app

Copy
1#!/bin/sh
2NAME=$(cat /app/config/name.txt)
3while sleep 5; do  echo  "Hello World! Hello $NAME"; done

src/config/name.txt - a config file

Copy
1Max

src/Dockerfile

Copy
1FROM  alpine
2COPY  hello-world.sh  /app/hello-world.sh
3COPY  config/name.txt  /app/config/name.txt
4CMD  \["sh","/app/hello-world.sh"\]

docker-compose.yml

Copy
1services:
2  hello:
3    build: ./src

we can start the service: docker compose up --detach Hint: if you do not have the latest version of docker, you will have to replace docker compose with docker-compose with the dash between the words!

Copy
1Creating network "docker-anon-volumes_default" with the default driver
2Building hello
3[+] Building 2.0s (8/8) FINISHED                                                           
4Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
5WARNING: Image for service hello was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
6Creating docker-anon-volumes_hello_1 ... done
7Attaching to docker-anon-volumes_hello_1
8hello_1  | Hello World! Hello Max
9hello_1  | Hello World! Hello Max

check the output of the service:

Copy
1$ docker compose logs -f
2hello_1  | Hello World! Hello Max
3hello_1  | Hello World! Hello Max
4hello_1  | Hello World! Hello Max
5...

now we change the content in file src/config/name.txt from Max to Rudi echo "Rudi">src/config/name.txt and restart the service with the new configuration docker compose up --build --no-deps --detach

check the output of the service:

Copy
1$ docker compose logs -f
2hello_1  | Hello World! Hello Rudi
3hello_1  | Hello World! Hello Rudi
4...

Exactly what we expected.

Check for anonymous volumes:

1. get the ID of the container

   Copy
   1$ docker ps        
   2CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS          PORTS     NAMES
   31df0f6ee87d4   1c356deb72b1   "sh /app/hello-world…"   40 seconds ago   Up 37 seconds             docker-anon-volumes_hello_1

2. run the command to list the volumes with your container id 1df0f6ee87d4 from docker ps

   Copy
   1$ docker inspect --type container -f '{{range $i, $v := .Mounts }}{{printf "%v\n" $v}}{{end}}' 1df0f6ee87d4

There should be no output by this command!

Now we change our Dockerfile to optional allow to bind the app/config directory to a local directory for development by adding the VOLUME command.

src/Dockerfile

Copy
1FROM  alpine
2COPY  hello-world.sh  /app/hello-world.sh
3COPY  config/name.txt  /app/config/name.txt
4VOLUME /app/config
5CMD  \["sh","/app/hello-world.sh"\]

restart the service with the new configuration docker compose up --build --no-deps --detach

check the output of the service:

Copy
1$ docker compose logs -f
2hello_1  | Hello World! Hello Rudi
3hello_1  | Hello World! Hello Rudi
4...

now we change the content in file src/config/name.txt from Rudi to Franz echo "Franz">src/config/name.txt

restart the service with the new configuration docker compose up --build --no-deps --detach

check the output of the service:

Copy
1$ docker compose logs -f
2hello_1  | Hello World! Hello Rudi
3hello_1  | Hello World! Hello Rudi
4...

We check for anonymous volumes:

1. get the ID of the container

   Copy
   1$ docker ps        
   2CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS          PORTS     NAMES
   32419dd47277a   696b927fee44   "sh /app/hello-world…"   About a minute ago   Up About a minute             docker-anon-volumes_hello_1

2. run the command to list the volumes with your container id 2419dd47277a from docker ps

   Copy
   1$ docker inspect --type container -f '{{range $i, $v := .Mounts }}{{printf "%v\n" $v}}{{end}}' 2419dd47277a
   2{volume c62baf7300ddc9dc66a7e871d6511e5e3a6274072cc20c3f14bb59bca5319935 /var/lib/docker/volumes/c62baf7300ddc9dc66a7e871d6511e5e3a6274072cc20c3f14bb59bca5319935/_data /app/config local rw true }

   What do we get:
3. We have an anonymous volume attached to our container volume c62baf7300ddc9dc66a7e871d6511e5e3a6274072cc20c3f14bb59bca5319935
4. The container ID here 2419dd47277a is different to the one before 1df0f6ee87d4. This is a new container with the changes name.txt
5. The content of the anonymous volume is mapped to the /app/config directory and is overlaying the new container with the old files

Fixing

1. remove the VOLUME command if not used
2. add --renew-anon-volumes to your up --build to recreate this anonymous volume

The number 1 reason why this happens is that you consume a third party image and you did never checked the exported VOLUMES of this image. As you are not in control of this image there is no way to fix this.

Let's try option 2):

change the content in file src/config/name.txt from Franzto Hans echo "Hans">src/config/name.txt we need to do this as the image has already created with name=Franz and a rebuild would not detect and changes what would lead to the up command not recreating the container and no recreation of the anonymous volumes

Copy
1docker compose up --build --no-deps --detach --renew-anon-volumes

check the output of the service:

Copy
1$ docker compose logs -f
2hello_1  | Hello World! Hello Hans
3hello_1  | Hello World! Hello Hans
4...

This is what we expected from the beginning :-)