Build/bundle in blowing away ENV variables in docker-compose.yml

Technical Support
, , , ,
1

Working on a system that sources docker-compose variables from a collocated .env file written during manufacturing test and commissioning. The torizoncorebuilder bundle command, run alone or by build, is performing variable substitution and bundle doesn’t seem to support the options to control this behaviour that build does…

nor is bundle sourcing anything from the shell’s env:

firee@FIREESCAPE:/mnt/wsl/docker-desktop-bind-mounts/Ubuntu-20.04/f08193a6451eeb70b4685212b7e5a9c8a5f1ee065a178dbb0a5dace4bd70f7d4$ echo $MY_JOURNAL_SERVER
MY_JOURNAL_SERVER
firee@FIREESCAPE:/mnt/wsl/docker-desktop-bind-mounts/Ubuntu-20.04/f08193a6451eeb70b4685212b7e5a9c8a5f1ee065a178dbb0a5dace4bd70f7d4$ torizoncore-builder bundle docker-compose-registry-release.yml
Creating Docker Container bundle...
The MY_JOURNAL_SERVER variable is not set. Defaulting to a blank string.

Right now I just shell into the programmed the device and manually fix this clobbering, but that’s obviously not scalable.

Is there ANY way to have bundle work with a docker-compose.yml that contains the string ${VAR_FROM_ESLEWHERE} ???

2

Greetings @ed-bear,

How do you have your environment for TorizonCore Builder set up?

As a test I did the following. My working directory only has a docker-compose.yml, and a .env files. The contents of the files look like:

# docker-compose.yml
version: '2.4'
services:
  hello-world:
    image: hello-world:latest
    environment:
    - a=${FOO}

# .env
FOO=blah

After running torizoncore-builder bundle docker-compose.yml, if I check the resulting bundle directory:

# bundle/docker-compose.yml
services:
  hello-world:
    environment:
      a: blah
    image: hello-world@sha256:53f1bbee2f52c39e41682ee1d388285290c5c8a76cc92b42687eecf38e0af3f0
version: '2.4'

As seen above the variable substitution happened based on the contents of the .env file. With that said, it should work properly. But perhaps your development environment differs slightly from mine?

Best Regards,
Jeremias

3

The target filesystem needs it’s own .env created when we test the Toradex in a larger system and commission a cloud system against the tested device. We need a=${FOO} to be included in the docker-compose.yml file that gets saved to filesystem, not replaced by bundle command. My setup behaves the same as you describe, but the a: blah substitution is exactly the problem.

4

Apologies I misunderstood you initially.

So you don’t want the variable to be substituted then? In that case you are correct that we don’t currently have an option to disable this automatic substitution. However there is a different way to go about this. Look at this example docker-compose.yml:

version: '2.4'
 
services:
  test:
    image: torizon/debian:2-bullseye
    environment:
    - MACHINE
    command: tail -F anything

Ignore the tail command that’s just to keep the container running forever. On TorizonCore by default “MACHINE” is already a defined environment variable in the shell, which is why I’m using it for this example.

After I run the bundle command the compose file gets transformed like so:

services:
  test:
    command: tail -F anything
    environment:
      MACHINE: null
    image: torizon/debian@sha256:993cbe0d0dd328ea2ff2e837a06bff90b81d1e681f24007949df64f5351e5215
version: '2.4'

After we deploy this to the device we can see the environment variable is transferred from the host shell to the container:

apalis-imx8-06738453:~$ echo $MACHINE 
apalis-imx8
apalis-imx8-06738453:~$ docker exec -it e bash
root@e5cdf792a479:/# echo $MACHINE
apalis-imx8

By doing this you can work with environment variables. Basically you just have to not use the ${FOO} notation and just use FOO in the compose file. But the environment variable name has to match the name of the variable outside the container. Like in my example “MACHINE” was already defined outside of the container, so it’s value gets passed to the “MACHINE” inside the container.

Does this help your use-case case? Or do you require something different?

Best Regards,
Jeremias

5

This won’t work as a drop in solution, as we are using docker-compose’s built in .env file processing Environment variables in Compose | Docker Documentation to modify the command.
bundle is still going to blow away the reference to the variable the the docker-compose YAML.

How would I tail -F ${anything} ?

6

Hi @ed-bear ,

I think I managed to do a workaround for your problem. Using the example made by @jeremias.tx :

# docker-compose.yml
version: '2.4'
services:
  hello-world:
    image: hello-world:latest
    environment:
    - A=${D_SIGN}${FOO}
    - B=${D_SIGN}${BAR}

# .env
D_SIGN='$'
FOO={FOO}
BAR={BAR}

The resulting bundle/docker-compose.yml after executing torizoncore-builder bundle docker-compose.yml is:

# bundle/docker-compose.yml
services:
  hello-world:
    environment:
      A: ${FOO}
      B: ${BAR}
    image: hello-world@sha256:53f1bbee2f52c39e41682ee1d388285290c5c8a76cc92b42687eecf38e0af3f0
version: '2.4'

You have to create a .env for your host machine like I did above, but it should do the trick. Just adapt it to your situation accordingly.

Try doing the above and see if it helps you.

Best regards,
Lucas Akira

1 Like
7

The D_SIGN='$' in bundle’s .env file works with command: ssh user@${D_SIGN}{MY_VARIABLE_REMOTE_SERVER} to generate a docker-compose.yml with command: ssh user@${MY_VARIABLE_REMOTE_SERVER}

8

Glad I was able to help!

Best regards,
Lucas Akira