Advanced ARG and ENV Dockerfile tricks

Or how to (not) hang yourself manipulating a container environment at build-time, runtime, across stages and across images boundaries

Jump directly to section “Advanced Tricks” if you think you do not need to read the refresher and pitfalls.

Refresher on ARG and ENV basic concepts

ENV

In a Dockerfile, ENV lets you declare environment variables that will be available during both the build phase and at runtime.

Here is an example:

FROM debian
# Declare the variable
ENV THIS_IS_ENV=default_env_value
# Echo it's value at build time
RUN echo "I'm inside the image at build time and THIS_IS_ENV value is ${THIS_IS_ENV:-}"
# This will be executed at runtime
CMD ["bash", "-c", "echo At runtime and THIS_IS_ENV value is ${THIS_IS_ENV:-}"]

At build-time, THIS_IS_ENV will always be set to the value being declared in the Dockerfile, but, at runtime, its value may be overridden by passing --env THIS_IS_ENV=overridden to your docker run call.

Tester:

# Build
docker build -f Dockerfile --progress=plain --no-cache -t tester .
# Run it as-is
docker run tester
# Run it overriding the value of the env variable
docker run --env THIS_IS_ENV=overridden tester

ARG

On the other hand, ARG allows you to define an environment variable that will only be available at build-time, and that may be overridden when building.

FROM debian as base
# Declare the arg
ARG THIS_IS_ARG=default_arg_value
# Echo its value at build time
RUN echo "I'm in the base image at build time and THIS_IS_ARG value is ${THIS_IS_ARG:-}"
# Notice that at runtime, it's undeclared
CMD ["bash", "-c", "echo At runtime and THIS_IS_ARG value is ${THIS_IS_ARG:-}"]

Tester:

docker build -f Dockerfile --progress=plain --no-cache -t tester .
docker build -f Dockerfile --progress=plain --no-cache -t tester --build-arg THIS_IS_ARG=overridden .
docker run --rm tester

Multi-stage builds

With multi-stage builds (where a stage starts from another), you inherit all ENV and ARG defined in any previous stage.

If you do override an ARG at build-time, all stages will get the modified value.

Evidently, ENV still cannot be altered at build-time, but only at runtime.

For example:

# First stage
FROM debian as base
ENV THIS_IS_ENV=default_env_value
ARG THIS_IS_ARG=default_arg_value
RUN echo "I'm in the base image at build time and THIS_IS_ENV value is ${THIS_IS_ENV:-}"
RUN echo "I'm in the base image at build time and THIS_IS_ARG value is ${THIS_IS_ARG:-}"
# Second stage
FROM base
RUN echo "I'm in the second stage image at build time and THIS_IS_ENV value is ${THIS_IS_ENV:-}"
RUN echo "I'm in the second stage image at build time and THIS_IS_ARG value is ${THIS_IS_ARG:-}"
CMD ["bash", "-c", "echo At runtime and THIS_IS_ENV value is ${THIS_IS_ENV:-}, THIS_IS_ARG value is ${THIS_IS_ARG:-}"]

Tester:

docker build -f Dockerfile --progress=plain --no-cache -t tester .
docker build -f Dockerfile --build-arg THIS_IS_ARG=overridden_arg --progress=plain --no-cache -t tester .
docker run --env THIS_IS_ENV=overridden_env tester

Pre-FROM args

If you would like to parameterize your FROM image, Docker neatly allows you to declare ARG before your first FROM at the top of your Dockerfile.

For example:

ARG FROM_IMAGE=debian
FROM $FROM_IMAGE

This allows you to parameterize and control your base image url at build time without changing anything in your Dockerfile, using --build-arg FROM_IMAGE=something_else .

This is of course very neat if you would like to pull your base image from a local registry instead of from, say, Docker Hub (assuming you have local copies of course), or if you would like to test quickly a newer/different version of your base image.

ONBUILD

Another neat Docker mechanism is the concept of ONBUILD that lets you preemptively declare ARG and ENV in a stage, that will actually only be available at any subsequent stage.

Here is a concrete example:

# First stage
FROM debian as base
ENV THIS_IS_ENV=default_env_value
ARG THIS_IS_ARG=default_arg_value
ONBUILD ENV THIS_IS_ONBUILD_ENV=default_onbuild_env_value
ONBUILD ARG THIS_IS_ONBUILD_ARG=default_onbuild_arg_value
RUN echo "I'm in the base image at build time"; env | grep THIS_
# Second stage
FROM base
RUN echo "I'm in the second stage image at build time"; env | grep THIS_
CMD ["bash", "-c", "echo At runtime; env | grep THIS_"]

ARGs and ENVs are position dependent

This may sound obvious, but is better said: the point at which you declare an ARG or an ENV inside your Dockerfile makes it available in the environment for any instruction that follows.

Common pitfalls

#1 ENVs cannot be declared before FROM, at the top

Unlike args, envs have to declared after the first FROM.

#2 overriding an ARG with -build-arg does override all occurrences of that arg in all stages

This sounds rather obvious.

#3 (ONBUILD) ARGs behave differently if defined in your base FROM image (as compared to using multi-stage)

What has been said above with regard to multi-stage images does not apply when you build FROM another, already built image.

An easy way to think about this is:

• in a multi-stage build, you are building everything
• in a FROM scenario, your base image has been already built and will not be built again

Specifically:

• ARGs declared inside your base image are not available in your inheriting image
• an ONBUILD ARG declared in your base image is only made available if it has been declared in the last stage of the base image

#4 you cannot use variables inside the --from segment of a COPY instruction (unlike in a FROM)

See trick below for a workaround.

#5 ARGs declared before your FROM are not available to the rest of your Dockerfile

See trick below for a workaround.

#6 if the same variable is defined as both an ARG and an ENV in prior stages, the ENV value will override the ARG one in subsequent stages (but not in the one where the ARG first appears in)

This one is a very convenient way to hurt yourself. Think twice before you use the same variable name for both an ENV and an ARG. While this is a powerful trick (see below), it is also a mind-bender.

Advanced Tricks

Parameter expansion

Both ENV and ARG (in ONBUILD as well), can reference other environment variables and support some parameter expansion.

This is pretty neat and allows you for example to derive computed values out of another variable:

FROM debian
ARG SET_OR_UNSET
ENV THIS_ENV=${SET_OR_UNSET:+value_if_arg_is_set}

In this scenario, THIS_ENV will:

• be empty if SET_OR_UNSET is not set
• will have the value value_if_arg_is_set if SET_OR_UNSET has been set

Note that FROM also support parameter expansion

Using repeat ENV at build time

The usefulness of repeating the same ARG at build time is rather limited. If it is overridden, all instances will end-up with the same value.

Similarly, repeating the same ENV has no impact at all at runtime.

But now, repeating ENV at build-time will allow for more advanced environment manipulation when combined with parameter expansion.

For example:

FROM debian
ARG SET_OR_UNSET
ENV THIS_ENV=${SET_OR_UNSET:+value_if_this_is_set}
ENV THIS_ENV=${THIS_ENV:-value_if_arg_is_not_set}

You have now successfully implemented a ternary.

Shadowing at build time

Declaring the same variable with ENV and ARG will allow the same variable to have a different value at build time than at runtime, letting the operator control both independently.

FROM debian
ENV THIS_VAR=value_at_runtime
ARG THIS_VAR=value_at_buildtime

Shadowing at build time, for just one stage

In a base image:

FROM debian
ENV THIS_VAR="always available to all stages"
ONBUILD ARG THIS_VAR="will override THIS_VAR *just* for the next first stage"

And in your inheriting image:

FROM my_image_built_from_above as first_stage
RUN echo "THIS_VAR value is that of the ONBUILD ARG $THIS_VAR"
FROM first_stage as second_stage
RUN echo "THIS_VAR value is that of the ENV: $THIS_VAR"

Workaround: access to pre-FROM ARGs

Easy. Declare your ARG twice.

ARG BASE_IMAGE=debian
FROM $BASE_IMAGE
ARG BASE_IMAGE
RUN echo "$BASE_IMAGE"

Note that you do not have to set the same default value. If left as-is, it will get the default value set before.

Workaround: parameterized COPY --from with image staging

As stated above, you unfortunately cannot use COPY --from=$PARAM

The easy way to workaround that is to leverage FROM instead, declare a named extra stage and use that fixed name in your COPY.

Concretely:

ARG COPY_SOURCE_IMAGE=golang
FROM $COPY_SOURCE_IMAGE as image_staging
FROM debian
COPY --from=image_staging /thing /destination

Concrete use-cases

A few useful scenarios

Easy registry selection, with “scratch” fallback

ARG FROM_REGISTRY=ghcr.io
ARG FROM_IMAGE=${FROM_REGISTRY:+$FROM_REGISTRY/me/my_image}
FROM ${FROM_IMAGE:-scratch}

The FROM will default to “ghcr.io/me/my_image”.

Overriding the build-arg FROM_REGISTRY will allow the builder to control from which registry this image is being sourced.

Overriding the build-arg FROM_IMAGE will further allow the builder to control the FQN of the image.

If FROM_REGISTRY is overridden to the empty string, the image will instead be scratch.

Controlling app specific behavior in a single subsequent stage through shadowing

In a base image:

FROM golang
ENV GOPROXY=off
ONBUILD ARG GOPROXY=https://go-proxy.local

Inheriting image

# Only this stage allows go to retrieve content
FROM my_base_trick as fetch_stage
RUN git clone whatever
RUN go mod download
# This stage and all subsequent will have GOPROXY=off
FROM fetch_stage
RUN echo "GOPROXY is: $GOPROXY"