Advanced ARG and ENV Dockerfile tricks
Or how to (not) hang yourself manipulating a container environment at build-time, runtime, across stages and across images boundaries
Jump directly to section “Advanced Tricks” if you think you do not need to read the refresher and pitfalls.
Refresher on ARG and ENV basic concepts
ENV
In a Dockerfile, ENV
lets you declare environment variables that will be available during both the build phase and at runtime.
Here is an example:
FROM debian
# Declare the variable
ENV THIS_IS_ENV=default_env_value
# Echo it's value at build time
RUN echo "I'm inside the image at build time and THIS_IS_ENV value is ${THIS_IS_ENV:-}"
# This will be executed at runtime
CMD ["bash", "-c", "echo At runtime and THIS_IS_ENV value is ${THIS_IS_ENV:-}"]
At build-time, THIS_IS_ENV
will always be set to the value being declared in the Dockerfile, but, at runtime, its value may be overridden by passing --env THIS_IS_ENV=overridden
to your docker run
call.
Tester:
# Build
docker build -f Dockerfile --progress=plain --no-cache -t tester .
# Run it as-is
docker run tester
# Run it overriding the value of the env variable
docker run --env THIS_IS_ENV=overridden tester
ARG
On the other hand, ARG
allows you to define an environment variable that will only be available at build-time, and that may be overridden when building.
FROM debian as base
# Declare the arg
ARG THIS_IS_ARG=default_arg_value
# Echo its value at build time
RUN echo "I'm in the base image at build time and THIS_IS_ARG value is ${THIS_IS_ARG:-}"
# Notice that at runtime, it's undeclared
CMD ["bash", "-c", "echo At runtime and THIS_IS_ARG value is ${THIS_IS_ARG:-}"]
Tester:
docker build -f Dockerfile --progress=plain --no-cache -t tester .
docker build -f Dockerfile --progress=plain --no-cache -t tester --build-arg THIS_IS_ARG=overridden .
docker run --rm tester
Multi-stage builds
With multi-stage builds (where a stage starts from another), you inherit all ENV
and ARG
defined in any previous stage.
If you do override an ARG
at build-time, all stages will get the modified value.
Evidently, ENV
still cannot be altered at build-time, but only at runtime.
For example:
# First stage
FROM debian as base
ENV THIS_IS_ENV=default_env_value
ARG THIS_IS_ARG=default_arg_value
RUN echo "I'm in the base image at build time and THIS_IS_ENV value is ${THIS_IS_ENV:-}"
RUN echo "I'm in the base image at build time and THIS_IS_ARG value is ${THIS_IS_ARG:-}"
# Second stage
FROM base
RUN echo "I'm in the second stage image at build time and THIS_IS_ENV value is ${THIS_IS_ENV:-}"
RUN echo "I'm in the second stage image at build time and THIS_IS_ARG value is ${THIS_IS_ARG:-}"
CMD ["bash", "-c", "echo At runtime and THIS_IS_ENV value is ${THIS_IS_ENV:-}, THIS_IS_ARG value is ${THIS_IS_ARG:-}"]
Tester:
docker build -f Dockerfile --progress=plain --no-cache -t tester .
docker build -f Dockerfile --build-arg THIS_IS_ARG=overridden_arg --progress=plain --no-cache -t tester .
docker run --env THIS_IS_ENV=overridden_env tester
Pre-FROM args
If you would like to parameterize your FROM
image, Docker neatly allows you to declare ARG
before your first FROM
at the top of your Dockerfile.
For example:
ARG FROM_IMAGE=debian
FROM $FROM_IMAGE
This allows you to parameterize and control your base image url at build time without changing anything in your Dockerfile, using --build-arg FROM_IMAGE=something_else
.
This is of course very neat if you would like to pull your base image from a local registry instead of from, say, Docker Hub (assuming you have local copies of course), or if you would like to test quickly a newer/different version of your base image.
ONBUILD
Another neat Docker mechanism is the concept of ONBUILD
that lets you preemptively declare ARG
and ENV
in a stage, that will actually only be available at any subsequent stage.
Here is a concrete example:
# First stage
FROM debian as base
ENV THIS_IS_ENV=default_env_value
ARG THIS_IS_ARG=default_arg_value
ONBUILD ENV THIS_IS_ONBUILD_ENV=default_onbuild_env_value
ONBUILD ARG THIS_IS_ONBUILD_ARG=default_onbuild_arg_value
RUN echo "I'm in the base image at build time"; env | grep THIS_
# Second stage
FROM base
RUN echo "I'm in the second stage image at build time"; env | grep THIS_
CMD ["bash", "-c", "echo At runtime; env | grep THIS_"]
ARGs and ENVs are position dependent
This may sound obvious, but is better said: the point at which you declare an ARG
or an ENV
inside your Dockerfile makes it available in the environment for any instruction that follows.
Common pitfalls
#1 ENVs cannot be declared before FROM, at the top
Unlike args, envs have to declared after the first FROM.
#2 overriding an ARG with -build-arg does override all occurrences of that arg in all stages
This sounds rather obvious.
#3 (ONBUILD) ARGs behave differently if defined in your base FROM image (as compared to using multi-stage)
What has been said above with regard to multi-stage images does not apply when you build FROM another, already built image.
An easy way to think about this is:
- in a multi-stage build, you are building everything
- in a FROM scenario, your base image has been already built and will not be built again
Specifically:
- ARGs declared inside your base image are not available in your inheriting image
- an ONBUILD ARG declared in your base image is only made available if it has been declared in the last stage of the base image
#4 you cannot use variables inside the --from segment of
a COPY instruction (unlike in a FROM
)
See trick below for a workaround.
#5 ARGs declared before your FROM are not available to the rest of your Dockerfile
See trick below for a workaround.
#6 if the same variable is defined as both an ARG and an ENV in prior stages, the ENV value will override the ARG one in subsequent stages (but not in the one where the ARG first appears in)
This one is a very convenient way to hurt yourself. Think twice before you use the same variable name for both an ENV and an ARG. While this is a powerful trick (see below), it is also a mind-bender.
Advanced Tricks
Parameter expansion
Both ENV and ARG (in ONBUILD as well), can reference other environment variables and support some parameter expansion.
This is pretty neat and allows you for example to derive computed values out of another variable:
FROM debianARG SET_OR_UNSET
ENV THIS_ENV=${SET_OR_UNSET:+value_if_arg_is_set}
In this scenario, THIS_ENV will:
- be empty if SET_OR_UNSET is not set
- will have the value
value_if_arg_is_set
if SET_OR_UNSET has been set
Note that FROM also support parameter expansion
Using repeat ENV at build time
The usefulness of repeating the same ARG at build time is rather limited. If it is overridden, all instances will end-up with the same value.
Similarly, repeating the same ENV has no impact at all at runtime.
But now, repeating ENV at build-time will allow for more advanced environment manipulation when combined with parameter expansion.
For example:
FROM debian
ARG SET_OR_UNSET
ENV THIS_ENV=${SET_OR_UNSET:+value_if_this_is_set}
ENV THIS_ENV=${THIS_ENV:-value_if_arg_is_not_set}
You have now successfully implemented a ternary.
Shadowing at build time
Declaring the same variable with ENV and ARG will allow the same variable to have a different value at build time than at runtime, letting the operator control both independently.
FROM debian
ENV THIS_VAR=value_at_runtime
ARG THIS_VAR=value_at_buildtime
Shadowing at build time, for just one stage
In a base image:
FROM debian
ENV THIS_VAR="always available to all stages"
ONBUILD ARG THIS_VAR="will override THIS_VAR *just* for the next first stage"
And in your inheriting image:
FROM my_image_built_from_above as first_stageRUN echo "THIS_VAR value is that of the ONBUILD ARG $THIS_VAR"FROM first_stage as second_stageRUN echo "THIS_VAR value is that of the ENV: $THIS_VAR"
Workaround: access to pre-FROM ARGs
Easy. Declare your ARG twice.
ARG BASE_IMAGE=debian
FROM $BASE_IMAGE
ARG BASE_IMAGE
RUN echo "$BASE_IMAGE"
Note that you do not have to set the same default value. If left as-is, it will get the default value set before.
Workaround: parameterized COPY --from with image staging
As stated above, you unfortunately cannot use COPY --from=$PARAM
The easy way to workaround that is to leverage FROM instead, declare a named extra stage and use that fixed name in your COPY.
Concretely:
ARG COPY_SOURCE_IMAGE=golang
FROM $COPY_SOURCE_IMAGE as image_staging
FROM debian
COPY --from=image_staging /thing /destination
Concrete use-cases
A few useful scenarios
Easy registry selection, with “scratch” fallback
ARG FROM_REGISTRY=ghcr.io
ARG FROM_IMAGE=${FROM_REGISTRY:+$FROM_REGISTRY/me/my_image}
FROM ${FROM_IMAGE:-scratch}
The FROM will default to “ghcr.io/me/my_image”.
Overriding the build-arg FROM_REGISTRY will allow the builder to control from which registry this image is being sourced.
Overriding the build-arg FROM_IMAGE will further allow the builder to control the FQN of the image.
If FROM_REGISTRY is overridden to the empty string, the image will instead be scratch.
Controlling app specific behavior in a single subsequent stage through shadowing
In a base image:
FROM golang
ENV GOPROXY=off
ONBUILD ARG GOPROXY=https://go-proxy.local
Inheriting image
# Only this stage allows go to retrieve content
FROM my_base_trick as fetch_stage
RUN git clone whatever
RUN go mod download
# This stage and all subsequent will have GOPROXY=off
FROM fetch_stage
RUN echo "GOPROXY is: $GOPROXY"