Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://stackoverflow.com/questions/59691207/docker-build-with-nvidia-runtime below:

docker build with nvidia runtime

Asked 5 years, 7 months ago

Viewed 51k times

I have a GPU application that does unit-testing during the image building stage. With Docker 19.03, one can specify nvidia runtime with docker run --gpus all but I also need access to the gpus for docker build because I do unit-testing. How can I achieve this goal?

For older version of docker that use nvidia-docker2 it was not possible to specifiy runtime during build stage, BUT you can set the default runtime to be nvidia, and docker build works fine that way. Can I do that in Docker 19.03 that doesn't need nvidia-docker anymore? If so, how?

asked Jan 11, 2020 at 2:44

1,33911 gold badge1414 silver badges3737 bronze badges

You need use nvidia-container-runtime as explained in docs: "It is also the only way to have GPU access during docker build".

Steps for Ubuntu:

1. Install nvidia-container-runtime:

   sudo apt-get install nvidia-container-runtime

2. Edit/create the /etc/docker/daemon.json with content:

{
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
         } 
    },
    "default-runtime": "nvidia" 
}

1. Restart docker daemon:

   sudo systemctl restart docker

2. Build your image (now GPU available during build):

   docker build -t my_image_name:latest .

answered May 11, 2020 at 19:06

2,58211 gold badge2020 silver badges1919 bronze badges

A "solution" I found is to first run a base image with the host nvidia drivers mounted on it

docker run -it --rm --gpus ubuntu

And then build my app within the container manually and commit the resulting image. This is not ideal and it would be best to have access to nvidia-smi during the build phase.

answered Feb 1, 2020 at 4:00

1,33911 gold badge1414 silver badges3737 bronze badges

I was to successfully run a custom-built ffmpeg in a docker build image. It was not easy to figure out.

1. Set nvidia to be the default docker runtime Make sure this line is in the top-level of your /etc/docker/daemon.json like @Anton Ganichev suggests above:

"default-runtime": "nvidia"

1. Remove buildx as suggested by @Mason

sudo apt remove docker-buildx-plugin

1. Restart docker

sudo systemctl restart docker

1. Run docker build with stdin instead of Dockerfile. The reason is that otherwise, the binary sent to docker daemon will be extremely extremely large (like in the hundreds of GBs)

docker build - < Dockerfile

I have the following ENV variables in my Dockerfile:

ENV PATH /opt/conda/bin:/usr/local/bin:${PATH}
ENV LD_LIBRARY_PATH /usr/local/cuda/lib64/stubs/:/usr/lib/x86_64-linux-gnu:/usr/local/cuda-12.2/compat/:/usr/local/cuda-12.2/targets/x86_64-linux/lib/stubs:$LD_LIBRARY_PATH
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES video,compute,utility

answered Oct 24, 2023 at 0:19

1,4842020 silver badges2222 bronze badges

Let me wrap up this issue with proper explanations.

Answer

• Follow the @Anton Ganichev's answer.

  
  

Explain

• With the legacy docker build, dockerd invokes the configured OCI runtime. If "default-runtime": "nvidia" is set in /etc/docker/daemon.json, Docker uses nvidia-container-runtime (a wrapper around runc) for build containers.

• You must install nvidia-container-runtime and configure /etc/docker/daemon.json.

• nvidia-container-runtime applies GPU settings according to its mode, e.g. legacy or jit-cdi (legacy, jit-cdi).

  
  

Answer

1. make a gpubuilder to help you use GPUs in your build process

docker buildx create --name gpubuilder \
  --driver-opt image=crazymax/buildkit:v0.23.2-ubuntu-nvidia \
  --bootstrap

1. modify your Dockerfile to specify you use GPUs, like

# syntax=docker/dockerfile:1-labs
FROM ubuntu
RUN --device=nvidia.com/gpu nvidia-smi -L

1. Finally, build the Dockerfile with the gpubuilder , like

docker buildx --builder gpubuilder build --progress=plain .

Reference: https://docs.docker.com/build/building/cdi/#set-up-a-container-builder-with-gpu-support

Alternative

DOCKER_BUILDKIT=0 docker build ...

Explain

• When the buildx plugin is installed, docker build runs via Buildx/BuildKit by default (the legacy builder is deprecated).

• BuildKit does not inherit the Engine’s "default-runtime": "nvidia" setting, so the legacy “set a default NVIDIA runtime” trick won’t enable GPUs during builds. This is intentional: BuildKit aims for sandboxed, reproducible builds (discussion).

• Therefore, we need to specify the other way to use GPU, such as gpubuilder.

• In the same context, it is also possible to use DOCKER_BUILDKIT=0 because it forces docker to use legacy builder rather then Buildkit

  
  

• The points above are about builds. For runtime containers, setting /etc/docker/daemon.json (e.g., "default-runtime": "nvidia") is sufficient.

63611 gold badge55 silver badges1818 bronze badges

Start asking to get answers

Find the answer to your question by asking.

Explore related questions

See similar questions with these tags.

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4