A RetroSearch Logo

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Search Query:

Showing content from https://docs.databricks.com/aws/en/compute/custom-containers below:

Customize containers with Databricks Container Service

Customize containers with Databricks Container Service

Databricks Container Services lets you specify a Docker image when you create compute. Some example use cases include:

You can also use Docker images to create custom deep learning environments on compute with GPU devices. For additional information about using GPU compute with Databricks Container Services, see Databricks Container Services on GPU compute.

For tasks to be executed each time the container starts, use an init script.

Requirements​ Limitations​ Step 1: Build your base​

Databricks recommends that you build your Docker base from a base that Databricks has built and tested. It is also possible to build your Docker base from scratch. This section describes the two options.

Option 1. Use a base built by Databricks​

This example uses the 17.x tag for an image that will target a compute with Databricks Runtime 17.0 LTS and above:

Bash

FROM databricksruntime/standard:17.x
...

To specify additional Python libraries, such as the latest version of pandas and urllib, use the container-specific version of pip. For the databricksruntime/standard:17.x container, include the following:

Bash

RUN /databricks/python3/bin/pip install pandas
RUN /databricks/python3/bin/pip install urllib3

Base images are hosted on Docker Hub at https://hub.docker.com/u/databricksruntime. The Dockerfiles used to generate these bases are at https://github.com/databricks/containers.

note

Docker Hub hosted images with Tags with “-LTS” suffix will be patched. All other images are examples and are not patched regularly.

note

The base images databricksruntime/standard and databricksruntime/minimal are not to be confused with the unrelated databricks-standard and databricks-minimal environments included in the no longer available Databricks Runtime with Conda (Beta).

Option 2. Build your own Docker base​

You can also build your Docker base from scratch. The Docker image must meet these requirements:

To build your own image from scratch, you must create the virtual environment. You must also include packages that are built into Databricks compute, such as Python and R. To get started, you can use the appropriate base image:

You can also refer to the example Dockerfiles in GitHub.

warning

Test your custom container image thoroughly on a Databricks compute. Your container may work on a local or build machine, but when your container is launched on Databricks the compute launch may fail, certain features may become disabled, or your container may stop working, even silently. In worst-case scenarios, it could corrupt your data or accidentally expose your data to external parties.

Step 2: Push your base image​

Push your custom base image to a Docker registry. This process is supported using the following registries:

Other Docker registries that support no auth or basic auth are also expected to work.

note

If you use Docker Hub for your Docker registry, be sure to check that rate limits accommodate the amount of compute that you expect to launch in a six-hour period. These rate limits are different for anonymous users, authenticated users without a paid subscription, and paid subscriptions. See the Docker documentation for details. If this limit is exceeded, you will get a “429 Too Many Requests” response.

Step 3: Launch your compute​

You can launch your compute using the UI or the API.

Launch your compute using the UI​

  1. On the Create compute page, specify a Databricks Runtime Version that supports Databricks Container Services.

  2. Under Advanced, select the Docker tab.

  3. Select Use your own Docker container.

  4. In the Docker Image URL field, enter your custom Docker image.

    Docker image URL examples:

  5. Select the authentication type. You can use secrets to store username and password authentication values. See Docker image authentication.

Launch your compute using the API​

  1. Generate an API token.

  2. Use the Databricks CLI to launch a compute with your custom Docker base.

    Bash

    databricks clusters create \
    --cluster-name <cluster-name> \
    --node-type-id i3.xlarge \
    --json '{
      "num_workers": 0,
      "docker_image": {
        "url": "databricksruntime/standard:latest",
        "basic_auth": {
          "username": "<docker-registry-username>",
          "password": "<docker-registry-password>"
        }
      },
      "spark_version": "16.4.x-scala2.12",
      "aws_attributes": {
        "availability": "ON_DEMAND",
        "instance_profile_arn": "arn:aws:iam::<aws-account-number>:instance-profile/<iam-role-name>"
      }
    }'
Docker image authentication​

Authentication requirements depend on your Docker image type. You can also use secrets to store authentication usernames and passwords. See Use secrets for authentication.

Use secrets for authentication​

Databricks Container Service supports using secrets for authentication. When creating your compute resource in the UI, use the Authentication field to select Username and password, then instead of entering your plain text username or password, enter your secrets using the {{secrets/<scope-name>/<dcs-secret>}} format. If you use the API, enter the secrets in the basic_auth fields.

For information on creating secrets, see Secret management.

Use an init script​

Databricks Container Services enable customers to include init scripts in the Docker container. In most cases, you should avoid init scripts and instead make customizations through Docker directly (using the Dockerfile). However, certain tasks must be executed when the container starts, instead of when the container is built. Use an init script for these tasks.

For example, suppose you want to run a security daemon inside a custom container. Install and build the daemon in the Docker image through your image building pipeline. Then, add an init script that starts the daemon. In this example, the init script would include a line like systemctl start my-daemon.

In the API, you can specify init scripts as part of the compute spec as follows. For more information, see the Clusters API.

Bash

"init_scripts": [
    {
        "file": {
            "destination": "file:/my/local/file.sh"
        }
    }
]

For Databricks Container Services images, you can also store init scripts in cloud storage.

The following steps take place when you launch a compute that uses Databricks Container Services:

  1. VMs are acquired from the cloud provider.
  2. The custom Docker image is downloaded from your repo.
  3. Databricks creates a Docker container from the image.
  4. Databricks Runtime code is copied into the Docker container.
  5. The init scripts are executed. See What are init scripts?.

Databricks ignores the Docker CMD and ENTRYPOINT primitives.

Enable Container Services​

To use custom containers on your compute, a workspace admin must enable Databricks Container Services.

Workspace admins can enable Databricks Container Service using the Databricks CLI. In a JSON request body, specify enableDcs to true, as in the following example:

Bash

databricks workspace-conf set-status \
--json '{"enableDcs": "true"}'

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4