To make third-party or custom code available to notebooks and jobs running on your compute resources, you can install a library. Libraries can be written in Python, Java, Scala, and R. You can upload Python, Java, and Scala libraries and point to external packages in PyPI, Maven, and CRAN repositories.
Databricks includes many common libraries in Databricks Runtime. To see which libraries are included in Databricks Runtime, look at the System Environment subsection of the Databricks Runtime release notes for your Databricks Runtime version.
Compute-scoped librariesâYou can install libraries on a compute resource so that they can be used by all notebooks and jobs running on the compute. Databricks supports Python, JAR, and R libraries. See Compute-scoped libraries.
You can install a compute-scoped library directly from the following sources:
Not all locations are supported for all types of libraries or all compute configurations. See Recommendations for uploading libraries for configuration recommendations.
important
Libraries can be installed from DBFS when using Databricks Runtime 14.3 LTS and below. However, any workspace user can modify library files stored in DBFS. To improve the security of libraries in a Databricks workspace, storing library files in the DBFS root is deprecated and disabled by default in Databricks Runtime 15.1 and above. See Storing libraries in DBFS root is deprecated and disabled by default.
Instead, Databricks recommends uploading all libraries, including Python libraries, JAR files, and Spark connectors, to workspace files or Unity Catalog volumes, or using library package repositories. If your workload does not support these patterns, you can also use libraries stored in cloud object storage.
For complete library support information, see Python library support, Java and Scala library support, and R library support.
Recommendations for uploading librariesâDatabricks supports most configuration installations of Python, JAR, and R libraries, but there are some unsupported scenarios. It is recommended that you upload libraries to source locations that support installation onto compute with standard access mode (formerly shared access mode), as this is the recommended mode for all workloads. See Access modes. When scheduling jobs with standard access mode run the job with a service principal.
important
Only use compute with dedicated access mode (formerly single user access mode) if required functionality is not supported by standard access mode. No isolation shared access mode is a legacy configuration on Databricks that is not recommended.
The following table provides recommendations organized by Databricks Runtime version and Unity Catalog enablement.
Python library supportâThe following table indicates Databricks Runtime version compatibility for Python wheel files for different compute access modes based on the library source location. See Databricks Runtime release notes versions and compatibility and Access modes.
In Databricks Runtime 15.0 and above, you can use requirements.txt files to manage your Python dependencies. These files can be uploaded to any supported source location.
note
Installing Python egg files is only supported on Databricks Runtime 13.3 LTS and below, and only for dedicated or no isolation shared access modes. In addition, you cannot install Python egg files on volumes or workspace files. Use Python wheel files or install packages from PyPI instead.
Java and Scala library supportâThe following table indicates Databricks Runtime version compatibility for JAR files for different compute access modes based on the library source location. See Databricks Runtime release notes versions and compatibility and Access modes.
For details on how to deploy Scala JAR files on a Unity Catalog-enabled cluster in standard access mode, see Deploy Scala JARs on Unity Catalog clusters. Note that on Unity Catalog standard clusters, classes in JAR libraries must be in a named package, such as com.databricks.MyClass, or errors will occur when importing the library.
The following table indicates Databricks Runtime version compatibility for CRAN packages for different compute access modes. See Databricks Runtime release notes versions and compatibility and Access modes.
Installer identityâWhen you install a library from Workspace files or Unity Catalog volumes, an identity may be associated with the installation depending on the compute access mode. The identity must have read access on the library file.
Notebook-scoped librariesâNotebook-scoped libraries, available for Python and R, allow you to install libraries and create an environment scoped to a notebook session. These libraries do not affect other notebooks running on the same compute. Notebook-scoped libraries do not persist and must be re-installed for each session. Use notebook-scoped libraries when you need a custom environment for a specific notebook.
note
JARs cannot be installed at the notebook level.
important
Workspace libraries have been deprecated and should not be used. See Workspace libraries (legacy). However, storing libraries as workspace files is distinct from workspace libraries and is still fully supported. You can install libraries stored as workspace files directly to compute or job tasks.
Python environment managementâThe following table provides an overview of options you can use to install Python libraries in Databricks.
Python library precedenceâYou might encounter a situation where you need to override the version for a built-in library, or have a custom library that conflicts in name with another library installed on the compute resource. When you run import <library>, the library with the high precedence is imported.
important
Libraries stored in workspace files have different precedence depending on how they are added to the Python sys.path. A Databricks Git folder adds the current working directory to the path before all other libraries, while notebooks outside Git folders add the current working directory after other libraries are installed. If you manually append workspace directories to your path, these always have the lowest precedence.
The following list orders precedence from highest to lowest. In this list, a lower number means higher precedence.
%pip install in notebooks).sys.path.RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4