note
Databricks recommends Databricks Asset Bundles for creating, developing, deploying, and testing jobs and other Databricks resources as source code. See What are Databricks Asset Bundles?.
In this article, you learn how to automate Databricks operations and accelerate development with the Databricks SDK for Java. This article supplements the Databricks SDK for Java README, API reference, and examples.
note
This feature is in Beta and is okay to use in production.
During the Beta period, Databricks recommends that you pin a dependency on the specific minor version of the Databricks SDK for Java that your code depends on. For example, you can pin dependencies in files such as pom.xml for Maven. For more information about pinning dependencies, see Introduction to the Dependency Mechanism.
Before you begin to use the Databricks SDK for Java, your development machine must have:
In your project's pom.xml file, instruct your build system to take a dependency on the Databricks SDK for Java. To do this, add the following <dependency> to the pom.xml file's existing <dependencies> section. If the <dependencies> section does not already exist within the pom.xml file, you must also add the <dependencies> parent element to the pom.xml file.
For example, to open your project's pom.xml file in IntelliJ IDEA, click View > Tool Windows > Project, and then double-click to open your-project-name > src > pom.xml.
XML
<dependencies>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>databricks-sdk-java</artifactId>
<version>0.0.1</version>
</dependency>
</dependencies>
note
Be sure to replace 0.0.1 with the latest version of the Databricks SDK for Java. You can find the latest version in the Maven central repository.
Instruct your project to take the declared dependency on the Databricks SDK for Java. For example, in IntelliJ IDEA, in your project's Project tool window, right-click your project's root node, and then click Reload Project.
Add code to import the Databricks SDK for Java and to list all of the clusters in your Databricks workspace. For example, in a project's Main.java file, the code might be as follows:
Java
import com.databricks.sdk.WorkspaceClient;
import com.databricks.sdk.service.compute.ClusterInfo;
import com.databricks.sdk.service.compute.ListClustersRequest;
public class Main {
public static void main(String[] args) {
WorkspaceClient w = new WorkspaceClient();
for (ClusterInfo c : w.clusters().list(new ListClustersRequest())) {
System.out.println(c.getClusterName());
}
}
}
note
By not setting any arguments in the preceding call to WorkspaceClient w = new WorkspaceClient(), the Databricks SDK for Java uses its default process for trying to perform Databricks authentication. To override this default behavior, see the following authentication section.
Build your project. For example, to do this in IntelliJ IDEA, from the main menu, click Build > Build Project.
Run your main file. For example, to do this in IntelliJ IDEA for a project's Main.java file, from the main menu, click Run > Run 'Main'.
The list of clusters appears. For example, in IntelliJ IDEA, this is in the Run tool window. To display this tool window, from the main menu, click View > Tool Windows > Run.
The Databricks SDK for Java implements the Databricks client unified authentication standard, a consolidated and consistent architectural and programmatic approach to authentication. This approach helps make setting up and automating authentication with Databricks more centralized and predictable. It enables you to configure Databricks authentication once and then use that configuration across multiple Databricks tools and SDKs without further authentication configuration changes. For more information, including more complete code examples in Java, see Databricks client unified authentication.
Some of the available coding patterns to initialize Databricks authentication with the Databricks SDK for Java include:
Use Databricks default authentication by doing one of the following:
DATABRICKS_CONFIG_PROFILE environment variable to the name of the custom configuration profile.Then instantiate for example a WorkspaceClient object with Databricks default authentication as follows:
Java
import com.databricks.sdk.WorkspaceClient;
WorkspaceClient w = new WorkspaceClient();
Hard-coding the required fields is supported but not recommended, as it risks exposing sensitive information in your code, such as Databricks personal access tokens. The following example hard-codes Databricks host and access token values for Databricks token authentication:
Java
import com.databricks.sdk.WorkspaceClient;
import com.databricks.sdk.core.DatabricksConfig;
DatabricksConfig cfg = new DatabricksConfig()
.setHost("https://...")
.setToken("...");
WorkspaceClient w = new WorkspaceClient(cfg);
See also Authentication in the Databricks SDK for Java README.
Use Databricks Utilities and Java with the Databricks SDK for JavaâDatabricks Utilities provides several helper functions to make it easy to work with object storage efficiently, chain and parameterize notebooks, and work with secrets. Databricks provides a Databricks Utilities for Scala library, which you can call with Java code, to enable you to programmatically access Databricks Utilities.
To use Java code to call the Databricks Utilities for Scala, do the following:
In your Java project, declare a dependency on the Databricks SDK for Java, as described in the previous section.
Declare a dependency on the Databricks Utilities for Scala library. To do this, add the following <dependency> to the pom.xml file's existing <dependencies> section:
XML
<dependency>
<groupId>com.databricks</groupId>
<artifactId>databricks-dbutils-scala_2.12</artifactId>
<version>0.1.4</version>
</dependency>
note
Be sure to replace 0.1.4 with the latest version of the Databricks Utilities for Scala library. You can find the latest version in the Maven central repository.
Instruct your project to take the declared dependency on the Databricks Utilities for Scala. For example, in IntelliJ IDEA, in your project's Project tool window, click your project's root node, and then click Maven > Reload Project.
Add code to import and then call the Databricks Utility for Scala. For example, the following code automates a Unity Catalog volume. This example creates a file named zzz_hello.txt in the volume's path within the workspace, reads the data from the file, and then deletes the file:
Java
import com.databricks.sdk.core.DatabricksConfig;
import com.databricks.sdk.scala.dbutils.DBUtils;
public class Main {
public static void main(String[] args) {
String filePath = "/Volumes/main/default/my-volume/zzz_hello.txt";
String fileData = "Hello, Databricks!";
DBUtils dbutils = DBUtils.getDBUtils(new DatabricksConfig().setProfile("DEFAULT"));
dbutils.fs().put(filePath, fileData, true);
System.out.println(dbutils.fs().head(filePath, 18));
dbutils.fs().rm(filePath, false);
}
}
Build your project and run your main file.
The following code examples demonstrate how to use the Databricks SDK for Java to create and delete clusters, create jobs, and list account-level groups. These code examples use the Databricks SDK for Java's default Databricks authentication process.
For additional code examples, see the examples folder in the Databricks SDK for Java repository in GitHub.
This code example creates a cluster with the specified Databricks Runtime version and cluster node type. This cluster has one worker, and the cluster will automatically terminate after 15 minutes of idle time.
Java
import com.databricks.sdk.WorkspaceClient;
import com.databricks.sdk.service.compute.CreateCluster;
import com.databricks.sdk.service.compute.CreateClusterResponse;
public class Main {
public static void main(String[] args) {
WorkspaceClient w = new WorkspaceClient();
CreateClusterResponse c = w.clusters().create(
new CreateCluster()
.setClusterName("my-cluster")
.setSparkVersion("12.2.x-scala2.12")
.setNodeTypeId("i3.xlarge")
.setAutoterminationMinutes(15L)
.setNumWorkers(1L)
).getResponse();
System.out.println("View the cluster at " +
w.config().getHost() +
"#setting/clusters/" +
c.getClusterId() +
"/configuration\n");
}
}
note
For Databricks Runtime 16.0 or above, JDK 17 is generally available and the default. For Databricks Runtime versions 13.1 to 15.4, JDK 8 is the default, and JDK 17 is in Public Preview.
This section provides a guide to creating a cluster using the Java Development Kit (JDK). Learn how to create a cluster with JDK 17 to use Java in your notebooks and jobs.
When you create a cluster, specify that the cluster uses JDK 17 for both the driver and executor by adding the following environment variable to Advanced Options > Spark > Environment Variables:
If you are using ARM-based clusters (for example, AWS Graviton instances), use the following environment variable instead.
Permanently delete a clusterâThis code example permanently deletes the cluster with the specified cluster ID from the workspace.
Java
import com.databricks.sdk.WorkspaceClient;
import java.util.Scanner;
public class Main {
public static void main(String[] args) {
System.out.println("ID of cluster to delete (for example, 1234-567890-ab123cd4):");
Scanner in = new Scanner(System.in);
String c_id = in.nextLine();
WorkspaceClient w = new WorkspaceClient();
w.clusters().permanentDelete(c_id);
}
}
This code example creates a Databricks job that can be used to run the specified notebook on the specified cluster. As this code runs, it gets the existing notebook's path, the existing cluster ID, and related job settings from the user at the terminal.
Java
import com.databricks.sdk.WorkspaceClient;
import com.databricks.sdk.service.jobs.JobTaskSettings;
import com.databricks.sdk.service.jobs.NotebookTask;
import com.databricks.sdk.service.jobs.NotebookTaskSource;
import com.databricks.sdk.service.jobs.CreateResponse;
import com.databricks.sdk.service.jobs.CreateJob;
import java.util.Scanner;
import java.util.Map;
import java.util.Collection;
import java.util.Arrays;
public class Main {
public static void main(String[] args) {
System.out.println("Some short name for the job (for example, my-job):");
Scanner in = new Scanner(System.in);
String jobName = in.nextLine();
System.out.println("Some short description for the job (for example, My job):");
String description = in.nextLine();
System.out.println("ID of the existing cluster in the workspace to run the job on (for example, 1234-567890-ab123cd4):");
String existingClusterId = in.nextLine();
System.out.println("Workspace path of the notebook to run (for example, /Users/someone@example.com/my-notebook):");
String notebookPath = in.nextLine();
System.out.println("Some key to apply to the job's tasks (for example, my-key): ");
String taskKey = in.nextLine();
System.out.println("Attempting to create the job. Please wait...");
WorkspaceClient w = new WorkspaceClient();
Map<String, String> map = Map.of("", "");
Collection<JobTaskSettings> tasks = Arrays.asList(new JobTaskSettings()
.setDescription(description)
.setExistingClusterId(existingClusterId)
.setNotebookTask(new NotebookTask()
.setBaseParameters(map)
.setNotebookPath(notebookPath)
.setSource(NotebookTaskSource.WORKSPACE))
.setTaskKey(taskKey)
);
CreateResponse j = w.jobs().create(new CreateJob()
.setName(jobName)
.setTasks(tasks)
);
System.out.println("View the job at " +
w.config().getHost() +
"/#job/" +
j.getJobId()
);
}
}
This code example demonstrates various calls to files functionality within WorkspaceClient to access a Unity Catalog volume.
Java
import com.databricks.sdk.WorkspaceClient;
import com.databricks.sdk.service.files.DirectoryEntry;
import com.databricks.sdk.service.files.DownloadResponse;
import java.io.*;
import java.nio.file.Files;
import java.nio.file.Paths;
public class Main {
public static void main(String[] args) throws IOException {
String catalog = "main";
String schema = "default";
String volume = "my-volume";
String volumePath = "/Volumes/" + catalog + "/" + schema + "/" + volume;
String volumeFolder = "my-folder";
String volumeFolderPath = volumePath + "/" + volumeFolder;
String volumeFile = "data.csv";
String volumeFilePath = volumeFolderPath + "/" + volumeFile;
String uploadFilePath = "./data.csv";
WorkspaceClient w = new WorkspaceClient();
w.files().createDirectory(volumeFolderPath);
try {
File uploadFile = new File(upload_file_path);
InputStream uploadInputStream = Files.newInputStream(Paths.get(upload_file_path));
w.files().upload(volumeFilePath, uploadInputStream);
} catch (java.io.IOException e) {
System.out.println(e.getMessage());
System.exit(-1);
}
Iterable<DirectoryEntry> volumeItems = w.files().listDirectoryContents(volumePath);
for (DirectoryEntry volumeItem: volumeItems) {
System.out.println(volumeItem.getPath());
}
Iterable<DirectoryEntry> volumeFolderItems = w.files().listDirectoryContents(volumeFolderPath);
for (DirectoryEntry volumeFolderItem: volumeFolderItems) {
System.out.println(volumeFolderItem.getPath());
}
DownloadResponse resp = w.files().download(volumeFilePath);
InputStream downloadedFile = resp.getContents();
try {
BufferedReader reader = new BufferedReader(new InputStreamReader(downloadedFile));
String line;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
} catch (java.io.IOException e) {
System.out.println(e.getMessage());
System.exit(-1);
}
w.files().delete(volumeFilePath);
w.files().deleteDirectory(volumeFolderPath);
}
}
This code example lists the display names for all of the available groups within the Databricks account.
Java
import com.databricks.sdk.AccountClient;
import com.databricks.sdk.core.DatabricksConfig;
import com.databricks.sdk.service.iam.Group;
import com.databricks.sdk.service.iam.ListAccountGroupsRequest;
public class Main {
public static void main(String[] args) {
AccountClient a = new AccountClient();
for (Group g : a.groups().list((new ListAccountGroupsRequest()))) {
System.out.println(g.getDisplayName());
}
}
}
You can use Scala projects with the Databricks SDK for Java. Before you begin, your development machine must have:
sbt. If you use IntelliJ IDEA, you can choose the sbt version to use during Scala project creation.To configure, build, and run your Scala project:
In your project's build.sbt file, take a dependency on the Databricks SDK for Java library by adding the following line to the end of the file, and then save the file:
libraryDependencies += "com.databricks" % "databricks-sdk-java" % "0.2.0"
note
Be sure to replace 0.2.0 with the latest version of the Databricks SDK for Java library. You can find the latest version in the Maven central repository.
Instruct your project to take the declared dependency on the Databricks SDK for Java. For example, in IntelliJ IDEA, click the Load sbt changes notification icon.
Add code to import the Databricks SDK for Java and to list all of the clusters in your Databricks workspace. For example, in a project's Main.scala file, the code might be as follows:
Scala
import com.databricks.sdk.WorkspaceClient
import com.databricks.sdk.service.compute.ListClustersRequest
object Main {
def main(args: Array[String]): Unit = {
val w = new WorkspaceClient()
w.clusters().list(new ListClustersRequest()).forEach{
elem => println(elem.getClusterName)
}
}
}
note
By not setting any arguments in the preceding call to val w = new WorkspaceClient(), the Databricks SDK for Java uses its default process for trying to perform Databricks authentication. To override this default behavior, see the following authentication section.
Build your project. For example, to do this in IntelliJ IDEA, from the main menu, click Build > Build Project.
Run your main file. For example, to do this in IntelliJ IDEA for a project's Main.scala file, from the main menu, click Run > Run 'Main.scala'.
The list of clusters appears. For example, in IntelliJ IDEA, this is in the Run tool window. To display this tool window, from the main menu, click View > Tool Windows > Run.
Databricks Utilities provides several helper functions to make it easy to work with object storage efficiently, chain and parameterize notebooks, and work with secrets. Databricks provides a Databricks Utilities for Scala library to enable you to programmatically access Databricks Utilities with Scala.
To call the Databricks Utilities for Scala, do the following:
In your Scala project, declare a dependency on the Databricks SDK for Java, as described in the previous section.
Declare a dependency on the Databricks Utilities for Scala library. For example, in your project's build.sbt file, add the following line to the end of the file, and then save the file:
libraryDependencies += "com.databricks" % "databricks-dbutils-scala_2.12" % "0.1.4"
note
Be sure to replace 0.1.4 with the latest version of the Databricks Utilities for Scala library. You can find the latest version in the Maven central repository.
Instruct your project to take the declared dependency on the Databricks Utilities for Scala. For example, in IntelliJ IDEA, click the Load sbt changes notification icon.
Add code to import and then call the Databricks Utility for Scala. For example, the following code automates a Unity Catalog volume. This example creates a file named zzz_hello.txt in the volume's path within the workspace, reads the data from the file, and then deletes the file:
Scala
import com.databricks.sdk.scala.dbutils.DBUtils
object Main {
def main(args: Array[String]): Unit = {
val filePath = "/Volumes/main/default/my-volume/zzz_hello.txt"
val fileData = "Hello, Databricks!"
val dbutils = DBUtils.getDBUtils()
dbutils.fs.put(
file = filePath,
contents = fileData,
overwrite = true
)
println(dbutils.fs.head(filePath))
dbutils.fs.rm(filePath)
}
}
note
By not setting any arguments in the preceding call to val dbutils = DBUtils.getDBUtils(), Databricks Utilities for Scala uses its default process for trying to perform Databricks authentication.
To override this default behavior, pass an instantiated DatabricksCfg object as an argument to getDBUtils. For more information, see the preceding authentication section.
Note, however, that if your code is running inside of the Databricks Runtime, this DatabricksCfg object is ignored. This is because the Databricks Utilities for Scala delegates to the built-in Databricks Utilities when running inside of the Databricks Runtime.
Build your project and run your main file.
To access Unity Catalog volumes, use files within WorkspaceClient. See Manage files in Unity Catalog volumes. You cannot use DBUtils.getDBUtils() to access volumes.
To test your code, use Java test frameworks such as JUnit. To test your code under simulated conditions without calling Databricks REST API endpoints or changing the state of your Databricks accounts or workspaces, use Java mocking libraries such as Mockito.
For example, given the following file named Helpers.java containing a createCluster function that returns information about the new cluster:
Java
import com.databricks.sdk.WorkspaceClient;
import com.databricks.sdk.service.compute.CreateCluster;
import com.databricks.sdk.service.compute.CreateClusterResponse;
public class Helpers {
static CreateClusterResponse createCluster(
WorkspaceClient w,
CreateCluster createCluster,
String clusterName,
String sparkVersion,
String nodeTypeId,
Long autoTerminationMinutes,
Long numWorkers
) {
return w.clusters().create(
createCluster
.setClusterName(clusterName)
.setSparkVersion(sparkVersion)
.setNodeTypeId(nodeTypeId)
.setAutoterminationMinutes(autoTerminationMinutes)
.setNumWorkers(numWorkers)
).getResponse();
}
}
And given the following file named Main.java that calls the createCluster function:
Java
import com.databricks.sdk.WorkspaceClient;
import com.databricks.sdk.service.compute.CreateCluster;
import com.databricks.sdk.service.compute.CreateClusterResponse;
public class Main {
public static void main(String[] args) {
WorkspaceClient w = new WorkspaceClient();
CreateClusterResponse c = Helpers.createCluster(
w,
new CreateCluster(),
"My Test Cluster",
"<spark-version>",
"<node-type-id>",
15L,
1L
);
System.out.println(c.getClusterId());
}
}
The following file named HelpersTest.java tests whether the createCluster function returns the expected response. Rather than creating a cluster in the target workspace, this test mocks a WorkspaceClient object, defines the mocked object's settings, and then passes the mocked object to the createCluster function. The test then checks whether the function returns the new mocked cluster's expected ID.
Java
import com.databricks.sdk.WorkspaceClient;
import com.databricks.sdk.mixin.ClustersExt;
import com.databricks.sdk.service.compute.ClusterDetails;
import com.databricks.sdk.service.compute.CreateCluster;
import com.databricks.sdk.support.Wait;
import com.databricks.sdk.service.compute.CreateClusterResponse;
import org.junit.jupiter.api.Test;
import org.mockito.Mockito;
import static org.junit.jupiter.api.Assertions.assertEquals;
public class HelpersTest {
@Test
public void testCreateCluster() {
WorkspaceClient mockWorkspaceClient = Mockito.mock(WorkspaceClient.class);
ClustersExt mockClustersExt = Mockito.mock(ClustersExt.class);
CreateCluster mockCreateCluster = new CreateCluster();
Wait<ClusterDetails, CreateClusterResponse> mockWait = Mockito.mock(Wait.class);
CreateClusterResponse mockResponse = Mockito.mock(CreateClusterResponse.class);
Mockito.when(mockWorkspaceClient.clusters()).thenReturn(mockClustersExt);
Mockito.when(mockClustersExt.create(Mockito.any(CreateCluster.class))).thenReturn(mockWait);
Mockito.when(mockWait.getResponse()).thenReturn(mockResponse);
CreateClusterResponse response = Helpers.createCluster(
mockWorkspaceClient,
mockCreateCluster,
"My Test Cluster",
"<spark-version>",
"<node-type-id>",
15L,
1L
);
assertEquals(mockResponse, response);
}
}
For more information, see:
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4