Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from http://cloud.google.com/bigquery/docs/object-table-remote-function below:

Analyze object tables by using remote functions | BigQuery

Stay organized with collections Save and categorize content based on your preferences.

This document describes how to analyze unstructured data in object tables by using remote functions.

You can analyze the unstructured data represented by an object table by using a remote function. A remote function lets you call a function running on Cloud Run functions or Cloud Run, which you can program to access resources such as:

• Google's pre-trained AI models, including Cloud Vision API and Document AI.
• Open source libraries such as Apache Tika.
• Your own custom models.

To analyze object table data by using a remote function, you must generate and pass in signed URLs for the objects in the object table when you call the remote function. These signed URLs are what grant the remote function access to the objects.

• To create the connection resource used by the remote function, you need the following permissions:

  • bigquery.connections.create
  • bigquery.connections.get
  • bigquery.connections.list
  • bigquery.connections.update
  • bigquery.connections.use
  • bigquery.connections.delete

• To create a remote function, you need the permissions associated with the Cloud Functions Developer or Cloud Run Developer roles.

• To invoke a remote function, you need the permissions described in Remote functions.

• To analyze an object table with a remote function, you need the bigquery.tables.getData permission on the object table.

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Verify that billing is enabled for your Google Cloud project.

4. Enable the BigQuery, BigQuery Connection API, Cloud Run functions APIs.

   Enable the APIs

5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

6. Verify that billing is enabled for your Google Cloud project.

7. Enable the BigQuery, BigQuery Connection API, Cloud Run functions APIs.

   Enable the APIs

8. Ensure that your BigQuery administrator has created a connection and set up access to Cloud Storage.

For general instructions on creating a remote function, see Working with remote functions.

When you create a remote function to analyze object table data, you must pass in signed URLS that have been generated for the objects in the object table. You can do this by using an input parameter with a STRING data type. The signed URLS are made available to the remote function as input data in the calls field of the HTTP POST request. An example of a request is:

{
  // Other fields omitted.
  "calls": [
    ["https://storage.googleapis.com/mybucket/1.pdf?X-Goog-SignedHeaders=abcd"],
    ["https://storage.googleapis.com/mybucket/2.pdf?X-Goog-SignedHeaders=wxyz"]
  ]
}

You can read an object in your remote function by using a method that makes an HTTP GET request to the signed URL. The remote function can access the object because the signed URL contains authentication information in its query string.

When you specify the CREATE FUNCTION statement for the remote function, we recommend that you set the max_batching_rows option to 1 in order to avoid Cloud Run functions timeout and increase processing parallelism.

The following Cloud Run functions Python code example reads storage objects and returns their content length to BigQuery:

import functions_framework
import json
import urllib.request

@functions_framework.http
def object_length(request):
  calls = request.get_json()['calls']
  replies = []
  for call in calls:
    object_content = urllib.request.urlopen(call[0]).read()
    replies.append(len(object_content))
  return json.dumps({'replies': replies})

Deployed, this function would have an endpoint similar to https://us-central1-myproject.cloudfunctions.net/object_length.

The following example shows how to create a BigQuery remote function based on this Cloud Run functions function:

CREATE FUNCTION mydataset.object_length(signed_url STRING) RETURNS INT64
REMOTE WITH CONNECTION `us.myconnection`
OPTIONS(
  endpoint = "https://us-central1-myproject.cloudfunctions.net/object_length",
  max_batching_rows = 1
);

For step-by-step guidance, see Tutorial: Analyze an object table with a remote function.

To call a remote function on object table data, reference the remote function in the select_list of the query, and then call the EXTERNAL_OBJECT_TRANSFORM function in the FROM clause to generate the signed URLs for the objects.

The following example shows typical statement syntax:

SELECT uri, function_name(signed_url) AS function_output
FROM EXTERNAL_OBJECT_TRANSFORM(TABLE my_dataset.object_table, ["SIGNED_URL"])
LIMIT 10000;

The following example shows how to process only a subset of the object table contents with a remote function:

SELECT uri, function_name(signed_url) AS function_output
FROM EXTERNAL_OBJECT_TRANSFORM(TABLE my_dataset.object_table, ["SIGNED_URL"])
WHERE content_type = "application/pdf";

Learn how to run inference on image object tables.

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-08-07 UTC.

[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-07 UTC."],[[["Remote functions allow for the analysis of unstructured data within object tables by interacting with services like pre-trained AI models, open-source libraries, and custom models."],["Analyzing object table data with remote functions requires passing signed URLs, which are generated for the objects and provide the necessary access authorization to the remote function."],["Specific permissions are needed to create connections, create, and invoke remote functions, as well as to access the data in the object table itself."],["When creating a remote function for object table analysis, it's recommended to set the `max_batching_rows` option to 1 to prevent Cloud Run function timeouts and enhance processing parallelism."],["To use a remote function on object table data, it must be referenced in the `SELECT` list and use `EXTERNAL_OBJECT_TRANSFORM` in the `FROM` clause to generate signed URLs, with the option to filter using `WHERE`."]]],[]]

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4