Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://developers.google.com/bigquery/docs/loading-data-cloud-datastore below:

Loading data from Datastore exports | BigQuery

Stay organized with collections Save and categorize content based on your preferences.

BigQuery supports loading data from Datastore exports created using the Datastore managed import and export service. You can use the managed import and export service to export Datastore entities into a Cloud Storage bucket. You can then load the export into BigQuery as a table.

To learn how to create a Datastore export file, see Exporting and importing entities in the Datastore documentation. For information on scheduling exports, see Scheduling an export.

You can control which properties BigQuery should load by setting the projectionFields property in the API or by using the --projection_fields flag in the bq command-line tool.

If you prefer to skip the loading process, you can query the export directly by setting it up as an external data source. For more information, see External data sources.

When you load data from Cloud Storage into a BigQuery table, the dataset that contains the table must be in the same region or multi-region as the Cloud Storage bucket.

When you load data into BigQuery from a Datastore export, note the following restrictions:

• You cannot use a wildcard in the Cloud Storage URI when you specify a Datastore export file.
• You can specify only one Cloud Storage URI when loading data from Datastore exports.
• You cannot append Datastore export data to an existing table with a defined schema.
• For a Datastore export to load correctly, entities in the export data must share a consistent schema with fewer than 10,000 unique property names.
• Data exported without specifying an entity filter cannot be loaded into BigQuery. The export request must include one or more kind names in the entity filter.
• The maximum field size for Datastore exports is 64 KB. When you load a Datastore export, any field larger than 64 KB is truncated.

Grant Identity and Access Management (IAM) roles that give users the necessary permissions to perform each task in this document.

To load data into BigQuery, you need IAM permissions to run a load job and load data into BigQuery tables and partitions. If you are loading data from Cloud Storage, you also need IAM permissions to access the bucket that contains your data.

To load data into a new BigQuery table or partition or to append or overwrite an existing table or partition, you need the following IAM permissions:

• bigquery.tables.create
• bigquery.tables.updateData
• bigquery.tables.update
• bigquery.jobs.create

Each of the following predefined IAM roles includes the permissions that you need in order to load data into a BigQuery table or partition:

• roles/bigquery.dataEditor
• roles/bigquery.dataOwner
• roles/bigquery.admin (includes the bigquery.jobs.create permission)
• bigquery.user (includes the bigquery.jobs.create permission)
• bigquery.jobUser (includes the bigquery.jobs.create permission)

Additionally, if you have the bigquery.datasets.create permission, you can create and update tables using a load job in the datasets that you create.

For more information on IAM roles and permissions in BigQuery, see Predefined roles and permissions.

To get the permissions that you need to load data from a Cloud Storage bucket, ask your administrator to grant you the Storage Admin (roles/storage.admin) IAM role on the bucket. For more information about granting roles, see Manage access to projects, folders, and organizations.

This predefined role contains the permissions required to load data from a Cloud Storage bucket. To see the exact permissions that are required, expand the Required permissions section:

The following permissions are required to load data from a Cloud Storage bucket:

• storage.buckets.get
• storage.objects.get
• storage.objects.list (required if you are using a URI wildcard)

You might also be able to get these permissions with custom roles or other predefined roles.

To load data from a Datastore export metadata file:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project, and then select a dataset.
3. In the Dataset info section, click add_box Create table.
4. In the Create table panel, specify the following details:
1. In the Source section, select Google Cloud Storage in the Create table from list. Then, do the following:
   1. Select a file from the Cloud Storage bucket, or enter the Cloud Storage URI. You cannot include multiple URIs in the Google Cloud console, but wildcards are supported. The Cloud Storage bucket must be in the same location as the dataset that contains the table you want to create, append, or overwrite.
      The URI for your Datastore export file must end with KIND_NAME.export_metadata or export[NUM].export_metadata. For example, in default_namespace_kind_Book.export_metadata, Book is the kind name, and default_namespace_kind_Book is the filename generated by Datastore.
   2. For File format, select Cloud Datastore Backup.
2. In the Destination section, specify the following details:
   1. For Dataset, select the dataset in which you want to create the table.
   2. In the Table field, enter the name of the table that you want to create.
   3. Verify that the Table type field is set to Native table.
3. In the Schema section, no action is necessary. The schema is inferred for a Datastore export.
4. Optional: Specify Partition and cluster settings. For more information, see Creating partitioned tables and Creating and using clustered tables.
5. Click Advanced options and do the following:
   • For Write preference, leave Write if empty selected. This option creates a new table and loads your data into it.
   • If you want to ignore values in a row that are not present in the table's schema, then select Unknown values.
   • For Encryption, click Customer-managed key to use a Cloud Key Management Service key. If you leave the Google-managed key setting, BigQuery encrypts the data at rest.
   For information about the available options, see Datastore options.
6. Click Create table.

Use the bq load command with source_format set to DATASTORE_BACKUP. Supply the --location flag and set the value to your location.

bq --location=LOCATION load \
--source_format=FORMAT \
DATASET.TABLE \
PATH_TO_SOURCE

Replace the following:

• LOCATION: your location. The --location flag is optional. For example, if you are using BigQuery in the Tokyo region, you can set the flag's value to asia-northeast1. You can set a default value for the location by using the .bigqueryrc file.
• FORMAT: DATASTORE_BACKUP.
• DATASET: the dataset that contains the table into which you're loading data.
• TABLE: the table into which you're loading data. If the table does not exist, it is created.
• PATH_TO_SOURCE: the Cloud Storage URI.

For example, the following command loads the gs://mybucket/20180228T1256/default_namespace/kind_Book/default_namespace_kind_Book.export_metadata Datastore export file into a table named book_data. mybucket and mydataset were created in the US multi-region location.

bq --location=US load \
--source_format=DATASTORE_BACKUP \
mydataset.book_data \
gs://mybucket/20180228T1256/default_namespace/kind_Book/default_namespace_kind_Book.export_metadata

Set the following properties to load Datastore export data using the API.

1. Create a load job that points to the source data in Cloud Storage.

2. Specify your location in the location property in the jobReference section of the job resource.

3. The source URIs must be fully qualified, in the format gs://[BUCKET]/[OBJECT]. The file (object) name must end in [KIND_NAME].export_metadata. Only one URI is allowed for Datastore exports, and you cannot use a wildcard.

4. Specify the data format by setting the JobConfigurationLoad.sourceFormat property to DATASTORE_BACKUP.

When you load Datastore export data into BigQuery, you can create a new table to store the data, or you can overwrite an existing table. You cannot append Datastore export data to an existing table.

If you attempt to append Datastore export data to an existing table, the following error results: Cannot append a datastore backup to a table that already has a schema. Try using the WRITE_TRUNCATE write disposition to replace the existing table.

To overwrite an existing table with Datastore export data:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project, and then select a dataset.
3. In the Dataset info section, click add_box Create table.
4. In the Create table panel, specify the following details:
1. In the Source section, select Google Cloud Storage in the Create table from list. Then, do the following:
   1. Select a file from the Cloud Storage bucket, or enter the Cloud Storage URI. You cannot include multiple URIs in the Google Cloud console, but wildcards are supported. The Cloud Storage bucket must be in the same location as the dataset that contains the table you want to create, append, or overwrite.
      The URI for your Datastore export file must end with KIND_NAME.export_metadata or export[NUM].export_metadata. For example, in default_namespace_kind_Book.export_metadata, Book is the kind name, and default_namespace_kind_Book is the filename generated by Datastore.
   2. For File format, select Cloud Datastore Backup.
Note: It is possible to modify the table's schema when you append or overwrite it. For more information about supported schema changes during a load operation, see Modifying table schemas.
2. In the Destination section, specify the following details:
   1. For Dataset, select the dataset in which you want to create the table.
   2. In the Table field, enter the name of the table that you want to create.
   3. Verify that the Table type field is set to Native table.
3. In the Schema section, no action is necessary. The schema is inferred for a Datastore export.
Note: It is possible to modify the table's schema when you append or overwrite it. For more information about supported schema changes during a load operation, see Modifying table schemas.
4. Optional: Specify Partition and cluster settings. For more information, see Creating partitioned tables and Creating and using clustered tables. You cannot convert a table to a partitioned or clustered table by appending or overwriting it. The Google Cloud console does not support appending to or overwriting partitioned or clustered tables in a load job.
5. Click Advanced options and do the following:
   • For Write preference, choose Append to table or Overwrite table.
   • If you want to ignore values in a row that are not present in the table's schema, then select Unknown values.
   • For Encryption, click Customer-managed key to use a Cloud Key Management Service key. If you leave the Google-managed key setting, BigQuery encrypts the data at rest.
   For information about the available options, see Datastore options.
6. Click Create table.

Use the bq load command with the --replace flag and with source_format set to DATASTORE_BACKUP. Supply the --location flag and set the value to your location.

bq --location=LOCATION load \
--source_format=FORMAT \
--replace \
DATASET.TABLE \
PATH_TO_SOURCE

Replace the following:

• LOCATION: your location. The --location flag is optional. For example, if you are using BigQuery in the Tokyo region, you can set the flag's value to asia-northeast1. You can set a default value for the location by using the .bigqueryrc file.
• FORMAT: DATASTORE_BACKUP.
• DATASET: the dataset containing the table into which you're loading data.
• TABLE: the table you're overwriting.
• PATH_TO_SOURCE: the Cloud Storage URI.

For example, the following command loads the gs://mybucket/20180228T1256/default_namespace/kind_Book/default_namespace_kind_Book.export_metadata Datastore export file and overwrites a table named book_data:

bq load --source_format=DATASTORE_BACKUP \
--replace \
mydataset.book_data \
gs://mybucket/20180228T1256/default_namespace/kind_Book/default_namespace_kind_Book.export_metadata

Set the following properties to load data from the API.

1. Create a load job that points to the source data in Cloud Storage.

2. Specify your location in the location property in the jobReference section of the job resource.

3. The source URIs must be fully qualified, in the format gs://[BUCKET]/[OBJECT]. The file (object) name must end in [KIND_NAME].export_metadata. Only one URI is allowed for Datastore exports, and you cannot use a wildcard.

4. Specify the data format by setting the JobConfigurationLoad.sourceFormat property to DATASTORE_BACKUP.

5. Specify the write disposition by setting the JobConfigurationLoad.writeDisposition property to WRITE_TRUNCATE.

To change how BigQuery parses Datastore export data, specify the following option:

BigQuery converts data from each entity in Datastore export files to BigQuery data types. The following table describes the conversion between data types.

RECORD

[{"lat","DOUBLE"},
 {"long","DOUBLE"}]
        

Each entity in Datastore has a unique key that contains information such as the namespace and the path. BigQuery creates a RECORD data type for the key, with nested fields for each piece of information, as described in the following table.

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-08-07 UTC.

[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-07 UTC."],[[["BigQuery can load data from Datastore exports that are stored in a Cloud Storage bucket, allowing you to analyze your Datastore entities in BigQuery."],["When creating a Datastore export for BigQuery, it is essential to specify an entity filter in the export command to ensure the data can be loaded correctly."],["Loading Datastore export data into BigQuery requires setting the source format to `DATASTORE_BACKUP` and specifying a Cloud Storage URI ending in `KIND_NAME.export_metadata`, while also ensuring that the dataset and bucket are in the same location."],["You can either create a new table with the Datastore export data or overwrite an existing table, but appending data to an existing table is not supported."],["Certain limitations exist, such as a maximum field size of 64 KB, the inability to use wildcards in the Cloud Storage URI, and the requirement for consistent schemas across exported entities."]]],[]]

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4