Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://cloud.google.com/storage/docs/batch-operations/overview below:

Storage batch operations | Google Cloud

• Discover

• Product overview
• Storage Intelligence overview

• Get started

• Terraform support for Cloud Storage
• Configure and manage Storage Intelligence

• Create buckets

• About buckets
• Create a bucket
• Bucket locations
• Storage classes
• Change the default storage class of a bucket

• Manage buckets

• List buckets
• Move and rename buckets
• Delete buckets
• Domain-named bucket verification

• Upload and download objects

• About objects
• Overview of uploads and downloads

• Manage objects

• Change an object's storage class
• List objects
• Copy, rename, and move objects
• Delete objects
• Object transcoding

• Get insights on your stored data

• Analyze stored data with Gemini

• Cache objects

• About caching

• Organize objects

• Folder types in Cloud Storage

• Control data lifecycles

• Options for controlling data lifecycles

• Make requests

• Consistency in Cloud Storage operations
• Batched requests
• Use long-running operations
• Paginate results
• URI wildcards

• Secure data

• Monitor data and usage

• Use Cloud Audit Logs with Cloud Storage
• Usage logs and storage logs
• Use Cloud Audit Logs with storage batch operations
• Use Cloud Logs with storage batch operations
• Use Cloud Audit Logs with Storage Insights
• Use gRPC client-side metrics
• Use client-side traces

• Protection, backup, and recovery

• Overview

• Mount buckets with Cloud Storage FUSE

• Overview
• Quickstart
• Install Cloud Storage FUSE
• Mount buckets
• Use metrics

• Work across products, Clouds, and platforms

• Best practices for media workloads
• Interoperability
• Use Cloud Storage with Big Data
• Integration with Google Cloud Platform services and tools

• Troubleshoot

• Troubleshooting

• AI and ML
• Application development
• Application hosting
• Compute
• Data analytics and pipelines
• Databases
• Distributed, hybrid, and multicloud
• Generative AI
• Industry solutions
• Networking
• Observability and monitoring
• Security
• Storage

• Access and resources management
• Costs and usage management
• Google Cloud SDK, languages, frameworks, and tools
• Infrastructure as code
• Migration

• Google Cloud Home
• Free Trial and Free Tier
• Architecture Center
• Blog
• Contact Sales
• Google Cloud Developer Center
• Google Developer Center
• Google Cloud Marketplace
• Google Cloud Marketplace Documentation
• Google Cloud Skills Boost
• Google Cloud Solution Center
• Google Cloud Support
• Google Cloud Tech Youtube Channel

Stay organized with collections Save and categorize content based on your preferences.

This document describes storage batch operations, a Cloud Storage capability that lets you perform operations on billions of objects in a serverless manner. Using storage batch operations, you can automate large-scale API operations on billions of objects, reducing the development time required to write and maintain scripts for each request.

To learn how to create storage batch operations jobs, see Create and manage storage batch operations jobs.

Storage batch operations let you run one of four transformations on multiple objects at once: placing an object hold, deleting an object, updating object metadata, and rewriting objects. To use storage batch operations, you create a job configuration that defines what transformations should be applied to which objects.

Creating a batch operation returns a long-running operation (LRO) that indicates the status of your request: whether the transformation has been applied to all specified objects in your request.

• Scalability: Perform transformations on millions of objects with a single storage batch operations job.
• Serverless execution: Run batch jobs in a serverless environment, eliminating the need to manage infrastructure.
• Automation: Automate complex and repetitive tasks, improving operational efficiency.
• Reduced development time: Avoid writing and maintaining complex custom scripts.
• Performance: Complete time-sensitive operations within the required time. With multiple batch jobs running concurrently on a bucket, you can process up to one billion objects within three hours.
• Automatic retries: Automatic retries for failed operations.
• Job monitoring: Detailed progress tracking to monitor the status and completion of all jobs.

When used with Storage Insights datasets, storage batch operations allow you to accomplish the following tasks:

• Security management:

  • Set encryption keys on multiple objects using the rewrite object method.
  • Apply or remove object holds to control object immutability.

• Compliance:

  • Use object holds to meet data retention requirements for regulatory compliance.
  • Delete data between specific timeframes, to meet wipeout compliance requirements.

• Data transformation: Perform bulk updates to object metadata.

• Cost optimization: Bulk delete objects in Cloud Storage buckets to reduce storage costs.

To create a storage batch operations job, you'll need to set the following job configurations. Job configurations are parameters that control how the job is defined for different processing requirements.

• Job name: A unique name to identify the storage batch operations job. This is used for tracking, monitoring, and referencing the job. Job names are alphanumeric, for example, job-01.

• Job Description (Optional): A brief description of the job's purpose. This helps with understanding and documenting the job details. For example, Deletes all objects in a bucket.

• Bucket name: The name of the storage bucket containing the objects to be processed. This is essential for locating the input data. For example, my-bucket. You can specify only one bucket name for a job.

• Object selection: The selection criteria that defines which objects to process. You can specify the criteria using any one of the following options:

  • Manifest: Create a manifest and specify its location when you create the storage batch operations job. The manifest is a CSV file, uploaded to Google Cloud, that contains one object or a list of objects that you want to process. Each row in the manifest must include the bucket and name of the object. You can optionally specify the generation of the object. If you don't specify the generation, the current version of the object is used.

    The file must include a header row of the following format:

    bucket,name,generation

    The following is an example of the manifest:

    bucket,name,generation
    bucket_1,object_1,generation_1
    bucket_1,object_2,generation_2
    bucket_1,object_3,generation_3
    

    Caution: Ensure the manifest only includes objects from the bucket provided in the storage batch operations job. Rows referencing other buckets are ignored.

    You can also create a manifest using Storage Insights datasets. For details, see Create a manifest using Storage Insights datasets.

  • Object prefixes: Specify a list of prefixes to filter objects within the bucket. Only objects with these prefixes are processed. If empty, all objects in the bucket are processed.

• Job type: Storage batch operations supports the following job types, running a single job per batch operation.

  • Object deletion: You can delete objects within a bucket. This is crucial for cost optimization, data lifecycle management, and compliance with data deletion policies.

    Caution: By default, Cloud Storage retains soft-deleted objects for a duration of seven days. If you have accidentally deleted the objects, you can restore these soft-deleted objects during this duration. However, if you have disabled soft delete for your bucket, you cannot recover deleted objects.

  • Metadata updates: You can modify the object metadata. This includes updating custom metadata, storage class, and other object properties.

  • Object hold updates: You can enable or disable object holds. Object holds prevent objects from being deleted or modified, which is essential for compliance and data retention purposes.

  • Object encryption key updates: You can manage the customer-managed encryption keys for one or more objects. This includes applying or changing encryption keys using the rewrite object method.

Storage batch operations has the following limitations:

• Storage batch operations jobs have a maximum lifetime of 14 days. Any ongoing job that doesn't complete within 14 days of its creation is automatically cancelled.

• We don't recommend running more than 20 concurrent batch operations jobs on the same bucket.

• Storage batch operations is not compatible with VPC Service Controls.

• Storage batch operations is not supported on the following buckets:

  • Buckets that have Requestor Pays enabled.

  • Buckets located in the eur4 or us-west8 regions.

• Create and manage storage batch operations jobs

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-07-02 UTC.

[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-07-02 UTC."],[],[]]

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4