Skip to main content

gcp_cloud_storage

Downloads objects within a Google Cloud Storage bucket, optionally filtered by a prefix.

# Common config fields, showing default values
input:
  label: ""
  gcp_cloud_storage:
    bucket: "" # No default (required)
    prefix: ""
    scanner:
      to_the_end: {}

Metadata​

This input adds the following metadata fields to each message:

- gcs_key
- gcs_bucket
- gcs_last_modified
- gcs_last_modified_unix
- gcs_content_type
- gcs_content_encoding
- All user defined metadata

You can access these metadata fields using function interpolation.

Credentials​

By default Bento will use a shared credentials file when connecting to GCP services. You can find out more in this document.

Fields​

bucket​

The name of the bucket from which to download objects.

Type: string

prefix​

An optional path prefix, if set only objects with the prefix are consumed. This field supports interpolation functions.

Type: string
Default: ""

scanner​

The scanner by which the stream of bytes consumed will be broken out into individual messages. Scanners are useful for processing large sources of data without holding the entirety of it within memory. For example, the csv scanner allows you to process individual CSV rows without loading the entire CSV file in memory at once.

Type: scanner
Default: {"to_the_end":{}}

delete_objects​

Whether to delete downloaded objects from the bucket once they are processed.

Type: bool
Default: false

credentials​

Optional manual configuration of GCP credentials to use. More information can be found in this document.

Type: object
Requires version 1.22.0 or newer

credentials.impersonate_service_account​

The email address of a service account to impersonate. The base credentials (Application Default Credentials) must be granted roles/iam.serviceAccountTokenCreator on this service account, or on the first delegate when impersonate_delegates is set.

Type: string
Default: ""

# Examples

impersonate_service_account: target@my-project.iam.gserviceaccount.com

credentials.impersonate_delegates​

An optional chain of service account email addresses to delegate through when impersonating. Each service account must be granted roles/iam.serviceAccountTokenCreator on the next service account in the chain, with the final one granted on impersonate_service_account.

Type: array
Default: []