Skip to main content

gcp_pubsub

Consumes messages from a GCP Cloud Pub/Sub subscription.

# Common config fields, showing default values
input:
  label: ""
  gcp_pubsub:
    project: "" # No default (required)
    subscription: "" # No default (required)
    endpoint: ""
    sync: false
    max_outstanding_messages: 1000
    max_outstanding_bytes: 1e+09

For information on how to set up credentials check out this guide.

Metadata​

This input adds the following metadata fields to each message:

- gcp_pubsub_publish_time_unix - The time at which the message was published to the topic.
- gcp_pubsub_delivery_attempt - When dead lettering is enabled, this is set to the number of times PubSub has attempted to deliver a message.
- gcp_pubsub_message_id - The unique identifier of the message.
- gcp_pubsub_ordering_key - The ordering key of the message.
- All message attributes

You can access these metadata fields using function interpolation.

Fields​

project​

The project ID of the target subscription.

Type: string

subscription​

The target subscription ID.

Type: string

endpoint​

An optional endpoint to override the default of pubsub.googleapis.com:443. This can be used to connect to a region specific pubsub endpoint. For a list of valid values check out this document.

Type: string
Default: ""

# Examples

endpoint: us-central1-pubsub.googleapis.com:443

endpoint: us-west3-pubsub.googleapis.com:443

sync​

Enable synchronous pull mode.

Type: bool
Default: false

max_outstanding_messages​

The maximum number of outstanding pending messages to be consumed at a given time.

Type: int
Default: 1000

max_outstanding_bytes​

The maximum number of outstanding pending messages to be consumed measured in bytes.

Type: int
Default: 1000000000

create_subscription​

Allows you to configure the input subscription and creates if it doesn't exist.

Type: object

create_subscription.enabled​

Whether to configure subscription or not.

Type: bool
Default: false

create_subscription.topic​

Defines the topic that the subscription should be vinculated to.

Type: string
Default: ""

extract_tracing_map​

EXPERIMENTAL: A Bloblang mapping that attempts to extract an object containing tracing propagation information, which will then be used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer.

Type: string
Requires version 1.21.0 or newer

# Examples

extract_tracing_map: root = @

extract_tracing_map: root = this.meta.span

EXPERIMENTAL: Starts a new root span with link to parent.

Type: bool
Requires version 1.21.0 or newer

credentials​

Optional manual configuration of GCP credentials to use. More information can be found in this document.

Type: object
Requires version 1.22.0 or newer

credentials.impersonate_service_account​

The email address of a service account to impersonate. The base credentials (Application Default Credentials) must be granted roles/iam.serviceAccountTokenCreator on this service account, or on the first delegate when impersonate_delegates is set.

Type: string
Default: ""

# Examples

impersonate_service_account: target@my-project.iam.gserviceaccount.com

credentials.impersonate_delegates​

An optional chain of service account email addresses to delegate through when impersonating. Each service account must be granted roles/iam.serviceAccountTokenCreator on the next service account in the chain, with the final one granted on impersonate_service_account.

Type: array
Default: []