Skip to main content

gcp_spanner_cdc

BETA

This component is mostly stable but breaking changes could still be made outside of major version releases if a fundamental problem with the component is found.

Consumes Spanner Change Stream Events from a GCP Spanner instance.

Introduced in version 1.11.0.

# Common config fields, showing default values
input:
  label: ""
  gcp_spanner_cdc:
    spanner_dsn: projects/{projectId}/instances/{instanceId}/databases/{databaseName} # No default (required)
    stream_name: "" # No default (required)
    heartbeat_interval: 3s
    start_time: 2006-01-02T15:04:05Z07:00 # No default (optional)
    end_time: 2006-01-02T15:04:05Z07:00 # No default (optional)
    prefetch_count: 1024

For information on how to set up credentials check out this guide.

Event Data Structure​

The data structure of the events emitted by this input can be found here:

Metadata​

This input adds the following metadata fields to each message:

- gcp_spanner_commit_timestamp - The time the records were committed in spanner.
- gcp_spanner_cdc_mod_type - The type of modification that occurred (INSERT, UPDATE, DELETE).
- gcp_spanner_table_name - The name of the table that was modified.
- gcp_spanner_cdc_server_transaction_id - The server transaction ID of the change.
- gcp_spanner_cdc_record_sequence - The sequence number of the record in the change stream.

Fields​

spanner_dsn​

The dsn for spanner from where to read the changestream.

Type: string

# Examples

spanner_dsn: projects/{projectId}/instances/{instanceId}/databases/{databaseName}

stream_name​

The name of the stream to track changes on.

Type: string

heartbeat_interval​

An optional field to configure the heartbeat interval for partitions.

Type: string
Default: "3s"

start_time​

An optional field to define the start point to read from the changestreams. If not set then the current time is used. The timestamp format should conform to RFC3339, for details on valid start times please see this document

Type: string

# Examples

start_time: 2006-01-02T15:04:05Z07:00

end_time​

An optional field to define the end time to read from the changestreams, timestamp format should conform to RFC3339

Type: string

# Examples

end_time: 2006-01-02T15:04:05Z07:00

prefetch_count​

The maximum number of messages to have pulled in.

Type: int
Default: 1024

# Examples

prefetch_count: 1024

credentials​

Optional manual configuration of GCP credentials to use. More information can be found in this document.

Type: object
Requires version 1.22.0 or newer

credentials.impersonate_service_account​

The email address of a service account to impersonate. The base credentials (Application Default Credentials) must be granted roles/iam.serviceAccountTokenCreator on this service account, or on the first delegate when impersonate_delegates is set.

Type: string
Default: ""

# Examples

impersonate_service_account: target@my-project.iam.gserviceaccount.com

credentials.impersonate_delegates​

An optional chain of service account email addresses to delegate through when impersonating. Each service account must be granted roles/iam.serviceAccountTokenCreator on the next service account in the chain, with the final one granted on impersonate_service_account.

Type: array
Default: []