Skip to main content

gcp_spanner_cdc

BETA

This component is mostly stable but breaking changes could still be made outside of major version releases if a fundamental problem with the component is found.

Consumes Spanner Change Stream Events from a GCP Spanner instance.

Introduced in version 1.11.0.

# Config fields, showing default values
input:
  label: ""
  gcp_spanner_cdc:
    spanner_dsn: projects/{projectId}/instances/{instanceId}/databases/{databaseName} # No default (required)
    stream_name: "" # No default (required)
    heartbeat_interval: 3s
    start_time: 2006-01-02T15:04:05Z07:00 # No default (optional)
    end_time: 2006-01-02T15:04:05Z07:00 # No default (optional)
    prefetch_count: 1024

For information on how to set up credentials check out this guide.

Event Data Structure​

The data structure of the events emitted by this input can be found here:

Metadata​

This input adds the following metadata fields to each message:

- gcp_spanner_commit_timestamp - The time the records were committed in spanner.
- gcp_spanner_cdc_mod_type - The type of modification that occurred (INSERT, UPDATE, DELETE).
- gcp_spanner_table_name - The name of the table that was modified.
- gcp_spanner_cdc_server_transaction_id - The server transaction ID of the change.
- gcp_spanner_cdc_record_sequence - The sequence number of the record in the change stream.

Fields​

spanner_dsn​

The dsn for spanner from where to read the changestream.

Type: string

# Examples

spanner_dsn: projects/{projectId}/instances/{instanceId}/databases/{databaseName}

stream_name​

The name of the stream to track changes on.

Type: string

heartbeat_interval​

An optional field to configure the heartbeat interval for partitions.

Type: string
Default: "3s"

start_time​

An optional field to define the start point to read from the changestreams. If not set then the current time is used. The timestamp format should conform to RFC3339, for details on valid start times please see this document

Type: string

# Examples

start_time: 2006-01-02T15:04:05Z07:00

end_time​

An optional field to define the end time to read from the changestreams, timestamp format should conform to RFC3339

Type: string

# Examples

end_time: 2006-01-02T15:04:05Z07:00

prefetch_count​

The maximum number of messages to have pulled in.

Type: int
Default: 1024

# Examples

prefetch_count: 1024