mongodb

EXPERIMENTAL

This component is experimental and therefore subject to change or removal outside of major version releases.

Executes a query and creates a message for each document received.

Introduced in version 1.0.0.

Common

# Common config fields, showing default values
input:
  label: ""
  mongodb:
    url: mongodb://localhost:27017 # No default (required)
    database: "" # No default (required)
    username: ""
    password: ""
    collection: "" # No default (required)
    query: |2 # No default (required)
        root.from = {"$lte": timestamp_unix()}
        root.to = {"$gte": timestamp_unix()}
    auto_replay_nacks: true
    batch_size: 1000 # No default (optional)
    sort: {} # No default (optional)
    limit: 0 # No default (optional)

Advanced

# All config fields, showing default values
input:
  label: ""
  mongodb:
    url: mongodb://localhost:27017 # No default (required)
    database: "" # No default (required)
    username: ""
    password: ""
    collection: "" # No default (required)
    operation: find
    json_marshal_mode: canonical
    query: |2 # No default (required)
        root.from = {"$lte": timestamp_unix()}
        root.to = {"$gte": timestamp_unix()}
    auto_replay_nacks: true
    batch_size: 1000 # No default (optional)
    sort: {} # No default (optional)
    limit: 0 # No default (optional)

Once the documents from the query are exhausted, this input shuts down, allowing the pipeline to gracefully terminate (or the next input in a sequence to execute).

Metadata

This input adds the following metadata fields to each message:

- mongo_database
- mongo_collection

You can access these metadata fields using function interpolation.

Fields

url

The URL of the target MongoDB server.

Type: string

# Examples

url: mongodb://localhost:27017

database

The name of the target MongoDB database.

Type: string

username

The username to connect to the database.

Type: string
Default: ""

password

The password to connect to the database.

Secret

This field contains sensitive information that usually shouldn't be added to a config directly, read our secrets page for more info.

Type: string
Default: ""

collection

The collection to select from.

Type: string

operation

The mongodb operation to perform.

Type: string
Default: "find"
Requires version 1.0.0 or newer
Options: find, aggregate.

json_marshal_mode

The json_marshal_mode setting is optional and controls the format of the output message.

Type: string
Default: "canonical"
Requires version 1.0.0 or newer

Option	Summary
canonical	A string format that emphasizes type preservation at the expense of readability and interoperability. That is, conversion from canonical to BSON will generally preserve type information except in certain specific cases.
relaxed	A string format that emphasizes readability and interoperability at the expense of type preservation.That is, conversion from relaxed format to BSON can lose type information.

query

Bloblang expression describing MongoDB query.

Type: string

# Examples

query: |2
    root.from = {"$lte": timestamp_unix()}
    root.to = {"$gte": timestamp_unix()}

auto_replay_nacks

Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to false these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation.

Type: bool
Default: true

batch_size

A explicit number of documents to batch up before flushing them for processing. Must be greater than 0. Operations: find, aggregate

Type: int
Requires version 1.0.0 or newer

# Examples

batch_size: 1000

sort

An object specifying fields to sort by, and the respective sort order (1 ascending, -1 descending). Note: The driver currently appears to support only one sorting key. Operations: find

Type: object
Requires version 1.0.0 or newer

# Examples

sort:
  name: 1

sort:
  age: -1

limit

An explicit maximum number of documents to return. Operations: find

Type: int
Requires version 1.0.0 or newer