Readers overview
The following is a summary of reader capabilities. For more information, see:
Sample applications for programmers for an introduction to WAEvent
WAEvent contents for change data. HP NonStop reader WAEvent fields, MySQL Reader WAEvent fields, Oracle Reader and OJet WAEvent fields, PostgreSQL Reader WAEvent fields, SQL Server readers WAEvent fields,
Parsers for discussion of many of the supported inputs
Avro Parser for discussion of AvroEvent
JSON Parser for discussion of user-defined JSON
SQL CDC replication examples and Replicating MongoDB data to Azure CosmosDB
How update and delete operations are handled in writers for discussion of "insert only"
Recovering applications for detailed information about recovery and its limitations
Note
Unless otherwise specified, Striim does not support reading from sources or cloud services whose endpoints are located in AWS GovCloud, Azure Government, or Google for Government.
Readers summary table
reader | input(s) | output stream type(s) | supports replication | recoverable |
|---|---|---|---|---|
Data stored as files or BLOBs in a storage container in Azure Data Lake Storage (ADLS). | JSONNodeEvent | no | A1P | |
Cosmos DB Reader (see Azure Cosmos DB using Core (SQL) API CDC reader) | Initial Load mode: Cosmos DB documents using Microsoft Azure Cosmos SDK for Azure CosmosDB SQL API Incremental mode: Cosmos DB documents using Cosmos DB's change feed | JSONNodeEvent | requires upsert support in writer, so only to Cosmos DB Writer and MongoDB Writer | yes, but see Cosmos DB Reader limitations |
JDBC from a supported DBMS (see Database Reader) | WAEvent | insert only | if output is persisted to a Kafka stream (or use Incremental Batch Reader instead) | |
Apache access log, Avro, binary, delimited text, free-form text (using RegEx), GoldenGate trail file, JSON, name-value pairs, Parquet, XML | AvroEvent (when input is Avro), user-defined JSON (when input is JSON), ParquetEvent (when input is Parquet) or WAEvent | for GoldenGate only | yes | |
Apache access log, Avro, binary, delimited text, free-form text (using RegEx), JSON, name-value pairs, Parquet, XML | JSONNodeEvent, ParquetEvent, user-defined, WAEvent, XMLNodeEvent | no | A1P | |
Data from a Google Ads instance | WAEvent | no | only available for objects with a valid check column and is not available when the resync option is enabled. | |
Apache access log, binary, delimited text, free-form text (using RegEx), JSON, name-value pairs, Parquet, XML | user-defined JSON (when input is JSON), ParquetEvent (when input is Parquet) or WAEvent | no | yes | |
NonStop TMF audit trail | WAEvent | yes | yes | |
Apache access log, binary, free-form text (using RegEx), JSON, name-value pairs, XML | WAEvent or user-defined JSON (when input is JSON) | no | if output is persisted to a Kafka stream | |
Standard HubSpot objects | WAEvent | no | A1P | |
JDBC from same sources as Database Reader | WAEvent | insert only | yes | |
Data from an Intercom instance. | WAEvent | no | A1P and E1P recovery for incremental load only | |
Data from a JIRA instance. | WAEvent | no | A1P | |
Apache access log, delimited text, free-form text (using RegEx), JSON, name-value pairs, XML | WAEvent or user-defined JSON (when input is JSON) | no | yes | |
Java Management Extensions (JMX) | JSONNodeEvent | no | yes | |
Apache access log, Avro, delimited text, free-form text (using RegEx), JSON, name-value pairs, XML | WAEvent, AvroEvent (when input is Avro), or user-defined JSON (when input is JSON) | no | yes | |
Apache access log, binary, delimited text, free-form text (using RegEx), JSON, name-value pairs, XML | WAEvent or user-defined JSON (when input is JSON) | no | yes | |
MariaDB Reader (see MariaDB / SkySQL CDC readers) | MariaDB Galera Cluster binary log (binlog) | WAEvent | yes | yes |
Mongo Cosmos DB Reader (see Azure Cosmos DB using Cosmos DB API for MongoDB CDC reader) | Initial Load mode: Azure Cosmos DB documents using mongo-driver-sync Incremental mode: Cosmos DB documents using Azure Cosmos DB API for MongoDB's change stream | JSONNodeEvent | insert and delete only requires upsert support in writer, so only to Cosmos DB Writer and MongoDB Writer | |
MongoDB Reader (see MongoDB CDC reader) | MongoDB replica set operations log (oplog.rs) | JSONNodeEvent | yes | yes |
Avro, delimited text, JSON, name-value pairs, XML | WAEvent, AvroEvent (when input is Avro), or user-defined JSON (when input is JSON) | no | yes | |
MS SQL Reader / MSJet (see SQL Server CDC readers) | SQL Server transaction log | WAEvent | yes | yes |
Apache access log, Avro, binary, delimited text, free-form text (using RegEx), JSON, name-value pairs, XML | WAEvent, AvroEvent (when input is Avro), or user-defined JSON (when input is JSON) | no | if output is persisted to a Kafka stream | |
MySQL Reader (see MySQL CDC reader) | MySQL binary log (binlog) | WAEvent | yes | yes |
an OPC-UA server | OPCUA Data Change Event | no | yes | |
OJet (see Oracle Database CDC readers) | Oracle logs | WAEvent | yes | yes |
Oracle Reader (see Oracle Database CDC readers) | Oracle logs | WAEvent | yes | yes |
PostgreSQL Reader (see PostgreSQL CDC reader) | replication slot | WAEvent | yes | yes |
Apache access log, Avro, binary, delimited text, free-form text (using RegEx), JSON, name-value pairs, Parquet, XML | AvroEvent (when input is Avro), user-defined JSON (when input is JSON) ParquetEvent (when input is Parquet), or WAEvent | no | yes | |
Salesforce CDC (Streaming API) using CometD | WAEvent | yes | yes (limited by retention of the Salesforce side) | |
Force.com REST API | WAEvent | yes | yes | |
Force.com REST API | WAEvent | yes | yes | |
Salesforce platform event message subscription | WAEvent | insert only | yes | |
Salesforce Streaming API | WAEvent | yes | yes | |
ServiceNow tables | WAEvent | insert and update only | yes | |
Snowflake tables using Snowflake JDBC | WAEvent | yes | yes | |
SQL Server | see MS SQL Reader / MS Jet, above | |||
Stripe API | WAEvent | no | yes (based on the value of the Incremental Load marker) | |
Apache access log, binary, delimited text, free-form text (using RegEx), JSON, name-value pairs, XML | WAEvent or user-defined JSON (when input is JSON) | no | if output is persisted to a Kafka stream | |
Teradata | see Database Reader, above | |||
Apache access log, binary, collectd, delimited text, free-form text (using RegEx), JSON, Kafka stream, name-value pairs, NetFlow v5 or v9, XML | WAEvent, CollectdEvent (when input is collectd), or user-defined JSON (when input is JSON) | no | if output is persisted to a Kafka stream | |
Windows Application, Security, or System event log | WindowsLogEvent | no | yes | |
Yellowbrick | see Database Writer, above | |||
replication slot | WAEvent | yes | yes | |
Zendesk Support API | WAEvent | no | A1P and E1P for incremental load only |