Databricks Writer
Databricks Writer writes data to Delta Lake tables in Databricks on AWS, Azure, or Google Cloud. Delta Lake is an open-source storage format that provides ACID transactions, versioning, and time-travel queries—capabilities typically found in relational databases.
Striim's Databricks Writer accepts data from any Striim source: transactional databases (Oracle, SQL Server), cloud applications (Salesforce, ServiceNow), NoSQL systems (MongoDB, Cosmos DB), object storage (Amazon S3, Google Cloud Storage), and other supported sources. You can use Databricks Writer to populate Delta Lake tables for analytics, reporting, real-time dashboards, and data warehousing.
Databricks Writer feature summary
Feature | Description |
|---|---|
Supported sources | Databricks Writer accepts data from all sources supported by Striim. |
Authentication | AWS/Azure: Personal Access Token (PAT) Google Cloud: Personal Access Token (PAT) or Service Principal (M2M OAuth) Service Principal-based Machine-to-Machine OAuth is supported for Databricks on all cloud platforms. See Service Principal Authentication for setup details. |
Supported write modes | Merge: Striim inserts, updates, and deletes records in Databricks to match source changes. The Databricks table reflects the current source data. Append Only: Striim inserts all source changes (inserts, updates, and deletes) as new rows in Databricks. Use this mode for historical analysis, compliance audit trails, or time-series tracking. |
Additional capabilities | Auto-quiesce: Striim automatically stops writing after an initial load completes (when configured with supported Readers). Schema evolution: Striim detects DDL changes in the source and propagates them to Databricks tables. |
Supported staging areas | Striim stages data before writing to Databricks. Supported staging areas depend on your cloud platform: AWS: S3 or Databricks File System (DBFS) Azure: Azure Data Lake Storage Gen2 (ADLS Gen2) or DBFS Google Cloud: Google Cloud Storage (GCS) or DBFS |
Resilience | Striim retries failed connections to avoid application halting on transient failures. See Runtime Considerations for recovery behavior. |
Performance | You can create multiple Databricks Writer instances in the same application to write in parallel. See Creating multiple writer instances (parallel threads) for configuration details. |
Configuration | Flow Designer TQL Web UI wizards (see Create a Databricks Writer Application) |
Metrics and monitoring | Striim exposes key performance metrics through its monitoring features. See Monitoring Guide for available metrics. |
Client libraries | Databricks JDBC driver 2.7.1; Databricks REST client 3.2.2 (for M2M OAuth); google-cloud-storage 2.43.1 (for GCS staging) |
Limitations | Striim writes data in batch mode using the COPY INTO command. Real-time (streaming) writes are not supported. See Limitations for additional constraints. |