Striim Platform 5.4.2 documentation

Databricks Writer supports multiple authentication mechanisms. Choose based on your deployment model and credential-management strategy.

See Service Principal Authentication for step-by-step setup.

In Merge mode, Striim updates Databricks to reflect all source changes, keeping Databricks synchronized with the source.

Requirement: The Databricks table must have a primary key or unique constraint matching the source. Otherwise, Striim cannot identify which row to update or delete.

Result: The Databricks table mirrors the source table at any point in time.

Example: If your source table has ID=1, Name='Alice', Age=30, and you update Age to 31 in the source, Databricks is updated to reflect the new value: ID=1, Name='Alice', Age=31.

When to use: Real-time analytics, operational reporting, compliance snapshots where data must match the source.

In Append-Only mode, Striim treats all source changes (inserts, updates, and deletes) as INSERT operations, preserving a complete history.

Result: The Databricks table contains a complete history of all changes. This allows you to see how data has evolved over time.

Example: Using the same source data, if you update Alice's age to 31, Databricks now contains both rows:

This allows you to see the age change as a separate historical entry.

When to use: Audit trails (all changes visible), historical analysis (track all versions), time-series reporting (month-over-month comparisons).

The uploadPolicy property controls when Striim flushes staged data and executes COPY INTO.

Default: eventcount:50000,interval:120s

Format: eventcount:N,interval:Ms

Example: eventcount:100000,interval:60s means flush when 100,000 events accumulate OR 60 seconds elapse (whichever comes first).

For detailed tuning guidance, see Upload Policy and Flush Behavior in Runtime Considerations.

Control how Striim retries failed connections.

Default: initialRetryDelay=10s, retryDelayMultiplier=2, maxRetryDelay=1m, maxAttempts=5, totalTimeout=10m

Format:

Striim does not write directly to Databricks tables. Instead, Striim stages data in an external storage area (GCS, S3, ADLS, DBFS, or Personal Staging Location), then uses the Databricks COPY INTO command to load batches into Delta Lake tables.

Write process:

Advantages: Batching improves throughput, guarantees ACID consistency (entire batch succeeds or fails), and allows validation before committing.

Implications: Writes are not instantaneous (delayed by uploadPolicy interval). If the application crashes before COPY INTO completes, staged files remain in the staging area.

The uploadPolicy property controls when Striim flushes staged data.

Default: eventcount:50000,interval:120s

Format: eventcount:N,interval:Ms

Example: eventcount:100000,interval:60s means:

Tuning guidance:

High throughput sources: Increase eventcount to reduce staging writes. Example: eventcount:500000,interval:300s (fewer but larger batches).

Low latency requirements: Decrease interval for faster data arrival in Databricks. Example: eventcount:10000,interval:30s (more frequent flushes).

Storage-constrained: Decrease eventcount to avoid large accumulated files. Example: eventcount:10000,interval:120s (smaller batches).

When you stop a Databricks Writer application:

Result: No data is lost. On restart, the application resumes from the checkpoint.

If the Striim server crashes or loses network connectivity:

On recovery (when the application restarts):

Important: You may have duplicate data if the checkpoint was not at the point of failure. Striim guarantees at-least-once delivery. If duplicates are unacceptable, use Merge mode with a primary key.

Staged files are deleted automatically after successful COPY INTO. However, if the application crashes before COPY INTO completes, or if COPY INTO fails, staged files may remain.

Best practice: Set storage lifecycle policies to delete old files:

GCS: Use lifecycle rules to delete objects older than 7 days:

gsutil lifecycle set - << 'EOF'
{
  "lifecycle": {
    "rule": [
      {
        "action": {"type": "Delete"},
        "condition": {"age": 7}
      }
    ]
  }
}
EOF

S3: Use S3 Lifecycle rules via AWS console or CLI to expire objects after 7 days.

ADLS: Use blob storage lifecycle management in the Azure portal.

When using GCS as the staging area, Striim uses the Databricks Storage Credential to authenticate COPY INTO commands. The Storage Credential is a reference to a Google Cloud service account created in Databricks.

Before COPY INTO can succeed:

If you want Striim to automatically create your GCS bucket (if it doesn't exist), the Google Cloud service account must have the storage.buckets.create permission.

If the permission is missing and the bucket doesn't exist, COPY INTO fails with "Bucket not found."

For best performance and cost:

Example: If your Databricks workspace is in us-central1, place your GCS bucket in us-central1 as well.

When using Service Principal authentication, Striim automatically obtains and refreshes OAuth access tokens:

If you need to rotate your Service Principal credentials:

Striim will use the new secret when the access token is next refreshed.

In the Striim UI:

Via API: See the Striim REST API documentation for metric query endpoints.

You can configure alerts to notify you when:

See Striim Alerts and Notifications documentation for details.

You can create multiple Databricks Writer instances in the same application to increase throughput. Each instance reads a subset of source data and writes independently to Databricks.

Configuration: In Flow Designer or TQL, set the parallelThreads property:

CREATE APPLICATION parallel_writer
OUTPUT TO Databricks (
  -- ... other properties ...
  parallelThreads: 4  -- Use 4 parallel instances
)
SELECT * FROM source;

Considerations:

Staged files consume storage. Monitor your staging area and set retention policies as described in Staged File Cleanup.

Multiple Databricks Writer applications can write to the same Databricks table safely using Delta Lake's ACID guarantees. Each COPY INTO command acquires a table lock for the duration of the operation.

Best practice: If you have multiple writers to the same table, configure each with a different uploadPolicy interval to avoid contention:

This staggers the COPY INTO operations and reduces lock wait times.

If multiple sources are updating the same Databricks table, Striim applies changes in order received (per source). Cross-source ordering is not guaranteed.

Implication: If source A and source B both update the same row, the final state in Databricks may not match either source perfectly if a specific ordering is expected.

Solution: Ensure your application architecture handles concurrent updates appropriately (use timestamps or sequence numbers to determine correct order).