Create a Databricks Writer application

After completing Databricks Writer Initial Setup, you are ready to create a Databricks Writer application in Striim. This page shows you how to configure Databricks Writer using the IL-CDC Wizard, Flow Designer, or TQL.

Choose the approach that best fits your workflow:

IL-CDC Wizard: Recommended for initial load + CDC with automatic table mapping
Flow Designer: Recommended for manual configuration, complex transformations, or multiple sources/targets
TQL: Recommended for scripted deployments, version control, and advanced configurations

Approach 1: IL-CDC Wizard (Recommended for Most Users)

The IL-CDC Wizard automatically creates an application that performs an initial load of all source data, then continuously captures and applies changes to Databricks.

When to Use the Wizard

You want automatic table mapping and schema detection
You have a simple 1:1 source-to-target setup
You want to get started quickly
Your source is a supported database or Salesforce

Supported Sources

Database Reader: BigQuery, MariaDB, MySQL, Oracle, PostgreSQL, Snowflake, SQL Server
Salesforce Reader

Creating an IL-CDC Application with the Wizard

Before you begin:

You have completed Databricks Writer Initial Setup for your cloud platform
Your source is configured in Striim
You have a Databricks Connection Profile created (or credentials ready to enter)

Steps:

In Striim, navigate to Applications > Create Application > Wizards.
Select IL-CDC Wizard.
Select your source (database or Salesforce). If your source is not yet configured, click Configure Source first.
Click Next.
Select Databricks as the target.
Click Next.
Configure the Databricks connection:
- Connection Profile: Select an existing Databricks Connection Profile, or enter connection details inline
- Authentication Type: Select Personal Access Token or Service Principal
- Workspace URL: Your Databricks workspace URL (format: https://myworkspace.cloud.databricks.com)
- Token (PAT) or Client ID / Client Secret (Service Principal): From Databricks Writer Initial Setup
- Catalog: Select the target catalog (must exist)
- Schema: Select the target schema (must exist)
Click Next.
Configure Databricks Writer options:
- Write Mode: Select Merge (for operational data) or Append Only (for audit trails)
- External Staging: Select your staging area (S3, ADLS Gen2, GCS, DBFS, or Personal Staging Location)
- Storage Credential (GCS only): Enter the Storage Credential name from Databricks Writer Initial Setup
- Schema Evolution: Choose how to handle DDL changes (Process, Ignore, or Stop)
Click Next.
Review the table mappings. The wizard auto-generates a 1:1 mapping for all source tables. To customize, click Edit Mapping.
Click Create Application.
Click Deploy and Start to begin the initial load.

Expected result: The application begins initial load of all source data. Once complete, it continuously captures and applies source changes to Databricks tables. Monitor progress on the application dashboard.

For troubleshooting, see Runtime Considerations.

Approach 2: Flow Designer (For Custom Configurations)

Flow Designer gives you complete control over source selection, transformations, and target configuration. Use this approach when you need custom table mappings, multiple sources, or data transformations.

When to Use Flow Designer

You need custom table mappings or transformations
You want to combine multiple sources into one target
You need to filter or transform data before writing to Databricks
You want to write to multiple targets in the same application

Creating an Application with Flow Designer

Before you begin:

You have completed Databricks Writer Initial Setup for your cloud platform
Your sources are configured in Striim
You have a Databricks Connection Profile created (or credentials ready to enter)

Steps:

In Striim, navigate to Applications > Create Application > Flow Designer.
Click Create Application.
Enter an application name and description.
Click Create.

Add a Source Reader:

In the canvas, click + to add a component.
Select Source and choose your source type (Database Reader, Salesforce Reader, etc.).
Configure the source:
- Select your configured source connection
- Choose the tables or objects to read
- (For Database Reader) Choose Initial Load, CDC, or both
Click Save.

Add Databricks Writer:

Click + again.
Select Target and choose Databricks Writer.
Configure Databricks Writer properties:

Connection:

Select an existing Databricks Connection Profile, or enter inline:
- Authentication Type: Personal Access Token or Service Principal
- Workspace URL: Your Databricks workspace URL
- Token or Client ID/Secret: From Databricks Writer Initial Setup

Target Location:

Catalog: Target catalog name (must exist)
Schema: Target schema name (must exist)

Write Mode:

Merge or Append Only: See Write Modes and Operation Semantics in Programmer's Reference

External Staging:

Staging Type: S3, ADLS Gen2, GCS, DBFS, or Personal Staging Location
Storage Credential (GCS): From Databricks Writer Initial Setup
Staging Configuration: Properties depend on staging type (see Databricks Writer Programmer's Reference)

Schema Evolution:

Choose how Striim handles DDL changes: Process, Ignore, or Stop

Click Save.

Connect Source to Databricks Writer:

Click and drag from the source to the Databricks Writer to connect them.
Configure the connection properties (if needed):
- Table Mapping: Define which source tables map to which Databricks tables
- Column Mapping: Map source columns to Databricks columns (auto-generated by default)

Deploy and Start:

Click Deploy and Start to deploy the application.
Striim validates the configuration. Address any errors and try again.
Once deployed, the application begins reading from the source and writing to Databricks.

Expected result: Source data flows through the pipeline to Databricks tables. Monitor the application dashboard to view events processed, errors, and performance metrics.

For troubleshooting, see Runtime Considerations.

Approach 3: TQL (Text-Based Configuration)

TQL (Transformation Query Language) is Striim's SQL-like configuration language. Use TQL when you want version-controlled, scriptable application definitions.

When to Use TQL

You want version control (save in Git)
You need to automate application deployment
You prefer text-based configuration
You want repeatable, reproducible setups

TQL Syntax for Databricks Writer

Basic structure:

CREATE APPLICATION <app_name>
INPUT FROM <source_type> (
  <source_properties>
)
OUTPUT TO Databricks (
  <databricks_properties>
)
SELECT <columns> FROM <source>;

TQL Example 1: Initial Load with PAT

CREATE APPLICATION databricks_il_pat
INPUT FROM Database (
  connectionName: "oracle_source",
  username: "striim_user",
  password: "${vault.oracle_password}"
)
OUTPUT TO Databricks (
  authenticationType: "PersonalAccessToken",
  connectionUrl: "jdbc:databricks://myworkspace.cloud.databricks.com:443",
  personalAccessToken: "${vault.databricks_pat}",
  catalog: "analytics",
  schema: "raw_data",
  externalStageType: "S3",
  stagingBucket: "my-staging",
  stagingRegion: "us-east-1"
)
SELECT * FROM ORACLE_SOURCE.CUSTOMERS, ORACLE_SOURCE.ORDERS;

What happens: Striim reads CUSTOMERS and ORDERS tables from Oracle, stages to S3, and writes to Databricks tables.

TQL Example 2: CDC with Service Principal and GCS

CREATE APPLICATION databricks_cdc_sp_gcs
INPUT FROM Database (
  connectionName: "postgres_source",
  username: "striim_replication",
  password: "${vault.postgres_password}",
  cdc: true
)
OUTPUT TO Databricks (
  authenticationType: "ServicePrincipal",
  clientID: "${vault.databricks_client_id}",
  clientSecret: "${vault.databricks_client_secret}",
  connectionUrl: "jdbc:databricks://myworkspace.cloud.databricks.com:443",
  catalog: "real_time",
  schema: "streaming",
  externalStageType: "GCS",
  storageCredentialName: "gcs-striim-staging",
  gcsBucketName: "my-staging-bucket",
  gcsBucketRegion: "us-central1",
  gcsProjectId: "my-gcp-project",
  gcsServiceAccountKey: "/striim/config/gcs-key.json",
  writeMode: "Merge",
  uploadPolicy: "eventcount:100000,interval:60s"
)
SELECT * FROM POSTGRES_SOURCE.ACCOUNTS, POSTGRES_SOURCE.TRANSACTIONS;

What happens: Striim captures CDC from Postgres, stages to GCS, and merges changes into Databricks tables using Service Principal auth.

TQL Example 3: Custom Table Mapping

CREATE APPLICATION databricks_custom_map
INPUT FROM Database (
  connectionName: "mysql_source",
  cdc: true
)
OUTPUT TO Databricks (
  authenticationType: "ServicePrincipal",
  clientID: "${vault.databricks_client_id}",
  clientSecret: "${vault.databricks_client_secret}",
  connectionUrl: "jdbc:databricks://myworkspace.cloud.databricks.com:443",
  catalog: "warehouse",
  schema: "dimensions",
  externalStageType: "DBFS",
  writeMode: "Append"
)
SELECT 
  MYSQL_SOURCE.PRODUCTS.PRODUCT_ID,
  MYSQL_SOURCE.PRODUCTS.PRODUCT_NAME,
  MYSQL_SOURCE.PRODUCTS.CATEGORY,
  MYSQL_SOURCE.INVENTORY.QUANTITY_ON_HAND
FROM MYSQL_SOURCE.PRODUCTS
JOIN MYSQL_SOURCE.INVENTORY ON MYSQL_SOURCE.PRODUCTS.PRODUCT_ID = MYSQL_SOURCE.INVENTORY.PRODUCT_ID;

What happens: Striim joins PRODUCTS and INVENTORY, stages to DBFS, and appends combined result to a single Databricks table.

TQL Example 4: Using Connection Profiles

CREATE APPLICATION databricks_with_profile
INPUT FROM Database (
  connectionName: "sql_server_source",
  cdc: true
)
OUTPUT TO Databricks (
  connectionProfileName: "databricks_sp_profile",
  catalog: "operations",
  schema: "live_data",
  externalStageConnectionProfileName: "gcs_staging_profile",
  writeMode: "Merge"
)
SELECT * FROM SQL_SERVER_SOURCE.*;

What happens: Striim uses saved Connection Profiles for Databricks auth and GCS staging, reading from all SQL Server tables.

Deploying Your Application

Via Flow Designer

Click Deploy and Start in the Flow Designer canvas.
Striim validates the configuration.
The application deploys and begins processing.

Via TQL

Copy your TQL into the TQL editor.
Click Deploy.
Striim validates and deploys.
Click Start to begin processing.

Monitoring After Deployment

After deployment, monitor your application:

In Striim, navigate to Applications and select your application.
View the Dashboard to see:
- Events received (from source)
- Events written (to Databricks)
- Events failed
- Data volume processed
- Current status (Running, Paused, Stopped)
View the Logs to see detailed messages, warnings, and errors.
View the Metrics to see performance trends.

For more details, see Monitoring and Metrics in Runtime Considerations.

Deployment Troubleshooting

For common deployment issues and solutions, see Runtime Considerations.

In this section:

Striim Platform 5.4.2 documentation