databricks-configure-target
Use Connection Profile: Enable if you have a Databricks connection profile you want to use or want to create one. If you use a connection profile for Databricks, you must also use a connection profile for your ADLS or S3 staging area.
Connection Profile Name: If you enable Use Connection Profile, select the connection profile here.
Connection URL: If you do not enable Use Connection Profile, enter the JDBC URL from the JDBC/ODBC tab of the Databricks cluster's Advanced options (see Get connection details for a cluster). If the URL starts with
jdbc:spark://change that tojdbc:databricks://.Personal Access Token: If you do not enable Use Connection Profile, see Generate a personal access token). The user associated with the token must have read and write access to DBFS (see Important information about DBFS permissions). If table access control has been enabled, the user must also have MODIFY and READ_METADATA (see Data object privileges - Data governance model).
Catalog Name: Enter the name of the Databricks catalog containing the database(s) to be written to.
How do you want Striim to respond to schema changes in the source database? This option appears when the source supports schema evolution. See Handling schema evolution for discussion of the options.
Choose Automatically propagate schema changes to the target to set the source's CDDL Capture property to True, its CDDL Action property to Process, and the target's CDDL Action property to Process.
Choose Do not propagate schema changes to the target and keep the pipeline running to set the source's CDDL Capture property to True, its CDDL Action property to Process, and the target's CDDL Action property to Ignore.
Choose Pause the pipeline when the schema changes to set the source's CDDL Capture property to True, its CDDL Action property to Quiesce, and the target's CDDL Action property to Process.
How would you like to write continuous changes to Databricks? See Choose which writing mode to use for discussion of Merge mode versus Append Only mode.
Choose Write continuous changes directly to use Merge mode.
Choose Write continuous changes as audit records to use Append Only mode.
If you choose this option, Striim will add two columns to each target table:
STRIIM__OPTIMEwill contain a timestamp of when the data was loaded from the source, andSTRIIM__OPTYPEwill contain a string describing the operation for the record: SELECT for initial load and INSERT, UPDATE, or DELETE for CDC.
Select external stage type: See Choose which staging area to use.
Caution
Support for personal staging locations has been deprecated by Databricks (see (Optional) Create metastore-level storage) and Microsoft (see (Optional) Create metastore-level storage).
To stage to the Databricks File System (not recommended), set to
DBFSROOT.To stage to Azure Storage, set to
ADLS Gen2and set the following properties. To authenticate using Entra ID (formerly Azure AD) or SAS, create a connection profile.Azure Account Access Key: Specify the account access key from Storage accounts > <account name> > Access keys. For more information see Azure / Learn / Storage / Manage storage account access keys.
Azure Account Name: Specify the name of the storage account.
Azure Container Name: Specify the name of the ADLS container (also called the "file system") to be used as the staging area. If it does not exist, it will be created automatically.
To stage to S3, set to
S3and set the following properties:S3 Access Key: Specify an AWS access key ID (created on the AWS Security Credentials page) for a user with read and write permissions on the bucket.
S3 Secret Access Key: Specify the secret access key for the access key.
S3 Region: Specify the AWS region of the bucket.
S3 Bucket Name: Specify the S3 bucket to be used for staging. If it does not exist, it will be created.