Databricks Writer Initial Setup
You must perform prerequisite setup in Databricks and your cloud platform before you can configure Databricks Writer in Striim applications. This page guides you through configuring Service Principal (M2M OAuth) authentication and setting up Google Cloud Storage as a staging area.
The setup steps vary by cloud platform and authentication choice. Use the navigation guides below to find the steps that apply to your configuration.
Universal Prerequisites (All Configurations)
Before you start any setup, verify that you have:
Administrative access to your Databricks account console
Administrative access to your cloud platform (Google Cloud IAM for GCP; equivalent for AWS/Azure)
Access to your Databricks workspace
Your Databricks workspace URL (format: https://myworkspace.cloud.databricks.com)
Your cloud platform Project ID or account ID
If you lack any of these, contact your administrator before proceeding.
Quick Navigation by Platform and Authentication
Use this guide to find the sections you need.
AWS or Azure Users
Configuration | Setup Required? | Go to |
|---|---|---|
PAT authentication with S3/ADLS staging | Not Required | Skip to Striim configuration (see Databricks Writer Programmer's Reference) |
Service Principal authentication (recommended for automation) | Yes | |
Service Principal + S3/ADLS staging | Yes (Section 1 only) |
Google Cloud Users
Configuration | Setup Required? | Go to |
|---|---|---|
PAT authentication with DBFS staging | Not Required | Skip to Striim configuration (see Databricks Writer Programmer's Reference) |
Service Principal authentication (recommended for automation) | Yes | |
GCS staging (recommended for GCP) | Yes | |
Service Principal + GCS (most common on GCP) | Yes | Service Principal Authentication followed by Set Up Google Cloud Storage as Staging Location |
Visual Setup Decision Tree
┌─────────────────────────────────────────────┐
│ Which cloud platform are you using? │
└─────────────────────────────────────────────┘
│
┌────────────┼────────────┐
│ │ │
AWS/Azure GCP (Google GCP
(skip GCS Cloud) (skip all)
section) (read all)│ │ │
│ └──┬──────┬──┘
│ │ │
│ ┌─────┘ │
│ │ │
┌────────▼──┐ ┌───▼──────┐ │
│Do you want│ │Do you need│ │
│Service │ │Service │ Not Required
│Principal? │ │Principal? │ (use PAT)
└─┬────────┬┘ └──┬───┬───┘
│ │ │ │
│ │ │ │
│ Not Req. Yes No
│ (use PAT) │ │
│ │ ├─────────┐
│ │ │ │
│ │ ┌─▼───┐ ┌──▼──┐
│ │ │Use │ │Use │
│ │ │GCS? │ │DBFS?│
│ │ └─┬─┬─┘ └──┬──┘
│ │ │ │ │
│ ┌──────┴────┘ └────┐ │
│ │ │ │
│ Section 1 ───────┐ │ Not Req.
│ (SP Auth) │ │ (Done)
│ Yes No
│ │ │
│ ┌──────┘ │
│ │ Section 1
│ │ (SP Auth)
│ │ +
│ │ Section 2
│ │ (GCS Setup)
│ │
│ ┌────▼─────────┐
│ │ │
│ Section 1 Section 2
│ (SP Auth) (GCS Setup)
│ │ │
└─────────┴───────────────┘
│
✓ Done
(proceed to
Create App)Section 1: Service Principal Authentication (M2M OAuth)
Applies to: AWS, Azure, Google Cloud Skip this section if: You are using Personal Access Token (PAT) authentication.
Overview
By default, Striim authenticates to Databricks using a Personal Access Token (PAT). Alternatively, you can configure Service Principal-based Machine-to-Machine (M2M) OAuth authentication. Service Principal authentication is useful when you need:
Long-lived applications (automatic token refresh prevents expiration)
Automation and CI/CD pipelines (no manual token rotation)
Compliance with credential-rotation policies
Service Principal authentication works with Databricks on AWS, Azure, and Google Cloud.
Before You Begin
You have account admin privileges in the Databricks account console
You understand basic Databricks terminology: workspace, catalog, schema, and permissions
Step 1: Create a Service Principal in Databricks
Goal: Create a Service Principal in Databricks and obtain the Client ID and Client Secret.
Steps:
Navigate to https://accounts.cloud.databricks.com.
Log in with your Databricks account admin credentials.
In the left sidebar, click User management.
Click the Service Principals tab.
Click Create Service Principal.
Enter a name for the Service Principal (for example, striim-m2m-service-principal).
Click Create.
On the Service Principal details page, copy the UUID displayed under the Service Principal name. This UUID is your Client ID.
[Screenshot: Databricks account console showing Service Principal details with UUID highlighted as Client ID]
Step 2: Generate a Client Secret
Goal: Create a Client Secret for the Service Principal.
Steps:
On the Service Principal details page, click the Credentials & Secrets tab.
Click Generate new secret.
Copy the Client Secret immediately from the popup dialog. This value will not be displayed again.
Store the Client Secret in a secure location. You will provide this value to Striim.
Important: The Client Secret is sensitive information. Use Striim's vault integration to store it securely.
[Screenshot: Databricks account console showing Client Secret generation dialog]
Step 3: Grant Workspace Access
Goal: Give the Service Principal permission to access your Databricks workspace.
Steps:
In the Databricks account console, click Workspaces in the left sidebar.
Select the workspace where Databricks Writer will run.
Click the Permissions tab.
Click Add Permissions.
Search for and select the Service Principal you created.
Assign the User permission.
Click Save.
[Screenshot: Databricks workspace permissions dialog showing Service Principal with User permission]
Step 4: Grant Catalog and Schema Permissions
Goal: Give the Service Principal permission to use the catalog and schema where Databricks Writer will write data.
Steps:
Log into your Databricks workspace.
Click Catalog in the left sidebar.
Select the catalog where Databricks Writer will write data.
Click the Permissions tab.
Click Grant permissions.
Search for and select the Service Principal by name.
Select the required permissions based on your use case:
USE CATALOG — Required to use the catalog
USE SCHEMA — Required to use the schema (grant on each schema)
CREATE TABLE — Required if Databricks Writer will create tables
MODIFY — Required for INSERT, UPDATE, and DELETE operations
Click Grant.
Important: Grant these permissions at the appropriate scope (catalog, schema, or table level). See the Databricks documentation on Unity Catalog permissions for guidance.
You're done when:
You have the Service Principal UUID (Client ID)
You have generated and saved the Client Secret
The Service Principal has User permission on the workspace
The Service Principal has required permissions (USE CATALOG, USE SCHEMA, CREATE TABLE, MODIFY) on your catalog/schema
Next step: Configure Striim to use these credentials (see Create a Databricks Writer Application), OR continue to Set Up Google Cloud Storage as Staging Location if you are on Google Cloud and need GCS staging.
Service Principal Configuration in Striim
After you have created the Service Principal and obtained the Client ID and Client Secret, configure Striim to authenticate using Service Principal credentials. See Create a Databricks Writer Application for detailed configuration steps.
For troubleshooting Service Principal errors, see Service Principal (M2M OAuth) Behavior in Runtime Considerations.
Section 2: Set Up Google Cloud Storage as Staging Location
Applies to: Google Cloud only Skip this section if: You are using AWS/Azure, OR you are using GCP but prefer DBFS staging instead of GCS.
Overview
When you use Databricks on Google Cloud, you can configure Databricks Writer to stage data in Google Cloud Storage (GCS) before loading into Delta Lake. This section guides you through the required setup steps.
What happens: Striim writes data files to your GCS bucket, then uses the Databricks COPY INTO command to load the data into Delta Lake tables. After successful load, Striim deletes the staged files.
Why use GCS?: GCS is the native cloud storage for Google Cloud, providing best performance and lowest egress costs.
Prerequisites for this section:
You have a Google Cloud project and can access Google Cloud IAM
Your Databricks workspace has Unity Catalog enabled
You have account admin access in Databricks
Understand the Permissions Model
GCS staging requires permissions in two systems: Google Cloud IAM (for GCS access) and Databricks (for COPY INTO access).
Permissions Required in Google Cloud IAM
These permissions allow the Databricks service account to access your GCS bucket.
Permission | Purpose |
|---|---|
storage.buckets.get | Read bucket metadata |
storage.objects.create | Write data files to the bucket |
storage.objects.delete | Delete staged files after successful load |
storage.objects.get | Read data files from the bucket |
storage.objects.list | List objects in the bucket |
storage.objects.update | Update object metadata |
storage.buckets.create | Create the bucket if it does not exist (optional) |
Granted to: The Google Cloud service account created in Databricks (Step 1 below), not your personal GCP service account.
Permissions Required in Databricks
These permissions allow the Databricks user or Service Principal to execute COPY INTO and work with the Storage Credential.
Permission | Scope | Purpose |
|---|---|---|
USE CATALOG | Catalog | Use the catalog |
USE SCHEMA | Schema | Use the schema |
CREATE TABLE | Schema | Create tables (if needed) |
MODIFY | Table | Perform INSERT, UPDATE, DELETE operations |
READ FILES | Storage Credential | Read from the Storage Credential during COPY INTO |
Granted to: The Databricks user or Service Principal that runs Databricks Writer.
Step 1: Create a Storage Credential in Databricks Unity Catalog
Goal: Create a Storage Credential that grants Databricks Writer permission to access your GCS bucket.
Steps:
Log into your Databricks workspace.
Click Catalog in the left sidebar.
In the left panel, expand Manage and click Credentials (under External Data).
Click Create Credential.
Set Credential Type to GCP Service Account.
Enter a Credential Name (for example, gcs-striim-staging). Save this name—you will specify it in Striim.
Click Create Credential.
Databricks creates a Google Cloud service account and displays its email address. Copy this email address (format: <name>@<project>.iam.gserviceaccount.com). You will need this in Step 2.
[Screenshot: Databricks Unity Catalog showing newly created Storage Credential with service account email]
Step 2: Grant GCS Permissions to the Databricks Service Account
Goal: Give the Databricks service account (created in Step 1) permissions to access your GCS bucket in Google Cloud IAM.
Steps:
Open the Google Cloud Console.
Select your Google Cloud project.
Navigate to IAM & Admin > IAM.
Click Grant Access.
In the New principals field, paste the service account email from Step 1.
Under Assign roles, select Storage Object Viewer.
Note
Storage Object Viewer provides minimum permissions (storage.objects.get, storage.objects.list, storage.buckets.get). If you want Striim to auto-create the GCS bucket, also grant Storage Object Creator.
(Optional) To restrict access to a specific bucket, click Conditions and add:
resource.name.startsWith("projects/_/buckets/<your-bucket-name>")Click Save.
[Screenshot: Google Cloud Console - IAM grant access dialog showing Storage Object Viewer role]
Step 3: Grant READ FILES Permission in Databricks
Goal: Give the Databricks user or Service Principal permission to read from the Storage Credential.
Steps:
In your Databricks workspace, click Catalog in the left sidebar.
Expand Manage and click Credentials.
Click on the Storage Credential you created in Step 1.
Click the Permissions tab.
Click Grant permissions.
Search for and select the Databricks user or Service Principal that will run Databricks Writer.
Select the READ FILES permission.
Click Grant.
[Screenshot: Databricks Storage Credential permissions dialog showing user with READ FILES permission]
You're done when:
You have created a Storage Credential in Databricks
You have granted GCS permissions to the Databricks service account in Google Cloud IAM
You have granted READ FILES permission to the Databricks user/Service Principal in Databricks
You have tested GCS connectivity (via Connection Profile test or initial app deployment)
Next step: You are ready to create a Databricks Writer application in Striim. See Create a Databricks Writer Application for detailed configuration.
For troubleshooting GCS setup errors, see GCS Staging-Specific Behavior in Runtime Considerations.
After Setup: Next Steps
Once you have completed the setup sections relevant to your configuration, you are ready to configure Databricks Writer in Striim. See Create a Databricks Writer Application for detailed guidance.