Skip to main content

Striim Platform 5.0 documentation

ADLS Gen2 Writer

Note

ADLS Reader and ADLS Gen2 Writer read from and write to the same Azure Data Lake Storage (ADLS) service. Microsoft dropped "Gen2" from the name of the service after it retired the ADLS Gen1 service in February 2024.

Writes to files in an Azure Data Lake Storage file system. A common use case is to write data from on-premise sources to an ADLS staging area from which it can be consumed by Azure-based analytics tools.

When you create the ADLS storage account, set Storage account kind to StorageV2 and enable Hierarchical namespace.

ADLS Gen2 Writer properties

property

type

default value

notes

Account Name

String

the storage account name

Compression Type

String

Set to gzip when the input is in gzip format. Otherwise, leave blank.

Directory

String

The full path to the directory in which to write the files. See Setting output names and rollover / upload policies for advanced options.

File Name

String 

The base name of the files to be written. See Setting output names and rollover / upload policies.

File System Name

String

the name of the ADLS Gen2 file system (container) where the files will be written

Rollover on DDL

Boolean

True

Has effect only when the input stream is the output stream of a MySQLReader or OracleReader source. With the default value of True, rolls over to a new file when a DDL event is received. Set to False to keep writing to the same file.

SAS Token

encrypted password

The SAS token for a shared access signature for the storage account. Allowed services must include Blob, allowed resource types must include Object, and allowed permissions must include Write and Create. Remove the ? from the beginning of the SAS token. Note that SAS tokens have an expiration date. See Best practices when using SAS.

Upload Policy

String

eventcount:10000, interval:5m

See Setting output names and rollover / upload policies. Keep these settings low enough that individual uploads do not exceed the underlying Microsoft REST API's limit of 100 MB for a single operation.

For best performance, Microsoft recommends uploads between 4 and 16 MB. Setting UploadPolicy to filesize:16M will accomplish that. However, if there is a long gap between events, this will mean some events will not be written to ADLS for some time. For example, if Striim receives events only during working hours, the last events received at the end of the day on Friday would not be written until Monday morning.

When the app is stopped, any remaining data in the upload buffer is discarded.

This adapter has a choice of formatters. See Supported writer-formatter combinations for more information.Supported writer-formatter combinations

ADLS Gen2 Writer sample application

CREATE APPLICATION ADLSGen2Test;

CREATE SOURCE PosSource USING FileReader (
  wildcard: 'PosDataPreview.csv',
  directory: 'Samples/PosApp/appData',
  positionByEOF:false )
PARSE USING DSVParser (
  header:Yes,
  trimquote:false )
OUTPUT TO PosSource_Stream;

CREATE CQ PosSource_Stream_CQ
INSERT INTO PosSource_TransformedStream
SELECT TO_STRING(data[1]) AS MerchantId,
  TO_DATE(data[4]) AS DateTime,
  TO_DOUBLE(data[7]) AS AuthAmount,
  TO_STRING(data[9]) AS Zip
FROM PosSource_Stream;

CREATE TARGET ADLSGen2Target USING ADLSGen2Writer (
  accountname:'mystorageaccount',
  sastoken:'********************************************',
  filesystemname:'myfilesystem',
  directory:'mydir',
  filename:'myfile.json',
  uploadpolicy: 'interval:15s'
)
FORMAT USING JSONFormatter ()
INPUT FROM PosSource_TransformedStream;

END APPLICATION ADLSGen2Test;
In this section: