Striim Platform 5.0 documentation

When enabled, Sentinel uses Striim AI to detect sensitive data in its input data stream. Striim AI uses a combination of classification algorithms, pattern matching, and analysis of the data and the metadata, including the field names, to identify sensitive information. 

Sentinel scans the input record for information that matches supported Sensitive Data Identifiers such as email addresses (EMAIL_ADDRESS), US Social Security Numbers (USA_SOCIAL_SECURITY_NUMBER), UK National Insurance Numbers (GBR_NATIONAL_INSURANCE_NUMBER), or the India Unique Identification Number (IND_AADHAAR_NUMBER). The accuracy of detection can depend on the AI engines that you use for Sentinel, and the AI engines may sometimes misclassify information. After Sentinel has identified and matched sensitive information to a supported Sensitive Data Identifier, it implements actions such as partial masking or encryption on the information that you have specified for the matching Sensitive Data Identifier.

These real-time actions on sensitive data – detection, identification, protection and event tagging (described later) – are driven by Striim AI, and collectively known as Actions on Data Identifiers because they are solely based on the source information matching a Sensitive Data Identifier, irrespective of the information’s location in the input record.

For example, in the figure below, we highlight the sensitive information that Sentinel has identified in the input record. Assume that you have configured Sentinel with the following actions in accordance with your organization’s data governance policies:

In other words, you want Sentinel to take no action on sensitive information that it classifies as NAME or PHONE_NUMBER and to let such information pass through to the output as is, while you want Sentinel to protect information that it classifies as EMAIL_ADDRESS or USA_SOCIAL_SECURITY_NUMBER and then write it to the output.

Sentinel executes your Actions on Data Identifiers as shown in the output record in the figure above.  Sentinel can detect and identify the sensitive information in your records, subject to the accuracy of the AI engines that you use for Sentinel, and then implement your specified actions on the identified sensitive information, all in real time.  You do not have to know about the sensitive information in your records or where they are located; Sentinel can detect and manage the sensitive data in the input data stream according to your specifications.  The Actions on Data Identifiers support schema evolution. If new columns, fields, files, tables, collections or objects are added to your configured source, and thereby to your application, then Sentinel can automatically detect, identify and protect sensitive information in the newly added data, thus ensuring that your application remains governed through its lifecycle.

These Actions on Data Identifiers depend solely on the type of sensitive information and not on its location in the input record. For example, in the input record in the figure below, a US Social Security Number has been mistakenly placed in the field named “Department”.  Since you have configured Sentinel to act on USA_SOCIAL_SECURITY_NUMBER, Sentinel is able to identify and partially mask this information, thereby preventing this sensitive information from flowing further downstream in its raw form.

Additionally, Sentinel can inform you about the numbers and types of sensitive information that it detected in the input record, and it can tag this event by adding this information as AIData to the Striim event type (such as WAevent or JSONNodeEvent).  AIData is conceptually similar to UserData (see: Adding user-defined data to WAEvent streams and Adding user-defined data to JSONNodeEvent streams), except that it is generated by Striim.  The output stream from Sentinel contains the event AIData, and you can query the output stream to learn about the sensitive data that Striim has detected in your record.  When you write the record to the target, you can also write the event AIData to the target so that you can know about the sensitive data that Sentinel detected in that record, else the AIData information for that record will be lost forever.

The figure above shows an example of the event AIData generated by Sentinel for the input record. The input record contains 1X NAME, 2X EMAIL _ADDRESS, 1X PHONE_NUMBER and 1X USA_SOCIAL_SECURITY_NUMBER, and Sentinel has informed you that it detected 5 occurrences (Occurrences) of 4 types of sensitive data (Identifiers) in the input record.  If you choose to tag your events with AIData, then Sentinel will always write the number and types of sensitive data types that it detected in the input record.

You can configure Sentinel to write the type and location of the sensitive data in the event to AIData.  In the figure above, Sentinel has informed you that it detected an email address in the fields named “Email.home” and “Email.work”, as you likely expected.  You can query the event AIData in the output stream from Sentinel or from the data written to the target to know if the input record contains the expected number and types of sensitive data and in their expected location.  For example, in the figure above, Sentinel informed you that it detected an US Social Security number in the field named "Department" that seems contrary to your expectations.  By querying the AIData, you can quickly know that you may have an upstream data quality issue and the records that are impacted, and you can take appropriate steps to address the problem.

You can also choose the Sensitive Data Identifiers that Sentinel must tag in the event AIData.  First, your Striim Admin must classify the supported Sensitive Data Identifiers as "Low Importance", "Medium Importance" or "High Importance", in accordance with your organization’s data governance policies.  Next, in your Sentinel configuration, you can specify the importance level of the Identifiers that Sentinel must tag in the event AIData.  For example, if you specify that all "High Importance" and "Medium Importance" Identifiers must be tagged, then Sentinel will write the location of any information that matches a Sensitive Data Identifier that your Striim Admin has classified as "High Importance" or "Medium Importance".

Sentinel supports another method to act on sensitive data.  If you are certain that your sensitive information is located in specific fields or columns in your source dataset such as a table, collection, directory or topic-partition, then you can use Sentinel’s Actions on Fields to take actions such as masking or encryption on the data located in those fields or columns.

For example, assume that you know that the EMP tables in your source database contain columns such as "SSN" and "Bank Account No." that you want to obfuscate before the data in these columns flows further downstream.  You can use Sentinel’s Actions on Fields to take actions such as masking or encryption on all data in the named columns such as "SSN" and "Bank Account No." in a specific table and take no action on data in other columns, as shown in the figure below.

You can configure a single Sentinel component to act on data in multiple columns in multiple tables, with a different action for each named field. As the name suggests, the Actions on Fields are designed solely to act on data in one or more specified fields. These Actions do not use Striim AI to detect sensitive data in every event in the the input data stream; therefore, they offer superior performance compared to Actions on Data Identifiers. Since the Actions on Fields do not know if the input stream contains sensitive data, they do not support tagging events with AIData. They are also unable to protect you in case sensitive information is erroneously placed in the source dataset. For example, in the above figure, a US Social Security Number that was erroneously placed in the "Email" column in the source table will flow downstream if your Actions on Fields did not protect the data in the "Email" column. Additionally, if you add a new column or table to the configured source dataset and, thereby, to the Striim application, any sensitive data in the newly added column or table will flow downstream as is until you manually update the Actions on Fields to cover the newly-added column or table.

You can use Sentinel’s Actions on Fields independently or in conjunction with Actions on Data Identifiers.  If you configure both Actions on Fields and Actions on Data Identifiers, then the Actions on Fields will override the Actions on Data Identifiers for information in the same field or column.  For example, assume that you are using Striim to read from 1,000 tables in your source database, and your organization requires you to protect all credit card numbers before they flow further downstream.  Assume that you need to share 10 tables with an external vendor and you have chosen to completely mask the credit card numbers in those 10 tables so that the vendor can never know the original credit card numbers.  You can easily achieve this using Sentinel by configuring the Actions on Data Identifiers to encrypt all credit card numbers, and then configuring the Actions on Fields to completely mask the fields in the 10 tables that contain credit card numbers.

Typical use cases for Actions on Fields include the following.

Sentinel supports the following actions on sensitive information.

Custom masking: Redaction

Sentinel can redact your sensitive information by masking all characters with “x” except for a fixed number of characters at the beginning and at the end of the information, as shown in the table below.  You must specify the number of characters that must be excluded from the masking.

If the number of characters in the input message is equal to or less than the total number of first and last characters that you want shown in the output message, then your input message will not be masked, as shown in the table below.

Custom masking:  Using regex

You can specify a regular expression (regex) pattern that Sentinel must search for and mask.  When you supply a regex pattern, Sentinel will replace each match with “x”, repeated for the length of the match.

For example, assume that your source contains US Social Security Numbers in the format 123-45-6789, and you want to mask these numbers while retaining the format.  You can provide a regex pattern such as (\\d{3})-(\\d{2}), and Sentinel’s output will be xxx-xx-6789 for sensitive information that it identifies as USA_SOCIAL_SECURITY_NUMBER  and that matches the supplied regex pattern.

Sentinel is available as a component in the flow designer palette.  You can place Sentinel in your application and configure it as shown below. You must set up the AI engines to work with Striim before you can run your application with Sentinel.

You can go the Sensitive Data Governance icon on the flow designer top bar and view the Sentinel results. You can select to view results for the last hour, or for the last 24 hours. Sentinel does not store results beyond the last 24 hours.

Results include: