Kafka Writer
Note
Kafka Writer versions 0.11 and 2.1.1 are no longer supported. As part of upgrading to Striim 5.4, all Kafka Writer adapters in created or quiesced applications will be upgraded to the new versions of the adapter and corresponding connection profiles will be created. If after upgrading you need to import TQL exported from Striim 5.2 or earlier, edit the TQL before importing to use the new properties, and create a connection profile with the appropriate properties. For more information, see Kafka Writer upgrade guide.
Kafka Writer supports writing to one or more topics in a Kafka cluster in Apache Kafka, Amazon MSK, Confluent Cloud, or Confluent Platform versions 2.0.0 to 4.0.1.
Kafka SmartMap Overview
What is Kafka SmartMap?
Kafka SmartMap is Striim's high-performance Kafka target capability for streaming data from Striim-supported sources into Apache Kafka and Kafka-compatible platforms, including Confluent Platform, Confluent Cloud, and Amazon MSK.
Kafka SmartMap writes initial load, change data, and event streams from sources such as relational databases, NoSQL databases, data warehouses, files, SaaS applications, message buses, and application adapters into Kafka topics. It is designed to make Kafka pipelines easier to configure and operate by supporting source-to-topic mapping, topic creation, schema creation and evolution, secure connection profiles, and configurable delivery semantics.
Kafka SmartMap does not perform change capture by itself. Instead, it writes data captured or produced by upstream Striim sources, including CDC sources, into Kafka with the topic structure, message format, keys, partitioning strategy, and reliability guarantees required by downstream consumers.
Kafka SmartMap: source streams to Kafka topics
Why Kafka SmartMap?
Kafka is often used as the real-time distribution layer between operational systems and downstream applications. However, writing all source changes into a single topic can make downstream processing more complex. Consumers may need to filter irrelevant events, coordinate with other teams, or handle mixed schemas and workloads in one stream.
Kafka SmartMap simplifies this model by preserving source-level separation when needed. Each source table, object, collection, or event category can be mapped to its own Kafka topic, while related streams can also be consolidated into a shared topic when that better matches the downstream use case.
This provides several benefits:
Selective consumption: downstream teams subscribe only to the topics they need.
Cleaner data products: topics can align with business entities such as customers, orders, payments, devices, or audit events.
Better scalability and isolation: a spike in one source entity or topic is less likely to affect unrelated consumers.
Simpler downstream logic: consumers do not need to filter a single mixed stream before processing.
Governed schemas: Avro and schema registry integration help manage data contracts and schema evolution.
Flexible reliability: exactly-once processing supports high-integrity workloads, while at-least-once processing supports higher-throughput use cases where downstream deduplication is acceptable.
Enterprise connectivity: connection profiles centralize Kafka connectivity, authentication, encryption, and credential management.
For example, a retail enterprise may stream inventory, order, customer, and payment changes into Kafka so fulfillment, marketing, analytics, and customer-facing applications can react in real time. If the pipeline reads from a replica, secondary database, warehouse, lake, or another upstream source rather than directly from the production database, it can reduce load on the primary system while still keeping downstream services current.