A naukri.com initiative
Amazon
7d
8
Image Credit: Amazon
Introducing the new Amazon Kinesis source connector for Apache Flink
- Apache Flink community has released version 5.0.0, which includes a new source connector for Amazon Kinesis Data Streams called Kinesis Streams Source, which replaces the Kinesis Consumer connector.
- The Kinesis Streams Source connector uses the new Flink Source interface, which implements the new FLIP-27 standard, and is compatible with Flink 2.x.
- The new connector saves position in checkpoints and snapshots to provide exactly-once guarantees, but migration from the Kinesis Consumer connector requires attention if your application doesn't use UIDs on each DataStream operator.
- The connector improves performance and reliability of Apache Flink applications and supports watermark idleness and alignment, and retains ordering even with stream resharding and when replaying old data.
- The new connector is based on the AWS SDK for Java 2.x and relies on the AWS SDK built-in retry strategy, and does not include the Kinesis Client Library and Kinesis Producer Library, reducing the JAR size by 99%.
- Connector artifacts for Python applications include both the Kinesis source and sink connectors, and flink-sql-connector-aws-kinesis-streams is the package that includes both the new and legacy sources.
- The new Kinesis source connector is available for both the DataStream API and the Table API and SQL.
- The new source connector provides a better default shard assigner and is compatible with Flink 2.x, removing the SourceFunction interface used by legacy connectors.
- Supporting cross-Region and cross-account consuming from streams, the new connector introduces several new features, including the ability to define a watermark strategy for better code readability.
- For the most up-to-date connector versions and a working example of a Java DataStream API application using the new connector, refer to Amazon Kinesis Data Streams Connector and Amazon Kinesis Data Streams SQL Connector.
Read Full Article
Like
For uninterrupted reading, download the app