A naukri.com initiative
Amazon
1M
363
Image Credit: Amazon
Reduce your compute costs for stream processing applications with Kinesis Client Library 3.0
- Kinesis Client Library (KCL) enables you to reduce your stream processing cost by up to 33% compared to previous KCL versions by redistributing the load evenly to all workers using a new load balancing algorithm
- KCL uses Kinesis Data Streams APIs to read data from the streams and handles the heavy lifting of balancing stream processing across multiple workers, managing failovers and checkpointing processed records
- Customers processing real-time data streams typically use multiple compute hosts such as Amazon Elastic Compute Cloud (Amazon EC2) to handle the high throughput in parallel
- With KCL 3.0, uneven load distribution across workers that results in higher costs can be addressed
- KCL 3.0 introduces a new load balancing algorithm that continuously monitors the resource utilization of workers and redistributes the load evenly to all workers
- Additionally, KCL 3.0 offers several other benefits including Amazon DynamoDB read capacity unit (RCU) reduction, Graceful handoff of shards from one worker to another, and Removal of the AWS SDK for Java 1.x dependency
- You don’t have to make any changes to your application code to migrate from KCL 2.x version to KCL 3.0
- We recommend checking the IAM permissions required for KCL consumer applications, maxLeasesForWorker configuration, and scaling policy after upgrading to KCL 3.0
- KCL 3.0 introduces significant enhancements that can help optimize the cost-efficiency and performance of KCL applications
- The new load balancing algorithm enables more even CPU utilization across worker instances, potentially allowing for right-sized and more cost-effective stream processing fleets
Read Full Article
21 Likes
For uninterrupted reading, download the app