A naukri.com initiative
Swiggy
1d
202
Image Credit: Swiggy
Rill State Implementation
- Swiggy relies on the Rill platform for real-time data processing, enabling quick data-driven decisions to enhance user experience and operational efficiency.
- Rill, Swiggy’s streaming analytics platform, allows users to process events via SQL queries or custom code, supporting applications like real-time monitoring and ML feature creation.
- Rill's architecture involves interactions with App Server, Database, S3, and Kafka for pipeline management and data processing using Flink on YARN with EMR.
- Built on Apache Flink, Rill supports various data sources like Kafka, Delta Tables, and output destinations like Kafka, Delta Tables, and Redis.
- Challenges with Rill include limitations in SQL usage, especially for historical data processing, which can be addressed by NoSQL routes for complex transformations.
- Rill overcame the longer lookback window problem by leveraging Flink's State Backend, utilizing RocksDB for intermediate state persistence, and ensuring fault tolerance in data handling.
- The Flink-based solution implemented key-value state storage for historical data aggregation, featuring record-level TTL, timers, and various state types like ListState and ValueState.
- Rill's state management allows storing data locally in memory or disk and periodically in S3, making the system fault-tolerant and reliable for seamless pipeline restarts.
- By partitioning data streams and utilizing MapState for customer order counting use cases, Rill provides efficient state handling for longer lookback periods.
- Trade-offs of using Rill's state implementation include increased resource requirements, larger checkpoint sizes, and potential checkpointing failures in cases of network latency.
- The article highlights how Rill's state implementation addresses real-time data processing challenges and acknowledges key contributors for guiding and developing new features.
Read Full Article
11 Likes
For uninterrupted reading, download the app