A naukri.com initiative
15Five
1M
432
Image Credit: 15Five
How We Built 15Five’s Data Platform
- 15Five's data platform aims to enhance reporting and analytics for HR admins by providing actionable insights into employee performance and engagement through trend analysis, machine learning, and deep learning models.
- The platform consolidates data from various systems into a unified solution that supports the Outcomes Flywheel and future application development.
- Utilizing AWS technologies like Redshift Serverless, the platform ensures scalability, security, and efficiency in handling fluctuating workloads for continuous data availability and analytics.
- The Data Lake architecture, with Amazon S3 as the central data source, facilitates agile decision-making through optimized data processing aligned with business requirements, reducing costs and overprovisioning.
- Data ingestion from transactional databases is managed by AWS DMS, ensuring seamless replication with hourly granularity partitioning and Parquet file format for efficient storage and retrieval.
- AWS Glue is used to create external schemas for Redshift to query data in S3, while a custom process with AWS SNS and Lambda ensures real-time Glue metadata and partition updates for reliability.
- Data transformation is conducted using dbt, following a medallion architecture approach to progressively refine data layers for downstream analytics, with thorough testing through dbt and Great Expectations for data quality validation.
- Apache Airflow orchestrates the data pipeline, including data transformation, reverse ETL, and infrastructure management through Terraform, ensuring automation, repeatability, and visibility into potential issues.
- The reverse ETL process unloads transformed data from Redshift into Postgres application databases for efficient upserts with INSERT ON CONFLICT, maintaining consistency without duplication.
- The robust data platform at 15Five leverages a variety of technologies to handle large data volumes, ensure data quality, and deliver timely insights for informed business decisions that benefit both internal teams and external customers.
- Principal Engineer Ujwal Trivedi leads the development of 15Five's Data Platform, focusing on scalable, data-driven infrastructure to power AI and analytics for HR leaders.
Read Full Article
26 Likes
For uninterrupted reading, download the app