A naukri.com initiative
Amazon
2M
169
Image Credit: Amazon
How Open Universities Australia modernized their data platform and significantly reduced their ETL costs with AWS Cloud Development Kit and AWS Step Functions
- Open Universities Australia (OUA) used AWS services such as AWS Glue, Amazon AppFlow, and AWS Step Functions to reduce ETL operational costs and enhance visibility and maintainability of data pipelines.
- OUA analyzed their contract for the third-party tool being used for their ETL pipelines and realized that they could replicate the functionality using AWS services.
- AWS services such as Amazon Redshift were used for storing data and making it available for analytics and BI. AWS Step Functions were used to define, orchestrate and execute the data pipelines.
- AWS Cloud Development Kit (AWS CDK) was used to consolidate the source code into a code repository that could be deployed using AWS CloudFormation.
- OUA aimed to use as few moving parts as possible, prioritize ease of use for developers, minimize idle resources, and roll out updates in stages to minimize disruption to existing business processes.
- AWS services used by OAU include AWS Glue, AWS Lambda, Amazon AppFlow, Amazon S3, AWS Step Functions, Amazon Redshift, and Amazon EventBridge.
- OUA defined four data pipeline patterns which included data transformation, load, and unload; data replication using AWS Glue; data replication using Amazon AppFlow, and reverse ETL.
- The re-architecture and migration process took 5 months to complete, with significant cost reductions when compared to continuing with the third-party ETL tool. Moving to a code-based approach also made the process of maintaining data pipelines quicker and easier.
- Overall, the move to AWS services was seamless for OUA end users and enabled faster turnaround, lower costs, and enabled rapid development and deployment of data solutions.
- Authors of the article include Michael Davies, a data engineer at OAU and Emma Arrigo, a solutions architect at AWS with expertise in education customers, leveraging cloud technology, and machine learning for business solutions.
Read Full Article
10 Likes
For uninterrupted reading, download the app