A naukri.com initiative
Amazon
1M
129
Image Credit: Amazon
Introducing Amazon MWAA micro environments for Apache Airflow
- Amazon MWAA introduces mw1.micro, the latest addition to its environment classes offering a cost-effective solution for running Airflow environments in the cloud.
- mw1.micro configuration provides a balanced set of resources suitable for small-scale data processing and orchestration tasks.
- mw1.micro environments support up to three concurrent tasks, can accommodate up to 25 DAGs, and particularly well-suited for small production workloads.
- Customers adopting a federated deployment model can provide isolated environments for different teams or departments and optimize cost with the introduction of mw1.micro.
- Amazon MWAA and Airflow key functionalities are maintained with the new environment class, including web server, custom plugins and requirements, startup scripts, and Airflow UI.
- mw1.micro introduces constraints such as only one scheduler/worker container and one Fargate task for the web server.
- Pricing dimensions for Amazon MWAA remain unchanged with metadata database storage pricing, consumed as pay-as-you-go.
- Amazon recommends monitoring key metrics such as metadata database memory usage and CPU utilization of the worker/scheduler hybrid container for optimal operation.
- Setting up an Amazon MWAA micro environment in your account and preferred AWS Region can be done using the AWS Management Console, API, AWS CLI, or automated using AWS CloudFormation, the AWS Cloud Development Kit (AWS CDK), or Terraform scripts.
- mw1.micro offers a streamlined and economical manner for Airflow power users to manage and orchestrate data pipelines in the cloud.
Read Full Article
7 Likes
For uninterrupted reading, download the app