A naukri.com initiative
Amazon
2M
119
Image Credit: Amazon
How Aqua Security exports query data from Amazon Aurora to deliver value to their customers at scale
- Aqua Security faced the challenge of efficiently exporting and analyzing large volumes of data to meet business requirements, including exporting and querying data at scale to share with customers for continuous monitoring and security analysis. Aqua needed a robust and scalable solution to export and share large volumes of security query data efficiently with their customers.
- To achieve an efficient and scalable data export, Aqua implemented parallelization techniques using the Step Functions map state and the aws_s3.query_export_to_s3 function.
- Aqua used Step Functions to orchestrate a serverless solution for exporting security query data from Aurora to Amazon S3, providing built-in error handling, automatic retries, fault tolerance, and detailed execution logs for efficient troubleshooting.
- The pipeline consists of the scheduler AWS Lambda function, the query exporter Lambda function that gets the required data and format it from the Aurora database using the query_export_to_s3 function, the poller Lambda function to retrieve the job's status, and the transferer Lambda function to transfer the output data from S3 to the customer’s predefined sink.
- The aws_s3.query_export_to_s3 function enabled Aqua to export the query output data split into files of 6 GB each, directly to an S3 bucket that was highly scalable and cost-effective data storage service. This function lacks the ability to track the status of the query export job.
- By integrating Step Functions, Lambda, Aurora, and the aws_s3.query_export_to_s3 function, Aqua achieved a highly scalable, automated, and cost-effective solution for exporting and storing security query data at scale.
- Aqua implemented a mechanism using PL/pgSQL to wrap the query export process with additional functionality, enabling monitoring the job status, capturing errors, and gathering metadata related to the export. The mechanism would trigger the transfer of data from S3 to the customer's designated sink.
- The solution implemented by Aqua using Step Functions and aws_s3.query_export_to_s3 function offers numerous benefits such as scalability, cost-effectiveness, automation, and enhanced security operations.
- The integration of Step Functions and the aws_s3.query_export_to_s3 function has empowered Aqua to address their query data export challenges effectively, and benefit from streamlined data export processes, enabling scalable and cost-effective data analysis for enhanced business operations.
- If you would like to learn more about the aws_s3.query_export_to_s3 function, please see the user guide documentation.
Read Full Article
7 Likes
For uninterrupted reading, download the app