We use AWS Glue to creat Etl pipelines for transforming and moving of data from different data sources like S3, snowflakes, postgres to Redshift and vice versa. Execution of spark jobs is really easy as it has auto generated code which establishes connections with source and target data bases securedly and helps in the cleansing of data like deduplication and performing validations on data. As it is Serverless it will automatically scale up and scale down the memory resources required to run the spark glue job.
Pros
Execution of spark jobs
Scaling of memory resources
Crawling the schemas
Cons
Incremental data sync
Real time data triggers
Grouping of small files
Likelihood to Recommend
ETL operations and jobs are well suited to perform with glue. If we want to transform or extract data from the data sources specially in the data stored in the AWS cloud . It is very well integrated with the other AWS services. It is easier to establish connections. We can schedule the crawlers or run on demand.
VU
Verified User
Team Lead in Information Technology (5001-10,000 employees)
One of the straightforward and quick cloud-based ETL tools is AWS Glue. It comes under the umbrella of AWS services. We use AWS Glue to analyze an extensive data set of USA based clinics and hospitals. Its HIPAA compliance for sensitive data. It comes with the support of python script, Schedular, and works very well with other AWS services like s3, rds.
Pros
Very quick for ETL job.
UI as well Command Interface with very few steps to create and schedule ETL Job.
Cons
Sample Code is very basic and not available in most of the scenario.
Likelihood to Recommend
AWS glue is best if your organization is dealing with large and sensitive data like medical record. Its comes with scheduler and easy deployment for AWS user. The data catalog keeps the reference of the data in a well-structured format. If you are already part of the AWS services, then AWS Glue is the best choice; otherwise, it's not a simple one for deployment.
VU
Verified User
Employee in Research & Development (501-1000 employees)
As an AWS Advanced Consulting Partner, we use AWS Glue in many of our Data and Analytics Solutions. We've implemented in the major enterprises in the Philippines that are in the media, telecommunication, logistics and Fintech industries. The company aims to centralize their data lake of operational raw data containing various shipping details by making use of the AWS platform.The architecture must involve an automation of the data extraction from an API. The data lake should also be visualized to provide graphical details using QuickSight, and the generated dashboards are to be embedded into the customer web portal. AWS Services implemented - Lambda, S3, Glue, Athena, Quicksight, EventBridge
Pros
After data cleansing, the team also implemented the best practices for using AWS platform services as a Data Lake, such as job bookmarking for AWS Glue jobs, proper delimiter for the AWS Glue crawlers, partitioning in AWS S3, and transformation to parquet file for compression and faster querying time in Amazon Athena.
Data modernization through combining data from multiple sources into a functioning datasets, rebuilding DW, and resctructuring data sources.
Aims to lessen customer complaints, eliminate manual data extraction requests via SR from different data sources, and Increase accuracy, consistency and speed up reconciliation process.
Cons
Faster processing, on cases where data is not partitioned efficiency
Cost optimization and pricing
Developer experience on new users
Likelihood to Recommend
Operational Excellence: A customer asked for guidance mostly from data ingestion to transformation. However, we advised the customer to use Amazon CloudWatch to monitor their own AWS Glue jobs since when we fix their glue job errors, we rely more on CloudWatch information to resolve issues.
