AWS Glue Professional, Scientific, and Technical Services Reviews & Insights

We have certain transformations needs, we use Glue to fulfil them. As its integrated service from AWS itself, the connectivity to other AWS service is pretty seamless. As it’s fully serverless, we don’t have to worry about the infrastructure as well. It can crawl the things for us so we don’t have worry about the updation in our source, we got to know itself. So it’s powerful ETL FOR US.

To integrate within AWS services, it’s great ETL tool which can work perfectly with our S3 buckets and SQL DB.

but when it comes to integrate it with Multi cloud infrastructure, its connectivity and compatibility lacks, i.e. if we have to connect to IBM cloud objects, it’s not possible in any way to have our data source being in other cloud vendor.

1) In my current use case we mainly use AWS Glue for Extract Transform Load to process batch data on daily basis. 2) the main problem we can able to solve or we can say the solution which Glue provides that is, it can easily integrate with other AWS services like S3, RDS, Athena 3) pricing model is also very like pay-as-you-go 4) the main business problem which glue solve we can ingest the data ewe can perform ETL on top that and can create spark or python shell jobs

well suited:- when you want to use it to transform your data then glue also provides there own transformation also in that option you can able to do PII masking of Data if you don't want to use any code approach. The second scenario would be when want to integrate glue with other AWS services. and also wants to run Spark on glue for faster processing. less appropriate:- If you want to integrate with other services which are outside of AWS. it does not support Java as of now so if you have java resources then you can not run it.

The automation of numerous tasks, including logging, alerting, monitoring, etc., is made possible by AWS Glue. Additionally, it is economical because you only pay for the resources you actually use. One of AWS Glue's most notable features that are helpful is that it aids in the generation and transformation of data in its data catalog.

One of AWS Glue's most notable features that aid in the creation and transformation of data is its data catalog. Support, scheduling, and the automation of the data schema recognition make it superior to its competitors aside from that. It also integrates perfectly with other AWS tools. The main restriction may be integrated with systems outside of the AWS environment. It functions flawlessly with the current AWS services but not with other goods. Another potential restriction that comes to mind is that glue operates on a spark, which means the engineer needs to be conversant in the language.

We use AWS Glue for ETL of the healthcare data. The input data come from different source systems and so with different formats. With help of the AWS Glue jobs, we translate the data into a common format. With help of python scripts and the scheduled job feature, the data is fetched in a periodic manner, processed with help of the python script, converted to the parquet format, and stored in the S3 bucket. The glue catalog generates the schema of the stored data and allows AWS Athena to query the same for analytics purposes.

When the data which requires ETL has different formats, schema, and volume, this service suits them best. So, when the volume is not consistent (typical use-case of healthcare and online shopping), AWS Glue can be the prime choice. When the data is available in both batch and streaming mode, the developer needs to generate a separate codebase. This increases the source code management efforts. So, prefer to go with Glue when the nature of the data is the same (either batched or streamed).