We are using Amazon Redshift as a warehousing solution, where we are doing multiple ETL sync from clickstream events as well as transaction DBs. We are doing analytics on the top this data and utilise this data to build and train data-science models. We are in gaming industry we are solving business problem such as increasing the number of user gameplay, increasing the revenue, increasing the registration as well as the acquisition.
Pros
Fast data retrieval from the table with complex joins via columnar storage and advanced query optimization techniques like parallel execution
Great reliable integration with AWS MSK using Amazon Redshift Streaming a low-latency streaming ingestion, AWS Glue and S3
Concurrency scaling and work load management - helps in segregating the load distribution based on roles
Decoupled storage and compute using RA3 instance type
Distribute cluster using Amazon Redshift data sharing i.e centralised write cluster with multiple readonly cluster
Cons
Data governance can be better
Data catalog and data discovery
Data lineage
Likelihood to Recommend
For data integration using Amazon MSK and seemless integration with Transaction DB. Faster data retrieval with complex joins as well as it is giving functionality to add dist key as well as sort key to make the performance better. Vacuum and Analyse command for improvement is the cheery on the top.
I use it as the data warehouse of our clients. I use it to build data transformations of user activity logs to ML features. I use the sql workbench to explore datasets and understand data schemas. Post that, I generally connect to the warehouse either through dbt or from jupyter notebooks.
Pros
Seamlessly integrates with the data in s3
Workbench provides useful way to query the tables within aws console
Postgres flavor of sql gives powerful capabilities such as window functions
Cons
Json support in sql is very limited.
Array type columns are missing. They are by default converted to strings
Sql workbench often goes unresponsive. I have to reload for the queries to run
A search option in the sql workbench would be great, which let's users search the whole db for a match on columns, tables etc
Likelihood to Recommend
It is a solid data warehouse on top of the AWS ecosystem. If most of your infra is on AWS, it makes good sense to go for it. But it is expected to be tuned well by a data engineer for an optimal performance. For a data scientist too, the SQL is a bit limited when it comes to unstructured columns in the tables. Arrays, jsons, etc have very poor support compared to other warehouses.
Amazon Redshift is our Data Warehouse, where we store our processed data (Hot data) for various initiatives like BI, Analytics, DataScience, etc
We also use Amazon Redshift Spectrum as our Data Lake, where we store raw (un-processed) data (Cold data) for historical analysis, trends, etc
We store various standard data in Redshift like: Bronze (ETL-ed data), Silver (Materialized Views data), and Gold (Rollups/Aggregated/Dashboard-ready data) in [Amazon] Redshift
Pros
[Amazon] Redshift has Distribution Keys. If you correctly define them on your tables, it improves Query performance. For instance, we can define Mapping/Meta-data tables with Distribution-All Key, so that it gets replicated across all the nodes, for fast joins and fast query results.
[Amazon] Redshift has Sort Keys. If you correctly define them on your tables along with above Distribution Keys, it further improves your Query performance. It also has Composite Sort Keys and Interleaved Sort Keys, to support various use cases
[Amazon] Redshift is forked out of PostgreSQL DB, and then AWS added "MPP" (Massively Parallel Processing) and "Column Oriented" concepts to it, to make it a powerful data store.
[Amazon] Redshift has "Analyze" operation that could be performed on tables, which will update the stats of the table in leader node. This is sort of a ledger about which data is stored in which node and which partition with in a node. Up to date stats improves Query performance.
Cons
Amazon Redshift is a Managed Service. But it is Not a 100% managed service. We still need to configure it with WLM (Work Load Management) settings, and add Query Queues to make sure it's resources aren't wasted and it is performant at it's best state, all the time
[Amazon] Redshift has a concept of "Vacuum", which is an operation to claim the disk space back from deleted data/tables. They recently started doing automated vacuuming. Prior to that we had to do that at regular intervals, to claim the data back.
Likelihood to Recommend
[Amazon] Redshift is suited for various use cases like Time series data, Structured / relational data, Semi structured data like JSON, etc.
[Amazon] Redshift might not work 100% well with full performance, for Graph DB use cases.
Amazon Redshift is being used by our whole organization. It is our primary tool for our data warehouse. We decided to switch to a cloud database because our in house servers just weren't able to keep up with our need for fast data delivery. We can adjust the speed up to where we need it to be and it has been very useful.
Pros
Aggregation
Extracting data
Postgres Based
Cons
Could be faster
Limited sql workbenches
Expensive when speeding up the processing
Likelihood to Recommend
Redshift is best suited for our data storage and designing our fact and dimension tables. We keep our non-structured data there that can be accessed at any time as well as our relational database. I'd say that if you don't have a need for a relational databasae, then Redshift probably isn't going to be a viable product.
Amazon Redshift was our enterprise data warehouse as a backend to our BI solutions.
Pros
Fixed cost.
Tunable table design.
Cons
Need to provision warehouse for highest capacity.
No real separation between computing and storage (even when considering Spectrum).
All users share the same infrastructure resulting in frequent 100% utilization error messages.
A leader node can become a bottleneck for too many concurrent aggregate queries.
Likelihood to Recommend
Redshift is appropriate when the number of concurrent users are low and pointed queries are the focus. It is not appropriate when a large number of concurrent users is to be supported,
VU
Verified User
Team Lead in Information Technology (501-1000 employees)
Amazon Redshift is a PostgreSQL based solution was seen as a drop-in replacement for several Postgres based databases (or schemas in Postgres parlance). The eventual product: a Bill Inmon principles-based Data Warehouse served as a point or source of a single truth. It aided in decision making, historical outlooks and forecasting across various organizational verticals - the Finance, Marketing, and Medical Research. It was also possible to deliver data extracts to 3rd parties or visualize data on demand.
Pros
Data retrieval experience really gets improved.
In terms of database management, it is really a no management at all in AWS. There is no even an OS to take care or worry about.
Auto or on-demand scaling is nice.
Integrates quite well with other products within the AWS ecosystem.
Cons
The number of connections is too small, I think at around 50 are allowed in parallel. With some ETL and apps connecting all the time, this brings an undesired possibility to some users or tools being unable to connect.
Needs some tuning.
The logging part is almost nonexistent.
Can be quite costly in the long run as opposed to just RDS or on-prem/dedicated solutions.
Likelihood to Recommend
If the number of connections is expected to be low, but the amounts of data are large or projected to grow it is a good solutions especially if there is previous exposure to PostgreSQL. Speaking of Postgres, Redshift is based on several versions old releases of PostgreSQL so the developers would not be able to take advantage of some of the newer SQL language features. The queries need some fine-tuning still, indexing is not provided, but playing with sorting keys becomes necessary. Lastly, there is no notion of the Primary Key in Redshift so the business must be prepared to explain why duplication occurred (must be vigilant for).
AWS Redshift is the cloud-based data warehouse where we store our application level datasets and is used further for insights from the stored dataset. It improves the decision support for our business based on data analytics on a large set of real-time datasets which force the business processes to the next level. It provides good performance with high availability for essential data analysis and valuable intelligence.
Pros
Easy query and fast execution
High performance and availability
Support of large datasets
Scalable solution
Cons
Database optimization
Time consuming process for schema design and modification
Integration is little bit difficult
Likelihood to Recommend
Amazon Redshift is the data warehouse under the umbrella of AWS services, so if your application is functioning under the AWS, Redshift is the best solution for this. For large amounts of data, the application is the best fit for real-time insight from the data and added decision capability for growing businesses. If your application is outside of AWS it might add more time in data management.
VU
Verified User
Employee in Research & Development (501-1000 employees)
1)ETL(Talend) data from source applications (Salesforce, Jira, OpenAir, NetSuite, Sharepoint, Active Directory, Office 365, etc.) to S3 bucket and from S3 bucket push data to Redshift (sync time is 10 minutes intervals. Data is available almost real-time only lagging 10 minutes). 2) All Department use it—Engineering, Sales, and Marketing. 3) As I said data is almost in real-time, so it is very useful for taking real-time decisions for upper management. We also reduced Salesforce licenses, because most of the users only used it to see reports. Now they are happy to used Redshift.
Pros
We reduced the number of Salesforce licenses— Engineering, Sales and Marketing guys are happy to query data from Redshift.
Very fast to provide a huge data set with complicated measure.
Some of the calculations failed in Salesforce. Redshift returns with the same calculations very fast.
Very easy to maintain, no need to worry about hardware failure.
Cons
We are not able to modify column size.
Likelihood to Recommend
I recommend all to use Redshift, It is easy to use and maintain. We have reduced the number of Salesforce licenses due to real-time data we have in Redshift. People are happy to use Redshift.
Redshift is being used by engineering for our data warehouse or data lake, if you will. It's part of our ETL pipeline, where the data is used to form dashboards and analytical queries across all of our initially segregated data. So it is kind of a source of truth linking data across the company. These dashboards are accessible across other departments in the organization. The data is consumed by everyone, not just engineers.
Pros
It's fast for data analytics across multiple columns.
Essentially, it's good for big datasets.
Cons
By using RedShift you're kind of married to using AWS's other services, e.g. Redash.
You need your data in the cloud.
No separate storage and computing.
No structured data types.
Doesn't scale easily.
Likelihood to Recommend
Use Redshift for data warehouses, especially if your data is already in the cloud (AWS). It's great for large datasets, and fast too, even if your dataset is column heavy. It's less so for when you have a bunch of rows. All in all, it's a good starting point for any aspiring data warehouse, but there are other promising solutions too. E.g. Snowflake.
