Amazon CloudWatch vs. Logstash

If you use any AWS services, CloudWatch is the natural choice to monitor & troubleshoot your workload. Thankfully, for most AWS services, CloudWatch is either built-in or very easy to set up. However, being proficient in browsing & tracking the log events would take some training & practice. Having some experienced people on the team would help immensely, especially in spreading the skill to the rest of the team.

Incentivized

Logstash is a must in an ELK stack, which I am sure is going to be the #1 case. At any point when you have several sources, Logstash can be the common point to aggregate, and categorize those data. Then send this new data to its destination. Very handy. It is free and open source. It may not be appropriate to analyze data-sets dependent on each other but from a different data source. Reason being Logstash works on data at hand, and not wait for other data to arrive. It would be unwise for Logstashh to handle complicated, long-running transformations because this is injected and ejected. The faster you do it, the safer.

Incentivized

• It provides lot many out of the box dashboard to observe the health and usage of your cloud deployments. Few examples are CPU usage, Disk read/write, Network in/out etc.
• It is possible to stream CloudWatch log data to Amazon Elasticsearch to process them almost real time.
• If you have setup your code pipeline and wants to see the status, CloudWatch really helps. It can trigger lambda function when certain cloudWatch event happens and lambda can store the data to S3 or Athena which Quicksight can represent.

Incentivized

• Plugin ecosystem allows modular extensions.
• Tight integration into the Elastic.com products of Beats and Elasticsearch, so minimal setup is required when using those tools.
• Filter plugins are powerful for extracting and enriching input data.

Incentivized

• Memory metrics on EC2 are not available on CloudWatch. Depending on workloads if we need visibility on memory metrics we use Solarwinds Orion with the agent installed. For scalable workloads, this involves customization of images being used.
• Visualization out of the box. But this can easily be addressed with other solutions such as Grafana.
• By design, this is only used for AWS workloads so depending on your environment cannot be used as an all in one solution for your monitoring.

Incentivized

• Memory: Logstash is a HOG, if you are deploying it on commodity (i.e. cheap and old) hardware: You will need at least 2GB, just for Logstash. So don't expect to run your entire ELK stack on one AMD Athlon machine.
• Overlap: Logstash fills in an area of the ELK stack that makes the most sense: as a log file transformer / shipper. However, if you start breaking that stack, with the addition of other components- you start seeing where features of Logstash may be implemented or solved in the additional components much easier (or better, or to a higher degree of resolution)
• More Overlap: Since my team employs Syslog-ng extensively- Logstash can sometimes get in the way (and this may be a problem for DevOps stacks overall): You can configure Syslog to record certain information from a source, filter that data, and even export that data in a particular format. Logstash will pick that data up, and then parse it. However, if you don't keep your Syslog-ng configuration files, and your Logstash configuration files in sync, your results will not be what you expected, and this will translate into (sometimes) hours/days of work, hunting down a line item in a configuration file.

Incentivized

Although the tool itself is easy to integrate and is readily available for use, it has its limitations. The key limitations of cloudwatch are with respect to cost incurred on log retention and log querying. While for key use cases this is sufficient, for more advanced use cases, Amazon CloudWatch doesn't work out. Also, obviously it is tightly coupled with AWS, which makes you look away if you need a single tool for all monitoring

Incentivized

As I said earlier, for a production-grade OpenStack Telco cloud, Logstash brings high value in flexibility, compliance, and troubleshooting efficiency. However, this brings a higher infra & ops cost on resources, but that is not a problem in big datacenters because there is no resource crunch in terms of servers or CPU/RAM

Support is effective, and we were able to get any problems that we couldn't get solved through community discussion forums solved for us by the AWS support team. For example, we were assisted in one instance where we were not sure about the best metrics to use in order to optimize an auto-scaling group on EC2. The support team was able to look at our metrics and give a useful recommendation on which metrics to use.

Incentivized

We use Cloudwatch for simpler monitoring, but these metrics and logs often feed into bigger ecosystems across our organization. The metrics and logs in Cloudwatch allow our developers quick and easy access to the data they need whilst easily integrating the same data into more prominent platforms for wider analysis, including Service desk support, SecOps, and ITOps monitoring within the organization.

Incentivized

MongoDB and Azure SQL Database are just that: Databases, and they allow you to pipe data into a database, which means that alot of the log filtering becomes a simple exercise of querying information from a DBMS. However, LogStash was chosen for it's ease of integration into our choice of using ELK Elasticsearch is an obvious inclusion: Using Logstash with it's native DevOps stack its really rational

Incentivized

• Positive for alarms and alert notifications once configured/customized.
• Has upfront learning curve, and cost can increase as does the alarm activity and monitoring details you may require.
• Cost-effective for any size organization keeping with AWS and utilizing its native tools is a savings in long-term ROI.

Incentivized

• It is very difficult to give any figures on ROI, as it depends on many factors, and in a Telcocloud environment, it is much complex to find out; however, I would give some points below on ROI
• ROI based on flexibility is very high, as it reduces the time to find RCA
• ROI based on integration is very high because it supports multi-vendor environments, avoiding vendor lock-in & works across multi-cloud setups
• ROI on resource consumption is less because Logstash in 2-3 times more resource-intensive as compared to its lightweight alternatives resulting in latency