Graylog, headquartered in Houston, offers their eponymous platform for centralized log management that helps users find meaning in data faster so as to take action immediately. Graylog is available via Enterprise and Cloud plans, but also has a Small Business Plan, and an Open (free) plan with limited features.
If you already have a basic understanding of Elasticsearch and/or MongoDB, Graylog will be a great fit when it comes to log aggregation. It will be a decent option even if you don't have any experience but have the time and willingness to roll up your sleeves that learning those tools will require. Graylog supports plugins to extend functionality for things like SNMP traps, telemetry collection, and solar flares. As is the case with most software with plugins, if the core functionality for which you are looking (i.e. not logging) is based on a plugin, Graylog probably isn't for you. The majority of the plugins in the marketplace are developed by third-parties looking to solve their specific use case so bug fixes and new features are not a given.
Logstash is a must in an ELK stack, which I am sure is going to be the #1 case. At any point when you have several sources, Logstash can be the common point to aggregate, and categorize those data. Then send this new data to its destination. Very handy. It is free and open source. It may not be appropriate to analyze data-sets dependent on each other but from a different data source. Reason being Logstash works on data at hand, and not wait for other data to arrive. It would be unwise for Logstashh to handle complicated, long-running transformations because this is injected and ejected. The faster you do it, the safer.
Memory: Logstash is a HOG, if you are deploying it on commodity (i.e. cheap and old) hardware: You will need at least 2GB, just for Logstash. So don't expect to run your entire ELK stack on one AMD Athlon machine.
Overlap: Logstash fills in an area of the ELK stack that makes the most sense: as a log file transformer / shipper. However, if you start breaking that stack, with the addition of other components- you start seeing where features of Logstash may be implemented or solved in the additional components much easier (or better, or to a higher degree of resolution)
More Overlap: Since my team employs Syslog-ng extensively- Logstash can sometimes get in the way (and this may be a problem for DevOps stacks overall): You can configure Syslog to record certain information from a source, filter that data, and even export that data in a particular format. Logstash will pick that data up, and then parse it. However, if you don't keep your Syslog-ng configuration files, and your Logstash configuration files in sync, your results will not be what you expected, and this will translate into (sometimes) hours/days of work, hunting down a line item in a configuration file.
As I said earlier, for a production-grade OpenStack Telco cloud, Logstash brings high value in flexibility, compliance, and troubleshooting efficiency. However, this brings a higher infra & ops cost on resources, but that is not a problem in big datacenters because there is no resource crunch in terms of servers or CPU/RAM
I am still unhappy with the pricing model for the enterprise. Graylog competes against the likes of IBM and Splunk, but your still the new kid on the block. To price Graylog enterprise at 50k for 20GB ingest an unrealistic data. It would require multiple facets of Graylog to be stood up and only forward pruned logs to the paid version.
Azure Monitor is not exactly what I mean, but I couldn't find Azure Application Insights. Anyway, for a large organization, Azure makes more sense than using Graylog because a lot of logging will already be inside Azure. And you don't want to have two "central" logging locations. But Azure is chaos and highly "not intuitive." So for small and mid-size organizations, Graylog is still the better option.
MongoDB and Azure SQL Database are just that: Databases, and they allow you to pipe data into a database, which means that alot of the log filtering becomes a simple exercise of querying information from a DBMS. However, LogStash was chosen for it's ease of integration into our choice of using ELK Elasticsearch is an obvious inclusion: Using Logstash with it's native DevOps stack its really rational
It is very difficult to give any figures on ROI, as it depends on many factors, and in a Telcocloud environment, it is much complex to find out; however, I would give some points below on ROI
ROI based on flexibility is very high, as it reduces the time to find RCA
ROI based on integration is very high because it supports multi-vendor environments, avoiding vendor lock-in & works across multi-cloud setups
ROI on resource consumption is less because Logstash in 2-3 times more resource-intensive as compared to its lightweight alternatives resulting in latency
