We are excited to announce the release of Era Software’s first ever State of Observability and Log Management report.
We are excited to announce the release of Era Software’s first ever 2022 State of Observability and Log Management report. IT and security teams face many obstacles when managing massive amounts of observability data. In February 2022, we ran a survey of 315 IT professionals across a variety of industries and roles including IT executives, enterprise architects, IT Operations/DevOps/SRE and sizes of organizations.
Our goal was to understand important observability trends and what’s behind the massive growth of observability data. We also wanted to get insight into what innovations IT executives and practitioners see as valuable and what approaches help teams be more productive. In this blog, I cover a few of our findings, while the 2022 State of Observability and Log Management report contains comprehensive compiled survey results.
Volumes of data are exploding, and log data is seen as critically important
According to 99% of survey respondents, log data is seen as critically important for IT outcomes, while 92% of them see log data as essential to business outcomes. Our survey discovers that log data growth is fueled by a variety of data sources, with the largest ones coming from infrastructure and security. That’s closely followed by cloud services, application development and containerized environments. We also found out that due to their enterprise-level insights, IT executives and enterprise architects report higher levels of most types of data growth compared with more operational roles.
IT struggles to keep up with log data volumes
Our research found that IT teams struggle to keep up with observability data growth and take various approaches to manage it. 76% of surveyed IT professionals try to reduce the overall log data volumes. The most common method they use is to store only critical log data. A fifth of respondents delete log data within 24 hours. Some choose to fly blind and disable logging completely.
However, only 12% of those surveyed think that those volume reduction efforts were successful. 78% state that they end up needing the data they erased which negatively impacts troubleshooting and forensics analyses.
97% report risks associated with scalability issues with log data tools
66% of IT professionals also report that if their log management tools don’t scale, then troubleshooting takes longer, while 62% report delays in incident resolution. But there are also some interesting findings such as if tools don’t scale, there’s more risk of accidental logging of PII data and credentials, as well as increased security risks. There is also a higher risk of false alarms and false negatives.
How can innovation in observability help?
The 2022 State of Observability and Log Management also finds out that IT practitioners unanimously agree that their organizations would benefit from innovation in observability. End-to-end visibility, ingesting various data formats, easy integration with other tooling in place, actionable insights from analytics, and ability to scale cost effectively are the five leading categories of as critically important functionalities.
According to 35% of survey respondents, innovation is also needed in streaming data sometimes known as observability data pipelines or observability data management. IT and security teams use observability pipelines within their organization to connect, filter, process, and route log data between different tools for the purposes of reducing complexity and costs.
We found out adoption of streaming data pipelines is a work in progress. Interestingly, IT executives and enterprise architects have a different view on adoption of streaming data pipelines compared to DevOps/Ops and IT practitioners.
To uncover what that difference is, level of maturity in streaming pipelines deployments, our recommendations, and many more exclusive findings, download the Era Software’s 2022 State of Observability and Log Management report.