Demystifying Splunk’s Data-to-Everything Vision

„What organizations can do with data is only limited by their own imagination,“ says Doug Merrit, Splunk’s CEO. But what does this mean for corporate IT, security and compliance teams, DevOps groups, and line of business staff? Is this one of today’s overblown mantras like „containerize everything“, „cloud-first“, or „there is a machine learning solution to every problem“? 

In addition to this analysis, you may also find my recent post on the six parts of the Splunk product portfolio interesting. For everyone interested in quick charts and short demos from SplunkConf19, please take a look at this rollup.

Forget about Machine Learning and AI – For Now

Machine learning, deep learning, and reinforcement learning are incredibly interesting topics to geek out about from a technological angle and from a cultural perspective. But while having these high-flying discussions we often forget that the core reason for all of the excitement around machine learning is the still unsubstantiated belief that we can solve many of our biggest problems through the use of Google’s, Amazon’s, or Microsoft’s latest algorithms, nVidia’s amazing graphics processors, or maybe even through the promise of Quantum computing’s ability to crunch numbers that are beyond human comprehension. In short, we are solving the right problem, but at the wrong time. Homework comes first.

Homework before Play

When examining why 90% of machine learning projects fail before or at the prototyping stage, we find the root-cause to be mostly unrelated to algorithms and GPUs. Typically, we see one bottleneck that singlehandedly prevents corporate staff from successfully planning and evaluating machine learning projects: Data.

Data-to-Everyone and Everything, Everywhere

When we think of Splunk, we instantaneously think of SIEM and log analytics as the company’s claim to fame. We understand the excitement of our security and IT operations guys about having a consolidated log analytics tool that lets them detect security threats and evaluate risk based on a broad and unified layer of log data across the organization. Security priorities often were the driving factor for extending the Splunk Platform further and further across data center, cloud, and ultimately DevOps systems. By providing all of these job roles with an increasingly comprehensive data platform that is easily searchable, Splunk is now in the unique (I’m typically very careful using that term) position to move beyond security, IT operations, and DevOps, and into the business.

Data Is Power – Splunk Has Data

The equation for business success is simple. Whoever is best able to serve customers in the most efficient and effective manner will be successful in the market place. And Splunk’s Data-to-Everything platform aims to leverage the company’s often very large footprint within the enterprise to create a comprehensive platform to ultimately enable staff, technical and non-technical, of any hierarchy and functional level to optimize their daily activities and decision making processes through the infusion of data points that provide more detail on customer pain points, investment priorities, and how the company can deliver, evaluate, and prioritize solutions quickly. The fact that this data plane is often already in place is the reason for Splunk’s excellent position of making the difficult jump from security and IT infrastructure into the business. Trying Splunk apps such as the freshly launched Splunk Business Flow or Splunk Investigate is low hanging fruit for enterprises, as the data is already in place. 

How the New Splunk Products bring Data-to-Everything and Everyone

Accessing Relevant Data: Splunk Data Stream Processor

The freshly GA Splunk Data Pipelines product provides granular and role-based access (RBAC) to any IT administrator, software developer, and business staff member. As a product manager, I can create my own alert based on a sentiment analysis in combination with some simple business rules that raises an orange flag when there is a larger than normal number of angry support tickets coming through the service desk. The system could then raise a full-on red flag when this flood of „angry tickets“ coincides with negative Tweets, Reddit posts, and product reviews online. 

Top 3 Advantages

1. Analyze data as it arrives to receive instant alerts
2. Enable everyone to click together their own reports and alerts
3. Centralized control of pipeline data access at a granular level

Accessing Any and All Data: Splunk Data Fabric Search

The new release of Splunk Data Fabric Search (DFS) now supports AWS S3 and HDFS (Hadoop) and enables users to search multiple Splunk instances. This new capability is so interesting as it opens up the world of S3 file storage to Splunk users, without the need to move around or manipulate production data. Only when users execute their search query will DFS automatically create the required data models in the background. 

Top 3 Advantages

1. Dashboards and reports can now include data streams from S3 and Hadoop
2. The required data structure is only created when a query is run
3. DFS now works across multiple Splunk clusters

Data-Driven DevOps: Splunk Investigate

Splunk Investigate is Splunk’s new collaboration platform for developers, DevOps teams, product managers, and SREs to collaboratively troubleshoot issues through a joint workspace. Considering that problem escalations often take up time of many staff members who are not involved in the problem and also time of business owners who do not need to be flooded with technical details, Splunk Investigate has significant potential to save a lot of staff time while accelerating MTTR. 

Top 3 Advantages

1. Connect to and query S3, CloudWatch, Kinesis and other data sources using SQL queries and real-time view of results
2. Benefit from team members‘ past queries and solutions
3. Provides a notebook-like user experience for teams to collaboratively and iteratively add and use data, views, dashboards, searches, text, and images. 

Visualizing Business Processes: Splunk Business Flow

The announcement and demonstration of Splunk Business Flow was my personal highlight from last year’s SplunkConf18. Both of my blog posts on this product provide a use case-centric deep-dive into how business users can finally benefit from corporate IT collecting more and more operations data. In a nutshell, Splunk Business Flow maps the actions of software users to a flow chart that illustrates the entire business process. Software developers and business owners can then see if end users are following the anticipated happy path workflow or if they are finding their own workarounds and shortcuts, often resulting in significant frustration. 

Top 3 Advantages

1. Use existing data to visualize business processes
2. View the impact of new software releases on business processes.
3. Provide real-time feedback to business users

In a Nutshell

Data Stream Processor, Data Fabric Search, Investigate, and Business Flow are four products that show Splunk’s strong position of becoming the first data center vendor to broaden the use of data analytics across the entire enterprise. Ultimately, every staff member could benefit from consuming and even creating their own reports and dashboards to complete tasks more efficiently and to simply get more done in less time and with less effort and aggravation. 

See my blog for more articles on this and related topics.