Installing and Managing Splunk Stream in a Distributed Environment | Splunk (2024)

Splunk Stream is great way to monitor network traffic from a host or via a network tap or span port. The software acts as a network traffic "sniffer." The web GUI interface allows you to choose individual metadata fields that are specific to a network protocol and write that metadata to your Splunk indexers for searching.

This means that you can capture all kinds of useful metadata through Splunk Stream, and even do limited full packet capture! Top data sources for Splunk Stream include DNS and DHCP (both protocols where logging is notoriously weak), but many people use it to capture HTTP transactions, database queries, emails, and more.

This blog post will focus on the bits needed to deploy, configure and manage Splunk Stream in a distributed environment. This may consist of hundreds or thousands of Splunk Universal Forwarders running on endpoints throughout your environment, receiving their initial Splunk Stream Technology Add-On (TA) from your central deployment server, and their subsequent Splunk Stream configuration from a central Splunk Stream server. These two roles (Deployment server and Splunk Stream server) may run on the same host, depending on the size and complexity of your configuration. Much more detail can be found on Splunk Docs, but this post will cover the high-level steps and requirements.

Overview

If you only have a small handful of stream hosts, it's by far easiest to just install the heavy-weight forwarder and manually configure it; but if you're planning to roll out a fleet of Stream sensors throughout your network, you will want to centrally monitor them. While Stream can be deployed via the Deployment server, the actual stream configuration is managed via a different model. We will walk through that model below, but the high-level summary is that you can deploy the Stream Technology Add-on (TA) onto Universal Forwarders (no requirement for heavy-weight forwarders for the TA) and tell them to all point to a central Stream configuration server over your standard Splunk port (default http over 8000).

Implementation

Note that there are two primary components in Splunk Stream. First is the Splunk Stream app, which provides the web interface and allows stream configuration. This component exposes the configuration you build to clients. The client (Splunk_TA_stream) gets its configuration from the Splunk Stream app via REST API.

In the above example of a standalone configuration, both of these components are installed (Splunk_TA_stream comes as part of the Splunk Stream app that you download from Splunkbase). In a standalone configuration, the request and transfer of configuration information from server to client takes place on the local network stack. In a distributed configuration, the request and transfer of configuration takes place over the wire.

1. Have a Splunk deployment running

In order to start capturing wire data in a distributed environment, you’ll first need a distributed environment. If you’re not there yet, feel free to read on, but you might want to go back to deploying Splunk in Splunk Docs.

The components you’ll need from the Splunk side are:

  • Search Head: The Splunk server used to search indexed data.
  • Indexer(s): The Splunk server used for ingesting and indexing the wire data from the Stream platform.
  • Deployment server: The central configuration point for Splunk Universal Forwarders in your environment.
  • Splunk Stream server: A full install of Splunk running the Splunk Stream app. This server will be used to deploy the Stream configuration to each of the universal forwarders. This server should not be configured as an indexer, but rather to forward all internal logs to the indexing tier. No Stream data will be received at this server, but rather will be sent to the indexing tier via the forwarder’s outputs.conf settings. In a smaller environment, this function could also be run on your deployment server.
  • Splunk Universal Forwarders: “The best piece of software ever written” according to James Brodsky, can run on multiple operating systems, and be used to capture numerous types of machine data, including running the Splunk Stream binaries to capture wire data at your endpoints.

See the following diagram for a breakdown:

Installing and Managing Splunk Stream in a Distributed Environment | Splunk (1)

2. Configure the Splunk Stream app for distributed management on a standalone server

Following the Splunk Stream installation guide, install Splunk Stream on a full instance of Splunk. You’ll use this host as your configuration point for all Stream configurations that will be pulled to your Universal Forwarders using subsequent configurations. Go ahead and disable all the default Streams that are enabled in the initial install. We will work on establishing a new Metadata Stream in a subsequent step.

3. Run up the Splunk Stream TA to deploy to Universal Forwarders from your deployment server

Using the Splunk Deployment server, add the Splunk Stream TA (which is available in the full Splunk Stream app download package, under the install folder of the splunk_stream_app) to a deployment server class to be pushed out to relevant Splunk Universal Forwarders. You may choose to deploy to a subset of forwarders, or all of the forwarders in your environment.

The most important step here is to define your Splunk Stream server (see point 2 above) in your TA’s inputs.conf file. The custom inputs.conf that resides in that app should point to your remote Stream server, as below.

[streamfwd://streamfwd]
splunk_stream_app_location = http://remote_stream_server:8000/en-us/custom/splunk_app_stream/
stream_forwarder_id =
disabled = 0

The following is a screenshot of an installed universal forwarder, with an example config file:

Installing and Managing Splunk Stream in a Distributed Environment | Splunk (2)

Don't forget to modify the protocol if you're using SSL/TLS on your Stream server.

This will allow your newly minted Splunk Stream TA—running in your Splunk Universal Forwarder—the ability to pull your defined Stream configurations from your central Splunk Stream server. This process is highlighted as follows:

Installing and Managing Splunk Stream in a Distributed Environment | Splunk (3)

4. Configure streams on your Stream Server to deploy to the universal forwarder fleet.

Using the configuration above, you’ll now have a bunch of Universal Forwarders phoning into the Splunk Stream server waiting for a configuration to become available. You’ll now need to create that configuration so that the Universal Forwarders pull it down in order to start sending Splunk Stream data to your deployment’s indexer tier.

After logging in to your Splunk Stream server, from the configuration page of Splunk Stream, click the “Collect data from other machines” button. You will need to enable the HTTP Event Collector first if you're going to use the Independent Stream Forwarder at a later stage (but if only using Universal Forwarder for forwarding, then you won’t need this):

Installing and Managing Splunk Stream in a Distributed Environment | Splunk (4)

Click on the “Configure Streams” menu item to begin configuring a stream for your deployment.

Installing and Managing Splunk Stream in a Distributed Environment | Splunk (5)

For our example, clone the default DNS stream by clicking the clone button:

Installing and Managing Splunk Stream in a Distributed Environment | Splunk (6)

Give the new Stream a meaningful name:

Installing and Managing Splunk Stream in a Distributed Environment | Splunk (7)

Now, configure the stream based on your requirements and enable it. Consider sending the data to a specific index and which fields you are going to capture:

Installing and Managing Splunk Stream in a Distributed Environment | Splunk (8)

Now you have a stream configured, go to “Distributed Forwarder Management” under the configuration menu and create a new group:

Installing and Managing Splunk Stream in a Distributed Environment | Splunk (9)

Installing and Managing Splunk Stream in a Distributed Environment | Splunk (10)

Choose which forwarders to deploy to using regex to define a group, if required.

Installing and Managing Splunk Stream in a Distributed Environment | Splunk (11)

Now choose which streams to capture under this new group configuration.

Installing and Managing Splunk Stream in a Distributed Environment | Splunk (12)

You should now start seeing data into your indexing tier.

5. Other considerations and notes

Don't forget that your stream forwarders will need to connect home to the Splunk Stream server, so network access will be required. You will also need to consider adjusting the frequency that they call home if you deploy a large number (hundreds or thousands), which you can do by adding the "pingInterval" setting on the streamfwd.conf. The default value is 5 seconds, but in larger environments an interval of many minutes is usually more than sufficient.

You may need to consider removing the default limit of 256KB/s network output for a Splunk Universal Forwarder, if you are deploying large Splunk Stream configurations. This limit may bottleneck the ability for the forwarder to send data in real-time to your indexing tier. To change this, modify your thruput stanza in limits.conf. Check out Splunk Docs for more information.

If you are a Splunk Cloud customer, you can still leverage all of this wire data goodness. The following diagram highlights the changes to the above deployment design that would be required in your install:

Installing and Managing Splunk Stream in a Distributed Environment | Splunk (13)

What Next?

Head over to Splunkbase and download the new Splunk Essentials for Wire Data app, which showcases 49 example use cases across security, IT ops and fraud, all using data solely from Splunk Stream. Grab it here.

Installing and Managing Splunk Stream in a Distributed Environment | Splunk (14)

Simon O'Brien

I am a passionate Splunker, traveller, family man, cook, basketballer, social advocate and security professional. I have the best job in the world, and live in the best place in the world.

Installing and Managing Splunk Stream in a Distributed Environment | Splunk (2024)

FAQs

How does distributed search work in Splunk? ›

In distributed search, a Splunk Enterprise instance, referred to as the search head, distributes search requests to other instances, called search peers, which perform the actual searching, as well as the data indexing. The search head merges the results back to the user.

What are 3 main components in a Splunk architecture? ›

Splunk Components. The primary components in the Splunk architecture are the forwarder, the indexer, and the search head.

What is distributed mode in Splunk? ›

A distributed environment describes the separation of indexing and searching logic in Splunk. In a non-distributed environment, you would have installed all the logic on a single machine, which does the indexing of data and also searches the data.

What are the 2 Splunk deployment models in the cloud? ›

"Small enterprise deployment: Single search head with multiple indexers" "Medium to large enterprise deployment: Search head cluster with multiple indexers"

What is the difference between centralized and distributed streaming commands in Splunk? ›

A centralized streaming command applies a transformation to any case that a search returns. But unlike distributed streaming commands, a centralized streaming command works only on the head of the search. We may also use the word "stateful streaming" to explain these orders.

What are the 3 modes in Splunk search? ›

Search mode has three settings: Fast, Verbose, and Smart.

Can Splunk be used for distributed tracing? ›

If you're responsible for a microservice-based system, equipping your enterprise with this powerful technology will optimize how you do your job. Try distributed tracing with Splunk Observability. Try it for free to get a real-time view of your tracing telemetry and start solving problems faster.

How to upgrade Splunk in clustered environment? ›

On the Splunk toolbar, select Apps > Manage Apps and click Install App from File. Click Choose File and select the Splunk Enterprise Security product file. Check the checkbox for Upgrade App. Click Upload.

Top Articles
Latest Posts
Recommended Articles
Article information

Author: Prof. Nancy Dach

Last Updated:

Views: 5593

Rating: 4.7 / 5 (57 voted)

Reviews: 88% of readers found this page helpful

Author information

Name: Prof. Nancy Dach

Birthday: 1993-08-23

Address: 569 Waelchi Ports, South Blainebury, LA 11589

Phone: +9958996486049

Job: Sales Manager

Hobby: Web surfing, Scuba diving, Mountaineering, Writing, Sailing, Dance, Blacksmithing

Introduction: My name is Prof. Nancy Dach, I am a lively, joyous, courageous, lovely, tender, charming, open person who loves writing and wants to share my knowledge and understanding with you.