Splunk 101

Beginner Friendly

By Yuki Ang
18/8/2023
Agenda
• Introducing Splunk

• Splunking Journey • Becoming a Splunker #1: Setting up Splunk

• Splunk Components & Architectures • Becoming a Splunker #2: GIGO

• Splunk Apps & Add Ons • Becoming a Splunker #3: Basic Searches

• Splunk Premium Solutions • Splunk’s SPL

• Enterprise Use Cases • Splunk Knowledge Objects

• Splunk User Groups and Roles

• Splunk Licenses
Introducing
Splunk
The Notorious B.I.G. D.A.T.A.
 IT
Operations

Security,
Compliance
The & Fraud
Data-to-Everything
Platform Payment &
Core
Banking

Application
Performance
 Computers Internet
ACT
Network Devices
Databases Devices
Virtual
Middleware Machines
Clouds
Any Structure XML

Any Source
Logs JSON CSV
ANALYZE INVESTIGATE
REST APIs
Any Time Scale
Metrics Directory
Any Volume HTTP
Files
Events Alerts
Other APM
Scripts
MONITOR Platforms
POV: Splunk as Make Up for Data
Apply Magic to Data in the way you want

Unreadable Beautiful Insights

Raw Data
Messy In whatever context
and you want
no context
Splunking
Journey
“ I am still learning.” - Michelangelo
 Poor
troubleshooting

Poor time Remember, SPL sucks 

management practice
and apply
Did not
RTFM
enough Knowledge
sharing

More Explore use

hands on cases,
solutions &
integrations
Yuki’s
Splunk
Splunk Free 101
Courses &
Documents
Splunk
Components &
Architecture
Types of Splunk Platform
1. Splunk Enterprise
• On-prem solution where all Splunk components are installed and administered
on-premises
• Supported Linux (most-preferred), Windows, and Mac OS

2. Splunk Cloud
• Cloud-based Splunk Enterprise as a service where infrastructure is hosted
and administered by Splunk Support
Splunk Components Overview
Search Tier & User Access Indexing & Data Storage Data Collection Tier Management Tier

Search Head Indexer Heavy Forwarder Deployment Server

Search Head Cluster Indexer Cluster Universal Forwarder License Manager

Monitoring Console

Cluster Manager

Deployer
As compared to Dynatrace Managed

User Access and Data Storage Data Collection Tier Management Tier

ActiveGate CMC
DT Cluster
OneAgent DMC
Splunk Components
Splunk Core

License Deployment
Search Head Indexer Manager Server

Searches indexed Processes and stores Splunk Enterprise Configuration

data, creates data as events in License Holder manager to all
dashboards, reports index forwarders
and alerts with SPL
Splunk Components
Forwarders
Heavy Forwarder Universal Forwarder
o Splunk Enterprise instance that used o Separate software – Splunk Universal
for ingest various data inputs, store Forwarder just to collect and forward
indexed data locally and parse data data to the receiving indexer
before forwarding it to the receiving
indexer

Note
• Configuration is required to define what data to be
sent (inputs) and where to send the data (indexer)
• TCP9997 is required for Splunk data forwarding
Splunk Components
Deployer, Cluster Manager, Monitoring Console
Deployer Cluster Manager Monitoring Console
o Distribute apps and o Distribute apps and o Splunk Enterprise
configuration updates configuration updates monitoring tool to provide
(configuration bundle) to (configuration bundle) to health insights for the
search head cluster indexer cluster. deployed Splunk instances
members.

Note
• Deployer and Cluster Manager are only applicable for
cluster deployment
Splunk Enterprise Architecture #1
Single Instance (Standalone)

License
Search Manager +
Indexer Forwarders
Head Deployment
Server

+ Monitoring console

All Splunk core components in one single Splunk Instance to serve all roles
Splunk Enterprise Architecture #2
Distributed with Single Indexer
License
Search Manager +
Head Deployment
Server
+ Monitoring console

Indexer Forwarders

Splunk core components’ roles are distributed across different Splunk instances
Splunk Enterprise Architecture #3
Distributed with Indexer Cluster
License
Search Manager +
Head Deployment
Server
+ Monitoring console

Indexer Cluster Master Node + Peer Nodes

Cluster
Indexer Indexer Indexer Forwarders
Manager
With Load Balancing

Splunk core components’ roles are distributed across different Splunk instances. Indexer are clustered (minimum of 3,
due to replication factor) and managed by Cluster Manager.
Splunk Enterprise Architecture #4
Clusters DNS-Based Load Balancing
Search Head Cluster: Search Captain + Search Peers

License
With Load Balancing Search Search Search Manager +
Head Head Head Deployment
Server
+ Monitoring console

Forwarders
Indexer Cluster: Master Node + Peer Nodes

Cluster
Indexer Indexer Indexer Manager +
Deployer

With Load Balancing

Splunk core clusters for bigger enterprise to build a high availability, high scalability and disaster recovery solution.
Key Factors in deciding Architecture Type

Daily indexing volume (license)

Number of users
Number of forwarders
Number of Splunk apps
Other design requirements
Splunk Apps
and Add-Ons
About Splunk Apps and Add-ons
Available at Splunkbase

Apps Add-ons
*with User Interface *not require User Interface
Extending Splunk Capabilities by Collection of configuration files to aid
providing: in:
• More use cases • Data ingestion
• Tailor-made Splunk Dashboards to • Data normalization
use case • Filtering and Parsing of Data
• Pre-configured saved searches, • Data Enrichment
reports and lookups
• Additional Visualizations
• Included relevant Add-on for data
source

1000+ apps available on Splunkbase or you can build your own: Read up Develop Splunk apps
Common Splunk Apps & Add-ons
Available at Splunkbase

Splunk App for Infrastructure Splunk IT Essential

Splunk Add-on for Infrastructure Splunk Security Essential
Splunk DB Connect Splunk Machine Learning Toolkit
Splunk Add-on for Microsoft Windows Splunk Dashboards app (beta)
Splunk Add-on for Unix and Linux Splunk Dashboard Examples
Splunk Add-on for AWS Splunk App for Lookup File Editing
Dynatrace App for Splunk Splunk Add-on Builder
Dynatrace Add-On for Splunk Splunk Custom Visualization add-ons
Splunk Supporting Add-on for Active Directory
Splunk
Premium
Premium Splunk Solutions
Require additional license

Security Observability

Splunk Enterprise Security Splunk Infrastructure Monitoring

Splunk SOAR Splunk IT Service Intelligence
Splunk Mission Control Splunk Application Performance Monitoring
Enterprise
Use Cases
Sample Use Cases
Database
Banking App: Capacity
Transaction Platform Reporting with Network Alert
Monitoring Monitoring Forecasting Monitoring

To detect and report One glance Reporting database One glance view on
on missing transaction application health utilization and provide all network devices
during transaction dashboard, forecast utilization alerts
pipeline processing consolidated data
sources from various
monitoring tools
Enterprise
Use Cases
Production Dashboard Screenshots
Transaction Monitoring
Foodpanda: TV Dashboard and Drilldown
Banking App: Platform Monitoring
CIMB Clicks: Performance data from Nagios, Entuity, Dynatrace and eG
Database Capacity Reporting with Forecasting
Selected Critical Application: RTB
Database Capacity Reporting with Forecasting (cont’)
Selected Critical Application: RTB
Network Alert Monitoring
Putting Alerts from BMC Entuity together
Becoming a
Splunker #1
Setting up Splunk
Plan and Install Splunk Enterprise
Prerequisites
1. Determine the deployment requirement
2. Provisioning of Splunk instances
3. Network and firewall requirement
4. Download or transfer relevant Splunk Enterprise installer packages (take note on version compatibility)
Implementation
1. Installation:
• *NIX–uncompress .tar.gz installer in the Splunk path (installation usually at /opt/splunk)
• Windows – run the .msi installer (installation usually at C:\Program Files\Splunk)
2. After installation:
• *NIX – start Splunk service manually (note: enable boot-start)
• Windows – Splunk service starts automatically Tips: Start,Stop or Restart Splunk services:

Post Implementation /$SPLUNK_HOME/bin/splunk/start

/$SPLUNK_HOME/bin/splunk/restart
1. Login to Splunk Web: <IP or hostname>.com:8000 /$SPLUNK_HOME/bin/splunk/stop

2. Install license on your Splunk License Manager instance

Plan and Install Splunk Universal Forwarder
Prerequisites
1. Determine the deployment design (if require deployment server)
2. Provisioning of Splunk instances
3. Network and firewall requirement
4. Download or transfer relevant Splunk Universal Forwarder installer packages (take note on version compatibility)
Implementation
1. Installation:
• *NIX–uncompress .tar.gz installer in the Splunk path (installation usually at /opt/splunkforwarder)
• Windows – run the .msi installer (installation usually at C:\Program Files\Splunk Universal Forwarder)
• Enter the necessary for Deployment Server and Receiving Indexer setting
2. After installation:
• *NIX – start Splunk service manually (note: enable boot-start) Tips: Start,Stop or Restart Splunk services:

• Windows – Splunk service starts automatically /$SPLUNK_HOME/bin/splunk/start

/$SPLUNK_HOME/bin/splunk/restart
Post Implementation /$SPLUNK_HOME/bin/splunk/stop

1. Configure receiving to listen on port 9997 in Splunk Web

2. Configure data sources to be collected and sent to receiving indexer
Becoming a
Splunker #2
Garbage In Garbage Out - GIGO

Getting the data in …

“ Data is like garbage.
You’d better know what
you are going to do with
it before you collect it. ”

~ Mark Twain, American Writer

Understanding Three phases in index time process:
the Data 1. Input
Ingestion Handled by source (usually forwarders) and sent to indexer
Process
2. Parsing
Handled by indexer (or happen in heavy forwarders before
sending to indexer)

3. Indexing
Aggregate the data through pipelines and finally writes the
data to the indexer disk
Data Input 1) Monitoring Files and Directories

Types 2) Monitoring Logs

3) Scripts

4) REST APIs data

5) Network data (via TCP or UDP)

6) HTTP Event Collector

The data inputs can be configured with apps

and add-on from Splunkbase or custom edit.
Optionally
add one-time 1. Upload data (e.g. logs, csv, json, or xml file)
data in
Splunk Web 2. Set Source Type
• Preview the data before indexing, set appropriate source type and
other settings for your data
• Many pretrained source types have default settings ready to apply

3. Input Setting
• Choose the app to save your input configuration (save in Search &
Reporting app if nothing specific)
• (Optional) Enter hostname for metadata tagging
• Select index to store the data (create new index first beforehand)

4. Review and Submit

5. Data are indexed as events and available for search

 Key configuration files to read up:
.conf
• inputs.conf controls how the forwarder collects
for Data data.
Inputs • outputs.conf controls how the forwarder sends
data to an indexer or other forwarder.

• server.conf for connection and performance

tuning.

• deploymentclient.conf for connecting to a

deployment server
 What happens next?
Searching indexed data from file inputs

Default Search App

Search Time Range
Smart Basic search to get
extraction your data displayed
based on set
source type
parse your Timeline of the indexed data
data into
events Search Result: data in
readable event format

Metadata &
Selected field
Metadata
Extracted
Fields
Becoming a
Splunker #3
Basic Searches
with Splunk Search & Reporting app
 • Default App that is shipped together with Splunk
Search App Enterprise

• Primary way to navigate data

• Used to run searches, save reports, create

dashboards and alerts
Keyword

type any word Events with

to search for matched keyword

matching result
Index Index is a repository for Splunk data.
• _internal: contains your Splunk instance logs,
useful for troubleshooting
• custom index: create and name index with
meaningful names to distinguish data

index =
“index name”
Other Fields Fields are searchable key/value pair in event data.

• Search data by <field name>=“value” will refine the

results accordingly

• Field names are case sensitive

<Field name>
= “value” Events with
matched field value
 1) AND
Using
Operator Example: status=404 AND status=500
Return no event as searching same field but different values
because logically one result could not have two status code

Example: status=404 AND hostname=isat

Return event with hostname is isat and has status of 404

2) OR
AND vs OR
Example: status=404 OR status=500
Return events that has status is 404 or status is 500

Example: status=404 OR hostname=isat

Return event that matches hostname is isat that has status not
just 404, or status is 404 but hostname not just isat
Using 1) =! (does not equal)

Exclusion Example: status !=200

Return events that has status field value does not equal 200

2) NOT

Example: NOT status 200

!= vs NOT Return events that has status field value does not equal 200
AND all other events does not have status field
Recommended 1) Time is important filter, try not to search all
time
Search
Practices 2) Make search terms as specific as possible,
can include as terms as possible to get
finest searched events

3) Try not to use exclusion unless no specific

term

4) Can use wildcard (*) in search terms or

field’s values

5) Understand modes of searching

Search
Processing
Language
SPL
Splunk Search Processing Language
Syntax

Diagram above represents a search string, pinned point by syntax components

Splunk Search Processing Language
5 basic components
Search Terms
Keywords, phrases, booleans

Commands
What to do with the results? – create table, charts, evaluate or format

Functions
How do you want to compute the result? – in statistics, sum, average, max

Arguments
What are the variables to apply the function? – field name for the average, do round up

Clauses
How do you want to display the result? – Rename fields, group fields
Splunk
Knowledge
Objects
 • Dashboards are views that are made up of
Dashboards dashboard panels

• Contains fields, charts, tables and reports which

are customized based on business rules & use
cases

• Can be interactive (with filters and drilldowns) or

static one glance view for TV display

• Types:
‒ Classic: xml format
‒ Dashboard studio: json format

• Dashboard can be created by

‒ Adding new search to dashboard panel
‒ Adding report to dashboard panel
 • Splunk reports are results saved search from
Reports a search action

• Eliminates the need to manually perform

recurring searches

• Reports can be scheduled to be generated

on a regular interval

• Can be exported to PDFs to be shared with

others

• Can be added to a dashboard

Alerts • Splunk alerts are based on searches that can
run either:
– On a regular scheduled interval
– In real-time

• Alerts are triggered when the results of the

search meet a specific condition that you
define
• Alerts can be sent via triggered actions (like
email)
Splunk Users
and Roles
 • Users are can be created through:
User & Roles ‒ Splunk Native
‒ Internal LDAP
‒ Other external authentication systems

• Users are assigned with Roles

• Roles are set of capabilities (object permissions)

• Predefined roles:
‒ admin
‒ power
‒ can_delete
‒ user

• Custom roles can be created to inherit the

predefined roles
Splunk
Licenses
Licensing Model
Splunk Enterprise Splunk Cloud

Volume based Infrastructure Volume based Workload based

based
Means the daily Means the daily Workload Pricing is
aggregate volume of The total vCPU aggregate volume of based on search
uncompressed data count across all uncompressed data intensity. But it can
for indexing. Splunk Enterprise for indexing. be more economical
search heads and if your workload
indexers count (*Storage given also profile includes lower
towards the vCPU up-to 90 Days of full search use cases.
licensed capacity consumption size)
Licensing Enforcement:
Violation

Broken connections between license manager and peers

License peer cannot communicate with the license manager for 72 hours or more, the peer is placed in
violation, and search is blocked.

Exceed maximum daily indexing entitlement

If you exceed your licensed daily usage in any single calendar day, you get a license warning.
To resolve the warning, license manager will wait until next day before validating your license usage again.
Live Demo
Wrapping Up
POV as a Consultant

Pros Cons
Monitor of Monitors Time consuming in Implementation
Able to ingest enormous amount of data Difficult administration and maintenance: most
Wide range of use cases are configuration files based
Efficient data enrichment Not 100% automatic discovery: No plug and play
Powerful Search Engine and Language feature
Noise reduction Steep learning curve: not intuitive to first-timer
Many supported Splunk Apps Expensive Certifications
Freedom in customization
Scalability
Splunk Resources
1. Splunk Official

2. Splunk Documentation

3. Splunkbase for Apps

4. Splunk Free Training

5. Browse all resources

Thank You
Happy Splunking!