Splunk: General Concepts

Splunk JOINS

Joins in Splunk aren't direct analogues of SQL JOINS. In many cases they will incorrectly return the wrong or only partial results:

• index=sales sourcetype=orders | join user [ search index=user_info ] - cross Index.
• When not using max=0
• When a Query hits the Subsearch Event Limit of 50,000 Rows.
• When a Sub-Query hits the Subsearch Runtime Timeout of 60 Seconds the Query will terminate and return early.

Splunk Architecture

Distributed Search:

1. Search Head (Clusters) - used for Searching Peers. Combines results before returning them back (think Map-Reduce).
2. Indexers - often clustered, these are responsible for indexing incoming data.
   • Indexers can be Search Heads.

Resources and Links

1. https://www.splunk.com/en_us/resources/splunk-quick-reference-guide.html
2. https://community.splunk.com/t5/Splunk-Search/Join-on-one-to-many-with-max-0-still-only-returns-1-row-per/m-p/363959
3. https://docs.splunk.com/Splexicon:Searchhead
4. https://docs.splunk.com/Splexicon:Indexer

Code samples:

1. https://github.com/Thoughtscript/splunk_2026

CCPU: Exam Overview

Notes I took before passing the Splunk - Core Certified Power User Exam.

Primarily using the Official Splunk STEP free coursework.

Personal Objectives

I've used Splunk in fairly limited ways, namely:

1. For basic querying, filtering (often in tandem with some other Data Warehouse/Observability Tool like Snowflake SQL).
   • Most good engineering teams emphasize the need for multiple lines of visibility since many Observability Tools might be momentarily unavailable or prone to (internal or vendor) error.
   • Many teams didn't leverage Knowledge Objects.
2. To compare one List against another.
3. To review aggregated logs, logging.
4. Setting or refining basic alarms (that've been Templated from org or company-wide boilerplate).
5. But haven't done much (or been asked to do much with) Knowledge Objects, advanced Filtering, etc.

Exam Overview

The Splunk Core Certified Power User is a mandatory prerequisite for any higher Certification level (unlike the AWS Certification Paths which recommend a particular sequence but which also waive any formal requirements if you choose to take the tests).

Exam Topics

Splunk Core Certified Power User:

1. Topic One: Introduction to Splunk
2. Topic Two: Fields
3. Topic Three: Time
4. Topic Four: Statistical Processing
5. Topic Five: Comparing Values
6. Topic Six: Result Modification
7. Topic Seven: Correlation Analysis
8. Topic Eight: Introduction to Knowledge Objects
9. Topic Nine: Creating Knowledge Objects
10. Topic Ten: Field Extractions
11. Topic Eleven: Data Models

Resources and Links

1. https://www.splunk.com/en_us/training/certification-track/splunk-core-certified-power-user.html
2. https://docs.splunk.com/Splexicon:Knowledgemanager
3. https://www.splunk.com/en_us/training/certification-track/splunk-core-certified-advanced-power-user.html

CCPU: Introduction to Splunk

Topic One.

1. Default User Roles: User, Power, Admin (in ascending order).
2. New Reports are Private and visible only for the Owner/User who created it, by default.
3. Default TTL: 7 days (saved Searches), 10 Minutes (ad-hoc, default).
4. Search Results are displayed in Reverse Chronological Order with Timestamps according to the searching User's Settings.
5. Search Terms aren't case sensitive.
   • termA termB = termA AND termB
   • AND OR NOT supported
   • Evaluation order: Expressions with parenthesis, NOT,AND, OR
6. search is further used to filter results (not filter - there's no filter Command in Splunk).
7. Knowledge Manager: by default, only a designated User with either the Admin or Power role.
   • Knowledge Managers can share and promote Knowledge Objects.
   • App level permissions can also be modified to allow sharing and promoting Knowlege Objects.

Resources and Links

1. https://help.splunk.com/en/splunk-enterprise/spl-search-reference/10.0/search-commands/search
2. https://help.splunk.com/en/splunk-cloud-platform/manage-knowledge-objects/knowledge-management-manual/10.3.2512/get-started-with-knowledge-objects/manage-knowledge-object-permissions
3. https://help.splunk.com/en/splunk-cloud-platform/manage-knowledge-objects/knowledge-management-manual/10.3.2512/get-started-with-knowledge-objects/manage-knowledge-object-permissions#id_45c40c5b_9706_4e9d_bbc5_699ad5f101f7--en__Enable_a_role_other_than_admin_and_power_to_set_permissions_and_share_objects

Code samples:

1. https://github.com/Thoughtscript/splunk_2026/blob/main/CCPU-Introduction.md

CCPU: Fields

Topic Two.

1. Renaming a Field (with eval or rename) requires using the new name thereafter.
2. source, host, and sourcetype are the three default selected Fields in Splunk.
3. Fields that appear in at least 20% of results appear as Interesting Fields.
4. Remove or add Fields using fields - or fields (with or without + - without defaults to +), respectively.
5. Field Discovery extracts Fields during a search.
   • Default fields are extracted at Index Time.

Resources and Links

1. https://help.splunk.com/en/splunk-enterprise/spl-search-reference/9.1/search-commands/rename
2. https://help.splunk.com/en/splunk-enterprise/manage-knowledge-objects/knowledge-management-manual/9.1/fields-and-field-extractions/when-splunk-software-extracts-fields

Code samples:

1. https://github.com/Thoughtscript/splunk_2026/blob/main/CCPU-Fields.md

CCPU: Time

Topic Three.

1. timechart default Spans depend on the selected Time Range:
   • 24 hours -> default Span 30 minutes
   • 1 hour -> default Span 1 minute
   • 7 days -> default Span 1 day
2. The default time Fields are:
   • date_hour
   • date_mday
   • date_minute
   • date_month
   • date_second
   • date_wday
   • date_year
   • date_zone
3. earliest, latest time modifiers will override the Time Picker.
4. bin correct syntax is ... | bin _time span=1m or ... | bin _time span=1mins when specifying a time span - the bin Command must specify the Field _time.
5. s_raw, _time are Internal Fields.
6. Time Units, earliest, and latest are applied during the initial Base Search (and cannot therefore be used with a preceding makeresults). Time Units supported in Splunk:
   • Minute: @m or @min
   • Hour: @h
   • Day: @d
   • Week: @w or @week (snaps to Monday)
   • Specific day: @w0 (Sunday) through @w6 (Saturday)
   • Month: @mon or @month
   • Year: @y or @year
   • The expression: earliest=-30d@d latest=@d snaps to the first, complete 30 Days preceeding and the latest Day (rounded down).

Resources and Links

1. https://help.splunk.com/en/splunk-enterprise/manage-knowledge-objects/knowledge-management-manual/9.0/fields-and-field-extractions/use-default-fields
2. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/10.2/search-commands/bin

Code samples:

1. https://github.com/Thoughtscript/splunk_2026/blob/main/CCPU-Time.md

CCPU: Statistical Processing

Topic Four.

1. stats
   • Multiple Fields can be returned with by.
   • Aggregate Statistical Functions (sum, values, list etc.) that are executed within a stats clause and returned as Field.
2. chart
   • Similar to stats but narrowed to summarize exactly one or two Fields.
   • over and by will typically produce the same result when using chart.
3. top
   • Returns the top n (limit=n) most common results for a Field.
4. rare
   • Essentially the opposite of the top Command.
   • Returns the n (limit=n) least common results for a Field.
5. eval
   • Evaluates an Expression and adds that result as a Field.
   • Eval Functions (including Multivalue Eval Functions mvjoin, mvdedup, round, etc.) that are analogous to stats Functions but that differ in functionality, use, and name.
6. sort
   • Sort results by a specified Field (ascending by default, using prefixed - for descending sort).

values

stats

| stats values(uuid) AS uuid_values

list

stats

| stats list(uuid) AS uuid_values

mvjoin

eval

| eval my_field = mvjoin(my_field, ", ")

mvdedup

eval

| eval my_field = mvdedup(my_field)

Transformations:

1. over - specifies the Field that will be tracked on the x-asix of a chart.
2. by - splits or subsegments data from one Field by another.
   • This can be combined with over.

Resources and Links

1. https://help.splunk.com/en/splunk-enterprise/spl-search-reference/9.1/search-commands/stats
2. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/10.4/search-commands/chart
3. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/10.4/search-commands/top
4. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/10.4/search-commands/rare
5. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/10.4/search-commands/eval
6. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/10.4/search-commands/sort
7. https://help.splunk.com/en/splunk-enterprise/search/search-manual/9.0/create-statistical-tables-and-chart-visualizations/about-transforming-commands-and-searches
8. https://help.splunk.com/en/splunk-cloud-platform/search/spl2-search-reference/evaluation-functions/multivalue-eval-functions

Code samples:

1. https://github.com/Thoughtscript/splunk_2026/blob/main/CCPU-Statistical-Processing.md

CCPU: Comparing Values

Topic Five.

Additional Eval Functions that are combined with the eval Command:

• | eval ExampleIfField = if((status_msg="OK"), "Is OK", "Isn't OK")
• | eval ExampleValidateField = validate(status_code="200", "Not 200", status_msg="OK", "Not OK")
• | eval ExampleInField= if(in(status_code, "404", "500"), "Not good", "Is OK")
• | eval ExampleMatchField= if(match(status_code, "^[0-9]{3}$"), "Valid", "Invalid")
• | eval ExampleReplaceField= replace(status_code, "^[0-9]{3}$", "xxx")
• | fieldformat ExampleFieldFormatField=tonumber(status_code, 10)
• | fillnull value="I'm a default value"

Comments:

• When using eval all Field names are Case Sensitive and require double-quotations ("") if quotes are used (e.g. - white spaces in Field name).
• Use isnull, isnotnull to filter NULL values in results.

Resources and Links

1. https://help.splunk.com/en/splunk-enterprise/spl-search-reference/9.1/evaluation-functions/evaluation-functions
2. https://help.splunk.com/en/splunk-enterprise/spl-search-reference/9.1/search-commands/fieldformat
3. https://help.splunk.com/en/splunk-enterprise/spl-search-reference/10.2/search-commands/fillnull

Code samples:

1. https://github.com/Thoughtscript/splunk_2026/blob/main/CCPU-Comparing-Values.md

CCPU: Result Modification

Topic Six.

1. appendpipe - Append new (or merged) Field to the very bottom of a result.
2. eventstats - Append new Field to every row of an existing result/Table.
3. streamstats - A realtime, streaming, eventstats variant (with nearly the same syntax and Function support).
4. xyseries - Converts results into a more suitable format for graphing. It's the inverse of the untable Command.
5. untable - Converts results into a more suitable Table.
6. foreach - iterate over Fields, values in a Multivalue Field, etc.

Resources and Links

1. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/9.4/search-commands/appendpipe
2. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/9.4/search-commands/eventstats
3. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/9.4/search-commands/streamstats
4. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/9.4/search-commands/xyseries
5. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/9.4/search-commands/untable
6. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/9.4/search-commands/foreach

Code samples:

1. https://github.com/Thoughtscript/splunk_2026/blob/main/CCPU-Result-Modification.md