Splunk: General Concepts

Splunk JOINS

Joins in Splunk aren't direct analogues of SQL JOINS. In many cases they will incorrectly return the wrong or only partial results:

• index=sales sourcetype=orders | join user [ search index=user_info ] - cross Index.
• When not using max=0
• When a Query hits the Subsearch Event Limit of 50,000 Rows.
• When a Sub-Query hits the Subsearch Runtime Timeout of 60 Seconds the Query will terminate and return early.

Splunk Architecture

Distributed Search:

1. Search Head (Clusters) - used for Searching Peers. Combines results before returning them back (think Map-Reduce).
2. Indexers - often clustered, these are responsible for indexing incoming data.
   • Indexers can be Search Heads.

Resources and Links

1. https://www.splunk.com/en_us/resources/splunk-quick-reference-guide.html
2. https://community.splunk.com/t5/Splunk-Search/Join-on-one-to-many-with-max-0-still-only-returns-1-row-per/m-p/363959
3. https://docs.splunk.com/Splexicon:Searchhead
4. https://docs.splunk.com/Splexicon:Indexer

Code samples:

1. https://github.com/Thoughtscript/splunk_2026

CCPU: Exam Overview

Notes I took before passing the Splunk - Core Certified Power User Exam.

Primarily using the Official Splunk STEP free coursework.

Personal Objectives

I've used Splunk in fairly limited ways, namely:

1. For basic querying, filtering (often in tandem with some other Data Warehouse/Observability Tool like Snowflake SQL).
   • Most good engineering teams emphasize the need for multiple lines of visibility since many Observability Tools might be momentarily unavailable or prone to (internal or vendor) error.
   • Many teams didn't leverage Knowledge Objects.
2. To compare one List against another.
3. To review aggregated logs, logging.
4. Setting or refining basic alarms (that've been Templated from org or company-wide boilerplate).
5. But haven't done much (or been asked to do much with) Knowledge Objects, advanced Filtering, etc.

Exam Overview

The Splunk Core Certified Power User is a mandatory prerequisite for any higher Certification level (unlike the AWS Certification Paths which recommend a particular sequence but which also waive any formal requirements if you choose to take the tests).

Exam Topics

Splunk Core Certified Power User:

1. Topic One: Introduction to Splunk
2. Topic Two: Fields
3. Topic Three: Time
4. Topic Four: Statistical Processing
5. Topic Five: Comparing Values
6. Topic Six: Result Modification
7. Topic Seven: Correlation Analysis
8. Topic Eight: Introduction to Knowledge Objects
9. Topic Nine: Creating Knowledge Objects
10. Topic Ten: Field Extractions
11. Topic Eleven: Data Models

Resources and Links

1. https://www.splunk.com/en_us/training/certification-track/splunk-core-certified-power-user.html
2. https://docs.splunk.com/Splexicon:Knowledgemanager
3. https://www.splunk.com/en_us/training/certification-track/splunk-core-certified-advanced-power-user.html
4. https://tmc222.medium.com/splunk-core-certified-power-user-splk-1002-certification-experience-8b59dfc6f353
5. https://www.examtopics.com/exams/splunk/splk-1002/view/
6. https://quizlet.com/216854360/splunk-fundamentals-and-power-user-certification-flash-cards/
7. https://quizlet.com/no/360840399/splunk-core-certified-power-user-flash-cards/
8. https://quizlet.com/gb/334313903/splunk-core-power-user-flashcards/
9. https://quizlet.com/327635266/splunk-fundamentals-2-flash-cards/
10. https://quizlet.com/330849049/splunk-2-power-user-exam-flash-cards/

CCPU: Introduction to Splunk

Topic One.

1. Default User Roles: User, Power, Admin (in ascending order).
2. New Reports are Private and visible only for the Owner/User who created it, by default.
3. Default TTL: 7 days (saved Searches), 10 Minutes (ad-hoc, default).
4. Search Results are displayed in Reverse Chronological Order with Timestamps according to the searching User's Settings.
5. Search Terms aren't case sensitive.
   • termA termB = termA AND termB
   • AND OR NOT supported
   • Evaluation order: Expressions with parenthesis, NOT,AND, OR
6. search is further used to filter results (not filter - there's no filter Command in Splunk).
7. Knowledge Manager: by default, only a designated User with either the Admin or Power role.
   • Knowledge Managers can share and promote Knowledge Objects.
   • App level permissions can also be modified to allow sharing and promoting Knowlege Objects.

Resources and Links

1. https://help.splunk.com/en/splunk-enterprise/spl-search-reference/10.0/search-commands/search
2. https://help.splunk.com/en/splunk-cloud-platform/manage-knowledge-objects/knowledge-management-manual/10.3.2512/get-started-with-knowledge-objects/manage-knowledge-object-permissions
3. https://help.splunk.com/en/splunk-cloud-platform/manage-knowledge-objects/knowledge-management-manual/10.3.2512/get-started-with-knowledge-objects/manage-knowledge-object-permissions#id_45c40c5b_9706_4e9d_bbc5_699ad5f101f7--en__Enable_a_role_other_than_admin_and_power_to_set_permissions_and_share_objects

Code samples:

1. https://github.com/Thoughtscript/splunk_2026/blob/main/CCPU-Introduction.md

CCPU: Fields

Topic Two.

1. Renaming a Field (with eval or rename) requires using the new name thereafter.
2. source, host, and sourcetype are the three default selected Fields in Splunk.
3. Fields that appear in at least 20% of results appear as Interesting Fields.
4. Remove or add Fields using fields - or fields (with or without + - without defaults to +), respectively.
5. Field Discovery extracts Fields during a search.
   • Default fields are extracted at Index Time.

Resources and Links

1. https://help.splunk.com/en/splunk-enterprise/spl-search-reference/9.1/search-commands/rename
2. https://help.splunk.com/en/splunk-enterprise/manage-knowledge-objects/knowledge-management-manual/9.1/fields-and-field-extractions/when-splunk-software-extracts-fields

Code samples:

1. https://github.com/Thoughtscript/splunk_2026/blob/main/CCPU-Fields.md

CCPU: Time

Topic Three.

1. timechart default Spans depend on the selected Time Range:
   • 24 hours -> default Span 30 minutes
   • 1 hour -> default Span 1 minute
   • 7 days -> default Span 1 day
2. The default time Fields are:
   • date_hour
   • date_mday
   • date_minute
   • date_month
   • date_second
   • date_wday
   • date_year
   • date_zone
3. earliest, latest time modifiers will override the Time Picker.
4. bin correct syntax is ... | bin _time span=1m or ... | bin _time span=1mins when specifying a time span - the bin Command must specify the Field _time.
5. s_raw, _time are Internal Fields.
6. Time Units, earliest, and latest are applied during the initial Base Search (and cannot therefore be used with a preceding makeresults). Time Units supported in Splunk:
   • Minute: @m or @min
   • Hour: @h
   • Day: @d
   • Week: @w or @week (snaps to Monday)
   • Specific day: @w0 (Sunday) through @w6 (Saturday)
   • Month: @mon or @month
   • Year: @y or @year
   • The expression: earliest=-30d@d latest=@d snaps to the first, complete 30 Days preceeding and the latest Day (rounded down).

Resources and Links

1. https://help.splunk.com/en/splunk-enterprise/manage-knowledge-objects/knowledge-management-manual/9.0/fields-and-field-extractions/use-default-fields
2. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/10.2/search-commands/bin

Code samples:

1. https://github.com/Thoughtscript/splunk_2026/blob/main/CCPU-Time.md

CCPU: Statistical Processing

Topic Four.

1. stats
   • Multiple Fields can be returned with by.
   • Aggregate Statistical Functions (sum, values, list etc.) that are executed within a stats clause and returned as Field.
2. chart
   • Similar to stats but narrowed to summarize exactly one or two Fields.
   • over and by will typically produce the same result when using chart.
3. top
   • Returns the top n (limit=n) most common results for a Field.
4. rare
   • Essentially the opposite of the top Command.
   • Returns the n (limit=n) least common results for a Field.
5. eval
   • Evaluates an Expression and adds that result as a Field.
   • Eval Functions (including Multivalue Eval Functions mvjoin, mvdedup, round, etc.) that are analogous to stats Functions but that differ in functionality, use, and name.
6. sort
   • Sort results by a specified Field (ascending by default, using prefixed - for descending sort).

values

stats

| stats values(uuid) AS uuid_values

list

stats

| stats list(uuid) AS uuid_values

mvjoin

eval

| eval my_field = mvjoin(my_field, ", ")

mvdedup

eval

| eval my_field = mvdedup(my_field)

Transformations:

1. over - specifies the Field that will be tracked on the x-asix of a chart.
2. by - splits or subsegments data from one Field by another.
   • This can be combined with over.

Resources and Links

1. https://help.splunk.com/en/splunk-enterprise/spl-search-reference/9.1/search-commands/stats
2. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/10.4/search-commands/chart
3. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/10.4/search-commands/top
4. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/10.4/search-commands/rare
5. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/10.4/search-commands/eval
6. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/10.4/search-commands/sort
7. https://help.splunk.com/en/splunk-enterprise/search/search-manual/9.0/create-statistical-tables-and-chart-visualizations/about-transforming-commands-and-searches
8. https://help.splunk.com/en/splunk-cloud-platform/search/spl2-search-reference/evaluation-functions/multivalue-eval-functions

Code samples:

1. https://github.com/Thoughtscript/splunk_2026/blob/main/CCPU-Statistical-Processing.md

CCPU: Comparing Values

Topic Five.

Additional Eval Functions that are combined with the eval Command:

• | eval ExampleIfField = if((status_msg="OK"), "Is OK", "Isn't OK")
• | eval ExampleValidateField = validate(status_code="200", "Not 200", status_msg="OK", "Not OK")
• | eval ExampleInField= if(in(status_code, "404", "500"), "Not good", "Is OK")
• | eval ExampleMatchField= if(match(status_code, "^[0-9]{3}$"), "Valid", "Invalid")
• | eval ExampleReplaceField= replace(status_code, "^[0-9]{3}$", "xxx")
• | fieldformat ExampleFieldFormatField=tonumber(status_code, 10)
• | fillnull value="I'm a default value"

Comments:

• When using eval all Field names are Case Sensitive and require double-quotations ("") if quotes are used (e.g. - white spaces in Field name).
• Use isnull, isnotnull to filter NULL values in results.

Resources and Links

1. https://help.splunk.com/en/splunk-enterprise/spl-search-reference/9.1/evaluation-functions/evaluation-functions
2. https://help.splunk.com/en/splunk-enterprise/spl-search-reference/9.1/search-commands/fieldformat
3. https://help.splunk.com/en/splunk-enterprise/spl-search-reference/10.2/search-commands/fillnull

Code samples:

1. https://github.com/Thoughtscript/splunk_2026/blob/main/CCPU-Comparing-Values.md

CCPU: Result Modification

Topic Six.

1. appendpipe - Append new (or merged) Field to the very bottom of a result.
   • Is executed immediately.
   • Multiples can be combined.
2. eventstats - Append new Field to every row of an existing result/Table.
3. streamstats - A realtime, streaming, eventstats variant (with nearly the same syntax and Function support).
4. xyseries - Converts results into a more suitable format for graphing. It's the inverse of the untable Command.
5. untable - Converts results into a more suitable Table.
6. foreach - iterate over Fields, values in a Multivalue Field, etc.
7. tostring - affix prepends, specify format: "binary", "hex", "commas", "duration"
   • e.g. - "$".tostring(cost, "commas")
8. tonumber - similarly formats numbers into string text with a supplied base.
9. substr - isn't 0-indexed.
10. coalesce - Consolidates two Fields with common Values into one (normalizing them/making their Fields uniform/consistent).

Resources and Links

1. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/9.4/search-commands/appendpipe
2. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/9.4/search-commands/eventstats
3. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/9.4/search-commands/streamstats
4. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/9.4/search-commands/xyseries
5. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/9.4/search-commands/untable
6. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/9.4/search-commands/foreach
7. https://help.splunk.com/en/splunk-enterprise/spl-search-reference/9.3/evaluation-functions/conversion-functions
8. https://help.splunk.com/en/splunk-enterprise/spl-search-reference/9.3/evaluation-functions/text-functions

Code samples:

1. https://github.com/Thoughtscript/splunk_2026/blob/main/CCPU-Result-Modification.md

CCPU: Correlation Analysis

Topic Seven.

https://www.splunk.com/en_us/pdfs/training/correlation-analysis-course-description.pdf

1. correlate - calculates and displays Field correlations.
2. append - appends the results of one query below another.
3. appendcols - appends Fields from one result to another as a column.
4. union - merges results.

Resources and Links

1. https://www.splunk.com/en_us/pdfs/training/correlation-analysis-course-description.pdf
2. https://help.splunk.com/en/splunk-cloud-platform/search/search-reference/10.4.2604/search-commands/correlate
3. https://help.splunk.com/en/splunk-cloud-platform/search/search-reference/10.4.2604/search-commands/append
4. https://help.splunk.com/en/splunk-cloud-platform/search/search-reference/10.4.2604/search-commands/appendcols

Code samples:

1. https://github.com/Thoughtscript/splunk_2026/blob/main/CCPU-Comparing-Values.md

CCPU: Introduction to Knowledge Objects

Topic Eight.

Key Splunk Knowledge Objects (covered in this exam):

• Calculated Fields
• Macros
• Workflow Actions
• Pivot Data Models
• Reports
• Alarms
• Dashboards
• Field Aliases

All Splunk Fields returned from Events will populate in the Fields Sidebar.

1. Calculated Fields can contain an eval Expression.
2. Macros
   • Are defined within the Settings > Advanced search > Search macros UI/UX Component.
   • Named, reusable, functionalities.
   • Support eval and search Expressions.
3. Workflow Actions
   • Defined within the Settings > Fields > Workflow actions UI/UX Component.
   • Intended to support follow-up analysis but don't themselves populate data into Splunk Fields.
   • Support actions such as opening HTTP GET, POST operations in a new browser window.
4. Reports can be scheduled.

Permissions:

1. The Admin User Role is required to make a Knowledge Object available to all Apps.
2. User, Power, Admin (in ascending order) can create Knowledge Objects to varying extents (as allowed by their permissions).
3. Default sharing options (in ascending order of visibility):
   • Private - Shared/Accessible only for creator User (READ, WRITE).
   • App - Shared/Accessible by all Users in same App (READ, WRITEdepending on Role specifics)
   • Global/All Apps - Accessible by all the Users of all deployed Splunk Apps (READ, WRITE depending on Role specifics).

Resources and Links

1. https://help.splunk.com/en/splunk-enterprise/manage-knowledge-objects/knowledge-management-manual/10.4/welcome-to-knowledge-management/what-is-splunk-knowledge
2. https://docs.splunk.com/Documentation/Splunk/8.2.12/Pivot/IntroductiontoPivot
3. https://help.splunk.com/en/splunk-enterprise/manage-knowledge-objects/knowledge-management-manual/10.4/search-macros/use-search-macros-in-searches#b8262fda_633e_4346_8c8f_6b3404aa989c--en__Use_search_macros_in_searches
4. https://help.splunk.com/en/splunk-enterprise/manage-knowledge-objects/knowledge-management-manual/9.2/workflow-actions/about-workflow-actions-in-splunk-web
5. https://help.splunk.com/en/splunk-enterprise/create-dashboards-and-reports/reporting-manual/10.4/reporting-overview/about-reports
6. https://help.splunk.com/en/splunk-cloud-platform/manage-knowledge-objects/knowledge-management-manual/10.5.2605/get-started-with-knowledge-objects/manage-knowledge-object-permissions

Code samples:

1. https://github.com/Thoughtscript/splunk_2026/blob/main/CCPU-Intro-Knowledge-Objects.md

CCPU: Creating Knowledge Objects

Topic Nine.

1. Search by tag:
   • Tag Name: | search tag=myvalue
   • Specific Field Tag: | search tag::host=production
   • Tags are defined within the Settings > Tags UI/UX Component.
2. Field Aliases are applied after Field Extractions and before: Lookups, Tags
   • Defined within the Settings > Fields > Field aliases UI/UX Component.
3. Event Type
   • Can be used to normalize Field Names, Tags and Field Extractions
   • Categorizes Events based on search constraints
4. Search Time operations sequence (in order):
   • Field Extractions
   • Field Aliases
   • Calculated Fields
   • Lookups
   • Event Types
   • Tags
5. Available Workflow Actions:
   • GET, POST, Search

Resources and Links

1. https://help.splunk.com/en/splunk-enterprise/spl-search-reference/9.4/search-commands/tags
2. https://help.splunk.com/en/splunk-enterprise/manage-knowledge-objects/knowledge-management-manual/9.4/get-started-with-knowledge-objects/the-sequence-of-search-time-operations

Code samples:

1. https://github.com/Thoughtscript/splunk_2026/blob/main/CCPU-Intro-Knowledge-Objects.md

CCPU: Creating Field Extractions

Topic Ten.

1. Field Extractions
   • Regular Expression - uses RegEx to extract Fields
     • These are automatically generated and suggested after one selects/highlights a part of Event text.
   • Delimitters - uses a specified symbol (Comma, Space, etc.) to separate and extract data (from CSVs, etc.)
   • Manually editing Regular Expressions within a Field Extraction locks the interface and prevents validation of the results.
   • Splunk Capture Groups are equivalent to RegEx Capture Groups
   • Importantly, Fields can be extracted before Indexing is complete (and afterwards).
2. Best Practices
   • Avoid RegEx Backtracking - avoid RegEx Pipes, Quantifiers, Wildcards.
   • Use Job > Inspect Job UI/UX Component to review costly Queries and optimize searches (akin to Explain, plan)

Resources and Links

1. https://help.splunk.com/en/splunk-cloud-platform/manage-knowledge-objects/knowledge-management-manual/10.3.2512/use-the-settings-pages-for-field-extractions-in-splunk-web/use-the-field-extractions-page
2. https://help.splunk.com/en/splunk-cloud-platform/search/spl2-search-manual/expressions-and-predicates/about-splunk-regular-expressions
3. https://help.splunk.com/en/splunk-cloud-platform/search/spl2-search-manual/expressions-and-predicates/about-splunk-regular-expressions#groups-quantifiers-and-alternation-0
4. https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/10.4/search-commands/regex
5. https://help.splunk.com/en/splunk-enterprise/manage-knowledge-objects/knowledge-management-manual/10.4/use-the-field-extractor-in-splunk-web/build-field-extractions-with-the-field-extractor

Code samples:

1. https://github.com/Thoughtscript/splunk_2026/blob/main/CCPU-Creating-Field-Extractions.md

CCPU: Data Models

Topic Eleven.

1. Splunk Data Models are composed of one or more Datasets.
   • They are hiearchical
   • Child Data Sets can't remove Parent Constraints (they can only add to them)
   • Typed: event, search, transaction, and child
   • Edited through the Data Model Editor
   • Knowledge Object
   • Can be uploaded, downloaded, or created

• Splunk Data Model Metadata is uploaded through the Settings > Data models > Upload Data Model button.
• Add .jsonl, .csv, and other files through Settings > Add data in the Splunk UI/UX.
• Splunk Pivots use Lookups or Data Models to create visualizations in Splunk.

Resources and Links

1. https://help.splunk.com/en/splunk-enterprise/manage-knowledge-objects/knowledge-management-manual/10.4/build-a-data-model/about-data-models
2. https://help.splunk.com/en/splunk-enterprise/manage-knowledge-objects/knowledge-management-manual/10.4/build-a-data-model/design-data-models
3. https://help.splunk.com/en/splunk-enterprise/manage-knowledge-objects/pivot-manual/9.4/pivot-overview/introduction-to-pivot

Code samples:

1. https://github.com/Thoughtscript/splunk_2026/blob/main/CCPU-Data-Models.md