Identify Untriggered Alerts in Splunk

Identify Untriggered Alerts in Splunk

In this blog we ll get to know whole about alerts where they are saved and how to list those alerts which are triggered and which are not triggered?

Why is need to track the alerts which are triggered or not??

In our splunk environment we created many apps and work on many ongoing projects, we have to create many alerts with respective of each app where we keep track on whats going on in our projects . Imagine we have more apps and we have alerts in each of them so we wont have that much spare time to go and check each and every alerts in their apps which are triggered or not

Which we ll come to know which of the query needs to rewrite and arises problem in them.

So to get rid of that time consuming work for checking each and every alert check we have created a this blog which will let you know alerts created in all apps in splunk environment and to identify which of the alerts is not triggered so we can identify them and work on them.

 

How Alert work flow:-

  • - Set a search which return events
  • - How often we want that search to run to get specific events
  • - Set an alert condition, when condition is met alert will get triggered

Alerts when created are saved in specific app locations it can also be shared in all apps too

Location is :- /opt/splunk/etc/apps/[app-name]/local/savedsearches.conf

 

In our saved searches we have reports and alerts saved

 

 

 

 

How to identify Alerts and Reports in savedsearches.conf file??

 

alert.track = 1 is for saved search is configured as Alert alert.track = 0 is for saved search is configured as Report alert.severity values are 1 – debug, 2 – info, 3 – warn, 5 – severe, 6 – fatal

When set alerts are triggered they can be seen in Activity → Triggered Alerts

 

Here is an alert set in Search & Reporting  app

 

 

 

Here is an alert set in Demo  app

 

 

 

Alerts are triggered or not are set in Alerts column that how many times it is been triggered 

If Alerts has 0 so that specific alert is not triggered or specific condition is not met for alert to get triggered

In above example for Search & Reporting App Alerts are triggered 144 times and for

Demo App created Alert count is 0 so that alert is not triggered yet

 

 

By selecting All App we can get results of all alerts which are configured in Splunk Environment

 

 

Query :-

|rest/servicesNS/-/search/saved/searches 

|  search alert.track=1 

|  fields title description search disabled triggered_alert_count actions action.script.filename alert.severity cron_schedule

| table title  search disabled triggered_alert_count alert.severity cron_schedule 

| rename title as "Alert Name", search as "Alert query", triggered_alert_count as "Triggered

Count", cron_schedule as "Scheduling Time", alert.severity as "Mode"

 

Above query is to get alerts which are scheduled in our environment it is for Search & Reporting app

 

 

For getting schedules alerts which are set in different app

|rest/servicesNS/-/[app - name]/saved/searches 

|  search alert.track=1 

|  fields title description search disabled triggered_alert_count actions action.script.filename alert.severity cron_schedule

| table title  search disabled triggered_alert_count alert.severity cron_schedule 

| rename title as "Alert Name", search as "Alert query", triggered_alert_count as "Triggered

Count", cron_schedule as "Scheduling Time", alert.severity as "Mode"

 

For getting scheduled alerts for all apps

|rest/servicesNS/-/-/saved/searches 

|  search alert.track=1 

|  fields title description search disabled triggered_alert_count actions action.script.filename alert.severity cron_schedule

| table title  search disabled triggered_alert_count alert.severity cron_schedule 

| rename title as "Alert Name", search as "Alert query", triggered_alert_count as "Triggered

Count", cron_schedule as "Scheduling Time", alert.severity as "Mode"

 

                                        Listing Alerts which are Triggered

 

Query :- 

|rest/servicesNS/-/-/saved/searches 

|  search alert.track=1 

|  fields title description search disabled triggered_alert_count actions action.script.filename alert.severity cron_schedule  |  fillnull value=0 triggered_alert_count

|  where disabled=0 AND triggered_alert_count>0

| table title  search disabled triggered_alert_count alert.severity cron_schedule 

| rename title as "Alert Name", search as "Alert query", triggered_alert_count as "Triggered

Count", cron_schedule as "Scheduling Time", alert.severity as "Alert Severity"

 

 

 

 

Listing Alerts which are not Triggered in our Environment

 

 

 

Query :-

|rest/servicesNS/-/-/saved/searches 

|  search alert.track=1 

|  fields title description search disabled triggered_alert_count actions action.script.filename alert.severity cron_schedule 

|  fillnull value=0 triggered_alert_count

|  where disabled=0 AND triggered_alert_count=0

| table title  search disabled triggered_alert_count alert.severity cron_schedule 

| rename title as "Alert Name", search as "Alert query", triggered_alert_count as "Triggered

Count", cron_schedule as "Scheduling Time", alert.severity as "Alert Severity"

 

Alerts are triggered or not can be identified by their Triggered Count if trigger count = 0 it is not triggered yet.

What's Your Reaction?

like
0
dislike
0
love
0
funny
0
angry
0
sad
0
wow
0