Since the launch of Kentik NMS a few years ago, Alerting has become an increasingly important part of how our customers use Kentik every day. And as we've expanded our visibility into network health with device-level metrics, syslog records, SNMP traps, and other telemetry, we've also expanded the range of conditions customers can monitor and alert on.

That's a good thing of course, but it also created conditions where alerting became more noisy and hard to manage -- which runs opposite our goal of providing network intelligence that provides clarity of insight and action.

As such, we are kicking off a series of improvements to make Kentik Alerting simpler to manage, less noisy, more automated and AI-assisted. Today we release the first phase of those improvements around managing alert noise with a substantial overhaul of Alert Supressions. 

TL;DR: Suppressions are now more flexible, easier to configure, faster to activate, and capable of working across different types of alerts. Read on for more.

Overview of the changes

There are three key aspects to today's post. I cover each in more detail below, but here is a quick summary: 

1: "Silencing" is going to be phased out. 

"Silencing" was initially built to prevent notifications from being generated and sent - "Suppressions", by contrast were used to avoid triggering an alert when the policy's conditions were met.

Existing "Silencing" instances will still be honored, but no new ones can be created from now on.

2: Suppressions are now the main and only concept

Under this new functionality, "Suppressed Alerts" still exist, get triggered and are stored in our SaaS cluster - in other words, there's now an attribute of every Alert Instance that indicates if it's Suppressed.

A suppressed Alert:

• Is not visible (at current) on the Alerting Landing Page
• Does not generate any notification, even if a notification channel is attached to its policy

3. Introducing Structured Alert Context

This specific feature is the one that packs the punch.

Previously, suppressions were tied to the keys defined in individual Alert Policies. If you wanted to be able to suppress alerts for a particular device or site, those attributes needed to be included as keys in each underlying policy at the policy level. And if you wanted to suppress different types of alerts, such as Traffic, NMS, or Events, you'd need to configure separate suppressions for each. This became burdensom to manage and configure across large, complex environments. 

Those days are over. 

Structured Alert Context introduces a new, common set of contextual attributes associated with every alert as its generated. Each alert producer (Flow, NMS, Syslog and SNMP Traps, Synthetics, etc.) enriches this context with relevant information about the alert, independent of the keys configured in its Alert Policy.

This means that the new Supressions engine can evaluate rules against a common set of context regardless of the alert or policy that generated it. 

For instance, let's say you're performing maintenance at a particular site. You can now create a single Suppression Rule that matches alerts associated with that site, whether they originate from Traffic, NMS, Events, or other supported sources -- without having to add the specific site keys associated with the actual policy that fires. 

4. Faster Time-to-Action

This one is simple enough -- we've drastically improved the speed at which suppressions take place. In the past, a 20min delay stood between the creation of a suppression rule and the moment at which it actively started suppressing new matching Alerts: this is not the case anymore. In our preliminary testing, newly created suppression rules were active in under a minute.

Feature tour

Okay -- let's see how this works via some product screenshots! 

First-off, Alert Suppression Rules are Named and come with a Comment field, so that users who haven't configured these can still understand which ones are ongoing or planned.

Another important aspect is that while we're still honoring ongoing suppressions that have been configured with the Legacy Model, no new one can be created moving forward, as we believe this new paradigm better serves our customers.

New Suppression Rules status

Let's go to Settings > Suppressions & Silences as this is where the new features start.
You will notice that we've added the notion of Suppression Status in the Suppressions Listing Table and associated left-side filter panel: 3 statuses are now available:

• Planned
  where both Suppression Start Time and End Time are in the future
• Ongoing
  where Suppression Start Time is in the past, and Suppression End Time is in the future
• Expired
  where both Suppression Start Time and End Time are in the past

These statuses go with the following rules:

• Planned suppressions can be fully Created/Edited/Deleted (unless they are legacy suppressions/silences)
• Ongoing or Expired suppressions cannot be Created/Edited/Deleted
• Ongoing suppressions can be stopped 

See how these statuses appear in the screen's filter bar - note that by default we're only showing Planned and Ongoing Suppression Rules.

A much more flexible suppressions creation experience

A complete overhaul of the Suppressions configuration UX has been released, it replaces the previous one.

Side note:
This new UX is also designed to be compatible with previous types of suppressions - our users using the current suppressions API (which hasn't yet been updated) will have their API-created suppressions still valid and visible in the UX - editable if they belong to the Planned bucket.

Legacy Suppressions Configurations leverage the Alert Dimension-Specific Filters. We built this new UX to be retro-compatible with the existing suppressions.
Remember that those legacy suppression configs are entirely dependent on the Keys defined in the Alert policy.

The new, standard Alert Suppression Rules can include both factors from Matching Criteria (the new fields applied to any type of Alerts) and Alert Dimension-Specific Filters both sections are AND'ed together as we evaluate which Alert gets suppressed upon instantiation.

These new Alerts can be scoped by Severity (Info, Warning, Critical...), Applications (Traffic/Flow, Protect, NMS/Event, Synthetics...) or Policy.

Note
Unless the Applications attribute is set to a list in the configuration dropdown, they by default apply to all types.

More flexible Suppression Criterias

A closer look at the "Filters" section of the "Matching Criteria" panel shows we can now build suppressions based on:

• A flexible, nested set of OR and AND conditions that feed off the Structured Context to determine which alerts need to be suppressed upon generation
• more flexible Filter conditions to capture broad sets of Alerts to suppress that have been commonly reported as needed by our users:
  • Ex: various string matching functions on "Device Name" or "Site Name" - including "String Contains" or "String Matches Regex"

Here's a view of what an intricate filter can look like:

Ability to test suppressions before setting them live

As a user is configuring suppressions in the new UX, they are now able to see how the suppression would behave on existing Alerts: this is done via the right-side panel.
The underlying idea is that users need a short feedback loop before enabling a Suppression Rule to be able to understand whether is it either too narrow or too broad otherwise the suppression creation process is known to be a wild guess.

Hitting Refresh after changing the matching criterias will re-initiate matching agains the existing alerts populated in the preview screen.

Note
A cooldown timer kicks in upon refresh to protect the backend systems from being swarmed by refresh requests - this limitation will be lifted in the future as we are currently working on scale improvements that will make it unnecessary.

Users can tailor the preview results by choosing which timespan to evaluate existing alerts for and preview on. In addition, they can select whether they want to show

• all Alerts, highlighting the ones that would be suppressed with the configured filter
• only the alerts which would be suppressed with the configured filter

Easy shortcuts to create Alert Suppression Rules

A number of Kentik Portal screens have been updated to allow simpler suppressions creation/configuration that do not require starting from the [+ Add Suppression] screen.
This is intended to support workflows where users want to create a simple/routine suppression such as for a point device, or an entire site.

Here are some good examples:

Creating a suppression rule for a point device directly from the device listing screen

Creating a suppression directly from the Device Details screen of a specific device

Creating a single bulk suppression for a group of Devices directly from the Device Listing Screen

Creating a suppression for an entire Site or set of Sites directly from the Site Listing screen

Creating a suppression for an entire Site, directly from the Site Details screen

This announcement says Phase1, what goes into Phase2 ?

As you can notice, this is just the beginning of a broader project aimed at generally improving the signal to noise ratio of the Alerting product.

We're already planning the next phases of it, which will include:

• Additional Filtering Options to configure the Suppressions Rules (like Interface Classification attributes, Collection Agents a device depends on to export its telemetry and such...)
• Automatic, dependency-based creation of Suppressions, such as when a device goes down, then all of the Alerts for its interfaces, sessions... so to be automatically suppressed
• A major overhaul of the Alerting Landing Page experience
  • for users to be able to visualize suppressed Alerts the same way they are visualizing existing ones
  • "big number" kind of stats that show users how efficient the Suppression system is, or to easily detect if a newly created suppression is suppressing more Alerts than initially intended.
• Programmatic ways to register suppressions/maintenances
  • updating the existing legacy suppressions API and/or creating a new one compatible with this newer method
  • using AI Advisor to instantiate new Suppression Rules based on a prompt

We have had a lot of customers asked for this feature in the past and we genuinely think this update satisfies their expectations - we'd love to hear what you think!

We'd also love to hear from you and what you believe should be the next evolutions of our Alerting Platform: we have a lot of other projects currently on our drawing board that we're excited to share with you, so stay tuned.