As consumers of alerts, you want every alert to be easy to understand, meaningful, and actionable. But when systems bombard you with alerts for metrics you don't need, it's hard to distinguish between the signal of what matters from the noise of what doesn't (cue alert fatigue). Claude Debussy said, “Music is the space between the notes.” That space enables resonation and expression. Music needs a degree of emptiness to be truly appreciated, and the same holds true when it comes to alerting.

We've listened to your feedback and have significantly improved our synthetics alerting capabilities to help reduce the noise and increase alert accuracy. The changes we've made with Synthetics Alerting give our customers the ability to receive more detailed alert notifications, while giving you agency over the individual metrics that are critical to your success – and perhaps more importantly, the ability to silence the metrics that are not. 

Flexible Health Options

For a long time, we have set extreme thresholds to disable certain metrics from causing an unhealthy health status in our tests. While this technically works, it does not make for a great user experience. As part of the new Alerting stack, the Synthetics team developed new switches that allow for individual metrics to be enabled or disabled when determining an unhealthy status and ultimately, an alert notification.

Single Metric Alerting & Improved Alert Notifications

Another big improvement to Synthetics Alerting is the change to alert tracking by single metrics. The system now keeps track of triggers for each health-enabled metric and will trigger alerts for them individually. When combined with Flexible Health Options, users can tailor their alerts to only the metrics that fit their use case. This provides more clarity of what is going wrong with a test when receiving an alert.

Based on user feedback, we have added a new option for the way our Network Mesh and Agent-to-Agent tests alert. Previously, alerts were tracked by test – calculating health and alerts for the test as a whole; however, this was not a perfect fit for every use case. 

The Synthetics team has now deployed changes allowing alerts to be calculated and tracked on a per agent basis. With this feature enabled, each agent in a test will have its own trigger for the test's health thresholds and one test can have multiple ongoing alerts for different agent-to-agent pairings. Simply select the Agent option in the Status Calculations field to have the test tracked on a per agent basis. 

If you have any feedback on this new option, please reach out to your CSM. We are continuing to improve our alerting options and aim to bring even more health and alerting features to Kentik Synthetics as we migrate to Alerts 2.0.

At Kentik, one of our core goals with synthetic monitoring is to enable your team to quickly and effectively triage any issues that come your way. This effective triage process starts with ensuring alerts are easily delivered to the right team in their preferred notification method. 

Previously, alerts associated with agent issues (downtime or recovery) would show and alert in the same notification channel and test results summary in the UI. Based on feedback, we realized this setup was not optimized for customer needs. 

We are pleased to share that agents can now be configured to send status alerts to their own notification channel. Simply navigate to the  'Synthetics' > ‘Agent Management’ section of the portal, select an agent, and click Configure to set up status alerting and notification channel(s) for that specific agent. To see agent downtime details, select the agent and click on the ‘Agent Downtime' tab.

With this setup, an email notification is sent for down or recovering agents.

The notification links to the ‘Agent Downtime Details’ tab under the ‘Agent Management’ part of the portal.

In many areas of Kentik Portal, users now have to input credentials that our systems will use for a variety of purposes: 

• HTTP Synthetic tests
  • HTTP(s)/API tests
  • PageLoad tests
  • Transaction tests
• Kentik-registered devices
  • SNMP polling community strings
  • Streaming Telemetry Credentials
  • BGP MD5

We are introducing Credentials Vault as an elegant way to manage these more centrally and securely.

Where are credentials the most used in Kentik Portal ?

Credentials HTTP Synthetic Tests

Imagine your company runs multiple tens or hundreds of Synthetic tests. Now also imagine that one of the credentials used in these tests needs rotating, which happens quite frequently. This would normally require a user to go and edit all of these tests one by one to update the credentials. This manual update process poses multiple problems:

• The obvious time sink involved to reconfigure every test
• If one of these credentials becomes compromised, users are unable to quickly swap out credentials in an efficient and quick manner, making it difficult for our users to harden their security posture and rotate credentials frequently.  

We aimed at fixing this by releasing our Credentials Vault.

Kentik-registered Devices

To enrich the Network Telemetry from your Kentik-registered devices, you provide us with SNMP polling credentials (whether v1, v2c or v3) to pull such attributes as interface descriptions and names at frequent intervals. Our users routinely have multiple hundreds of devices, and this poses the following issues:

• Copy/Pasting credentials across devices definitely increases chances of a typo
• These credentials are defined with each device registered with Kentik -> it makes changing them on large sets of devices time consuming and tedious
• Again, local definition of credentials increases the friction preventing companies from being able to efficiently and frequently rotate credentials

This is another reason we built Credentials Vault.

What is the Credentials Vault 

The Credentials Vault can be accessed in the company menu, as shown in the screenshot below:

It is a central facility allowing Kentik users to securely store their credentials.

1. Securely: 
   • All credentials are double encrypted at rest with a unique key for all Kentik tenants and a global key that only our backend systems know
   • Credentials are write-only: you can modify an existing credential, but you cannot view it
   • Management Capabilities are governed by our newly release RBAC engine
2. Centrally: 
   • Credentials defined in the Vault can be used in different parts of the portal – the initial release focuses on Synthetic tests, but we will extend it in the future quarters.
   • Modify a credential in use, and any portal component leveraging it (Synthetic Tests, and even more in the near future) leveraging this credential will immediately use the updated one. 
   • Delete a credential and all tests immediately stop functioning
3. Flexibly: Each credential is either
   • 1) a templated credential with fixed fields (this feature will be leveraged in a future release)
   • 2) a free form Key/Value store: this means you can store multiple useful fields within a single credential – a good example is for an HTTPS API Credential where you will store
     • the name of the HTTP header to put your token in
     • the username part of the header value
     • the token part of the header value

Using a Credentials Vault secret in Synthetic HTTP(s) Tests

With your credentials ready, you can now summon them in any Synthetic HTTP test, and selectively configure each field of your test with a field of your choice from this credential, as shown below:

Clicking on the Credentials Vault button will summon a credentials manager where you will be able to pick from and copy/paste into whichever field you want, see below:

As you can notice, the fields of the test where the credential key/values are summoned do not contain the actual value, but a programmatic expression of them, such as $vault("kentik_api_token.token_value"). The value for a key in a credential follows this nomenclature: $vault(".") and assigns the value for credential_key to the test configuration field.

Note:
In order to make this possible, you will notice that Credential Names and Key Names within a credential follow strict rules. This is simply because these can also be summoned in a transaction test, which is the reason why we wanted them to have a javascript friendly format.

What's next ?

We are already working on the next areas of Kentik Portal where Credentials Vault is going to be made available.
One of them is a secret project we are currently working on (be patient, it's coming very soon!), and the other obvious one is Kentik-registered devices, which we are hoping to release within the first quarter of this year.

Next on the list, we are evaluating requests to add Synchronization with Secret Vaults as a Service providers such as AWS or Hashi Corp's Vault – more to come on that in the future.

Lastly, we will eventually turn to Kentik Integrations such as Notification Channels, so that credentials from the Vault can be used in their configurations.

We have improved our options for synthetic network testing by supporting different protocols for basic network tests (Ping tests). In addition to ICMP, we have expanded the options to test TCP and UDP connectivity between Synthetics private agents or between agents and any target.

As a part of this feature, we have added the ability to start the TCP or UDP listener on our Synthetics private agents which acts as an “echo service”:

• Specify one or more TCP ports to listen and respond to TCP connections
• Specify one UDP port to listen and respond to UDP packets as an echo service

Please note that UDP protocol is susceptible to spoofing attacks, therefore running this service is recommended only within private networks and with the appropriate access control to the open UDP ports by using network access lists or firewall rules.

Additionally, the network connectivity test configuration is extended with the following testing methods/protocols:

• TCP: Based on the TCP Syn packet sent to a configurable port and expecting the TCP Syn+Ack packet response from the target IP.
• UDP-ICMP: Based on the UDP packet sent to a configurable port and expecting ICMP Port unreachable received from the target IP.
• UDP-ECHO: Based on the UDP packet sent to a configurable port and expecting the UDP packet response from the target IP.

Our customers have requested a feature to mute the alarms and notifications for Synthetic tests during their maintenance windows or other activities on their network. Alert Suppression is now available under the Alerting and Notifications section inside the Test Settings.

Users can specify the start and end time of their silence/maintenance window and will not be alerted during the selected period.

The DNS Server monitor tests are improved with the additional Health check Allowed DNS Results. This field allows the user to configure the list of expected values, which depend on the selected DNS Record Type:

• If the DNS Record Type is “A” or “AAAA”, this field allows the user to configure the list of IP addresses that are considered “healthy” for the resolved IP address of the testing DNS query. If the DNS Server monitor test query resolves to multiple IP addresses, they should all be part of the configured list. If the resolved IP addresses are not in the allowed list, the test Health will be set to Critical.
• If the DNS Record Type is “CNAME*”,* the field allows the user to specify a comma-separated list of canonical name records that are considered "healthy”.
• If the DNS Record Type is “NS”, the field allows the user to specify a comma-separated list of server names that are considered "healthy”.

The following screenshot shows the Allowed DNS Results field in the DNS Server monitor configuration.

The Kentik Synthetics HTTP/API and Web Page Load tests now support checking of the HTTP Response Headers to determine the Health condition of the test.

The user can configure one or more expected Response Headers:

• If the Value is omitted, then any value will be considered valid
• Value can be specified as a regular expression
• Value can be specified as a comma-separated list of possible valid values

If the configured headers exist in the response, but are not equal to the expected values, or do not exist in the Response at all, the test health will be set to Critical.

The screenshot below demonstrates the example configuration of this feature.

DNS was designed in the 1980s when the Internet was much smaller, and security was not a primary consideration in its design. As a result, when a recursive resolver sends a query to an authoritative name server, the resolver has no way to verify the authenticity of the response. DNSSEC was designed to address this issue.

DNSSEC adds two important features to the DNS protocol:

• Data origin authentication allows a resolver to cryptographically verify that the data it received actually came from the zone where it believes the data originated.
• Data integrity protection allows the resolver to know that the data hasn't been modified in transit since it was originally signed by the zone owner with the zone's private key.

Up until today, the DNS Server Monitor Test only allowed a user to monitor the DNS resolution for a given hostname from specified Name Servers. Users can be alerted if the resolution time crosses a particular threshold or if an unexpected DNS response code is received, or a non-allowed IP is answered.
However, these tests previously did not validate the DNSSEC trust chain of the received record.
Enter DNSSEC Validation.

How can you configure DNSSEC validation?

When enabled for a given domain, the test will recursively check the validity of each signing entity in the chain from the authoritative name server up to the root server. The result will be either a positive or a negative response. The DNSSEC record is either fully verified or it isn’t.

When the option is active, the test results will show the DNSSEC validation status for each one of the Agents involved in the test.

Validity of DNSSEC is based on querying DS and DNSKEY for any of the successive parts of the domain name: for a DNS test target of subdomain.domain.tld, each of tld., domain.tld., subdomain.domain.tld. and . (root) will be tested.

Traces for the DNSSEC validation for each agent will be available by clicking on their respective status icon on the previous screengrab.

DNSSEC validation impact on subtest health

Health options remain the same as the DNS Server Monitor test. DNSSEC validation will have a boolean result. If validation is successful it’s a healthy result, if not, it's critical. 

If enough agents have a critical results (see screenshot above) to meet the sub-test threshold condition, an alert will be triggered.

IMPORTANT NOTE: App Agents vs Network Agents

Be advised that setting DNSSEC validation is available to all agents except Private Network Agents. As a reminder, our new Private App Agents not only include all of the capabilities of the legacy Network Agents, but also include the capabilities required for advanced Web tests such as Page Load Tests and Transaction Tests.

If you currently run our legacy Network Agents, please consider replacing them with our new App Agents to gain access to all of the feature we will add in the future. Kentik's entire fleet of Network Agents has already been migrated, and support for the Network Agents will be phased out in 2023 (more to come on this soon)

As of now, users can set their own DSCP values in any Synthetic Monitoring Network test.

What's new?

While Network Tests initially defaulted to Diffserv codepoint "0: Best Effort" for every Network test, they can now be set to any value, allowing our users to more realistically test the behavior of their own network and test along all classes of services they implement.

To see it in action, edit or create any IP Hostname Network test: in the Advanced Options, both Ping and Traceroute aspects of it can be set to different DSCPs.

IMPORTANT NOTE: App Agents vs Network Agents

Be advised that setting DSCP values for the aforementioned tests is available to all agents except Private Network Agents. As a reminder, our new Private App Agents not only include all of the capabilities of the legacy Network Agents, but also include the capabilities required for advanced Web tests such as Page Load Tests and Transaction Tests.

If you currently run our legacy Network Agents, please consider replacing them with our new App Agents to gain access to all of the feature we will add in the future. Kentik's entire fleet of Network Agents has already been migrated, and support for the Network Agents will be phased out in 2023 (more to come on this soon)