Today, we are adding a set of new endpoints related to various workflows in the Kentik Portal. 

Read on!

The first thing to know before we get into the details is that you can always access our API sandbox from the navigation menu here:

We have recently added a set of previously unreleased endpoints to our v6 API, all can be visible at the following URLs in the portal:

• https://portal.kentik.com/v4/core/api-tester/ on the US SaaS Cluster
• https://portal.kentik.eu/v4/core/api-tester/ on the EU SaaS Cluster

These new endpoints cover three main areas of functionality:

• Custom AS Groups CRUD functionality
  This new endpoint set only covers the management of these AS Groups. Ongoing work is scheduled for the Data Explorer Query API to be compatible with AS Group rollups, but the release date isn't yet set.

  

• Workflow APIs

  • for Capacity Planning
    The set of endpoints we’ve added to Capacity Planning is for now exclusively centered around viewing either a summary of all capacity plans or details of specific plans. 
    

    

    

  • for Kentik Market Intelligence
    This set of API endpoints allows users to either get any ranking with any Customer-base type in any market or get customers and providers of any given network in any given local market.

    
    

    

Important note
The sandbox shown in the article is the one of the v6 of our API. We still have a v5 API which still covers a large amount of legacy unmigrated endpoints - the testing UI for them is located here:
https://api.kentik.com/api/tester/v5

This may not look like much, but it's big. I have been a Product Manager for more than 6 years at Kentik and when joining in 2016, I remember asking why our users couldn't give names with dots (".") and dashes ("-") when registering them in Kentik.

It wasn't easy back then but we've progressively removed all the technical hurdles to progressively enable this. (Trust me, it wasn't easy).

Today, I am very happy to announce that your device can now contain dots and dashes in their names!

As they are saying: "Great things come to those who wait"

Important note
While you can now rename all your existing devices, please be warned that any Saved Filter relying on Device Names will have to be updated accordingly to reflect any change you make to its name.
Similarly, if any dynamic Cost Group or Capacity Group relies on device naming you will also have to update them

Our CDN Analytics workflow relies on a detection engine which both takes into account your own Kentik configuration (via Interface Classification) and its constant scanning of the internet to detect IPs from CDN Servers out on the Internet. In this release, we added means for our users to help it better detect, or take into account undetected CDN caches that they are embedding in their networks.

Important note: New configuration possibilities offered by this workflow update are highly advanced - if you feel like you need to use this feature, please seek advice from your Customer Success Engineer.

Understanding Kentik's CDN detection engine

Kentik enriched flow data includes src_cdn and dst_cdn fields, and these are accessible in Data Explorer as well as the namesake workflow CDN Analytics, they are also leveraged within the OTT Service Tracking workflow. 

Values for these flow columns come from Kentik’s special sauce: every day a complex engine runs and evaluates multiple data-sets to map as many IPs as possible to CDNs. src_cdn and dst_cdn are then populated in flow records using this global (<IP>,<CDN>) mapping table.

Datasets used by the engine for classification

CDNs can be a mix of servers either hosted on the CDN’s own network infrastructure, or inside of other networks. While servers hosted on the CDN’s own infrastructure are somewhat easy to detect, last mile embedded caches aren’t, because they’re addressed with the embedding broadband ISP’s address space.

The CDN detection engine uses a combination of the methods below, when it runs every day:

• Internet Scans data
  Determines how public CDN cache server IPs are detected, as well as a portion of last mile embedded caches.
  This process is not detailed in the present article.
• Kentik Customer data
  This mechanism helps with detecting IPs of embedded caches that are not identified by the previous datasets.
  This detection mechanism is based on a customer’s Interface Classification accuracy.
  The engine pulls and inspects all interfaces whose Connectivity Type is set to Embedded Cache and look at the Provider value for the interface. We perform a loose match with the Provider field then assigns the interface's subnet to the CDN that has been matched with our database.

This example shows an Interface Classification Rule which will result in the associated interfaces' subnets to be picked up by the CDN detection engine and associated to a CDN.

Corner cases for Embedded Cache detection

While this approach has shown good results, some customer setups make it difficult to entirely (or sometimes correctly) detect Embedded Cache IPs. The examples below detail such corner cases.

Situation #1

Here, the interface tagged Embedded Cache is actually a /31 or /30 point-to-point subnet to connect with another Network Device on which the Embedded Caches are located, with the aggravation that this Network Device is not sending Flow Telemetry to Kentik.

In this case, two things will happen:

1. the engine will mislabel the /30 or /31 to a CDN if its provider field is set and matched (this happens for Facebook FNA caches, CDN77 caches... which often come with a router shipped with the caches by the CDN provider)
2. the engine will be missing the Caching Server Subnet below for not having any Interface Classification from the non-kentik-registered router.

Situation #2

This situation is pretty similar to the previous one, with the additional complication that the interface sending flow data is not tagged as Embedded Cache by the Interface Classification rules.

In this instance above, the CDN Engine cannot detect the subnet where the cache servers are, because it doesn't have an Embedded Cache interface to pickup the subnet from.

Solving for these situations: the new CDN Analytics configuration screen

Inspecting and amending CDN Engine detected cache entries

To remedy both of the aforementioned situations, the Configuration button in the CDN Analytics workflow will allow for fine grain additional configuration.

The first tab of this configuration screen, Detected Embedded Caches, will:

• List all the subnets detected from Interface Classification and the associated Provider values, as well as the CDN that's been assumed from the Provider value
• Let the user disable those that are tagged Embedded Cache but whose subnet is not a cache subnet (example of Situation #1)
• Add a CDN Provider to any interfaces labeled Embedded Cache where the Provider value hasn't been set, therefore which the CDN Engine couldn't make a CDN attribution decision on.

Manually declaring undetected Embedded Caches

To also address the situations of having caches behind non-Embedded Cache classified interfaces, and therefore undetected by the CDN Detection Engine, users have the option to manually declare CIDR blocks and assign them to CDNs, via the Additional Embedded Caches tab of this configuration screen.

Entries in this configuration section will now be picked-up with every daily run of the CDN Detection Engine and will be added to the global  (<IP>,<CDN>) dataset in use by all our customers.

Understanding how the CDN Analytics workflow leverages your Network Telemetry

Bolting src_cdn and dst_cdn values into your enriched flows is not the only thing that needs to happen for your CDN Analytics workflow to display accurate data.

The workflow needs to inspect the right flow data from your Network Devices to deliver a cohesive analysis of the CDN traffic that not only enters your network, but also is generated from embedded caches within it. To do so, this workflow uses this generic filter:

As you will notice, this filter will work perfectly as long as all of the Embedded Caches in your network are connected to a router exporting flow data to Kentik. When this is not the case, the flow data to include will simply not be collected by Kentik from the interface facing the cache. 

it is important to understand that mapping Cache Server IPs and selecting all the cache related flow telemetry are two aspects that need to work for the CDN Analytics workflow to display accurate results.

In the previously discussed Situation #1 and Situation #2, flows coming from the cache servers will not be considered by the filter displayed above, therefore we need ways to add these to the filter-set that regulates what flows the CDN Analytics workflow will consider.

Tuning the CDN Analytics workflow to include non Embedded Cache traffic

Let's go back to Situation #2

Now we want the CDN Workflow to include traffic from the Caching Server Subnet to the data it will display in its analysis. This will be done by using the Advanced Filtering section of the CDN Analytics configuration options:

As you can see in the screenshot above, traffic from eth0/0/1 on the edge-01 network device coming from the Cache Subnet would not normally be caught by the default filter, adding it modifies the CDN Analytics behavior to include it now.

The Streaming Telemetry (ST) is now officially supported in the Kentik Portal. Users can enable Streaming Telemetry from the Device configuration dialog which is shown in the screenshot below.

This will enable Kentik Saas Ingest or Kentik kproxy to start receiving Streaming Telemetry from that Device.

The details about the configuration of the kproxy for using Streaming Telemetry can be found on the this KB article.

The status of the Straming Telemetry is shown on the Settings → Devices page with the additional “ST” flag. This flag shows the status only for devices which have ST enabled. The Streaming Telemetry status is additionally shown when the device is selected or in the Network Explorer Device Quick view page under the “More Info” Tab. Example screenshots are shown below.

At the moment, Kentik supports the following ST formats:

• Junos native JTI telemetry sent over UDP. Currently supported sensors are for interfaces statistics:
  • /junos/system/linecard/interface/
  • /junos/system/linecard/interface/logical/usage/
• Cisco IOS XR native dialout telemetry with self-describing GPB format which is sent over TCP. Currently supported sensor path is for interfaces statistics:
  • Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface/latest/generic-counters

If a user wants to configure device to send ST directly to Kentik SaaS ingest, it should send it to port 20023. If the ST is sent to kproxy, it can be sent on any port, which is configurable as part of the kproxy configuration (a port 9555 in used Kentik’s documentation as an example).

More information about the ST can be found in our Knowledge Base:

• SNMP and ST
• Kproxy configuration for ST
• Example device configuration snippets for Juniper MX and Cisco XR

Please let us know if you are interested to enable Streaming Telemetry from your devices and if you would like to have support for additional sensors or other Telemetry formats?

DNS was designed in the 1980s when the Internet was much smaller, and security was not a primary consideration in its design. As a result, when a recursive resolver sends a query to an authoritative name server, the resolver has no way to verify the authenticity of the response. DNSSEC was designed to address this issue.

DNSSEC adds two important features to the DNS protocol:

• Data origin authentication allows a resolver to cryptographically verify that the data it received actually came from the zone where it believes the data originated.
• Data integrity protection allows the resolver to know that the data hasn't been modified in transit since it was originally signed by the zone owner with the zone's private key.

Up until today, the DNS Server Monitor Test only allowed a user to monitor the DNS resolution for a given hostname from specified Name Servers. Users can be alerted if the resolution time crosses a particular threshold or if an unexpected DNS response code is received, or a non-allowed IP is answered.
However, these tests previously did not validate the DNSSEC trust chain of the received record.
Enter DNSSEC Validation.

How can you configure DNSSEC validation?

When enabled for a given domain, the test will recursively check the validity of each signing entity in the chain from the authoritative name server up to the root server. The result will be either a positive or a negative response. The DNSSEC record is either fully verified or it isn’t.

When the option is active, the test results will show the DNSSEC validation status for each one of the Agents involved in the test.

Validity of DNSSEC is based on querying DS and DNSKEY for any of the successive parts of the domain name: for a DNS test target of subdomain.domain.tld, each of tld., domain.tld., subdomain.domain.tld. and . (root) will be tested.

Traces for the DNSSEC validation for each agent will be available by clicking on their respective status icon on the previous screengrab.

DNSSEC validation impact on subtest health

Health options remain the same as the DNS Server Monitor test. DNSSEC validation will have a boolean result. If validation is successful it’s a healthy result, if not, it's critical. 

If enough agents have a critical results (see screenshot above) to meet the sub-test threshold condition, an alert will be triggered.

IMPORTANT NOTE: App Agents vs Network Agents

Be advised that setting DNSSEC validation is available to all agents except Private Network Agents. As a reminder, our new Private App Agents not only include all of the capabilities of the legacy Network Agents, but also include the capabilities required for advanced Web tests such as Page Load Tests and Transaction Tests.

If you currently run our legacy Network Agents, please consider replacing them with our new App Agents to gain access to all of the feature we will add in the future. Kentik's entire fleet of Network Agents has already been migrated, and support for the Network Agents will be phased out in 2023 (more to come on this soon)

As of now, users can set their own DSCP values in any Synthetic Monitoring Network test.

What's new?

While Network Tests initially defaulted to Diffserv codepoint "0: Best Effort" for every Network test, they can now be set to any value, allowing our users to more realistically test the behavior of their own network and test along all classes of services they implement.

To see it in action, edit or create any IP Hostname Network test: in the Advanced Options, both Ping and Traceroute aspects of it can be set to different DSCPs.

IMPORTANT NOTE: App Agents vs Network Agents

Be advised that setting DSCP values for the aforementioned tests is available to all agents except Private Network Agents. As a reminder, our new Private App Agents not only include all of the capabilities of the legacy Network Agents, but also include the capabilities required for advanced Web tests such as Page Load Tests and Transaction Tests.

If you currently run our legacy Network Agents, please consider replacing them with our new App Agents to gain access to all of the feature we will add in the future. Kentik's entire fleet of Network Agents has already been migrated, and support for the Network Agents will be phased out in 2023 (more to come on this soon)

This may be a minor, but a widely requested improvement on the device screen: all SNMP community and SNMPv3 passphrases are now obfuscated in the UI.

This complies with widespread company policies to never display passwords in a web UI.

See screenshot below

When digging around in enriched Network Telemetry data, you'll find yourself noticing a bump, or a trough in any of the displayed time series and think to yourself: 

"Hmmm... is that peak an anomaly, or did it behave the same last day|week|... at the same time?"

Ask yourself no more, "Compare over previous period" is here!

In a nutshell, this new Data Explorer feature lets you look at the same time range you are currently looking at, but in the past, and help you visualize variations between each of your query result's time series within the same time range in the past.

This feature comes with a streamlined redesign of the Time Range query panel in Data Explorer.

Hitting the Compare over previous period switch will unveil options to be able to compare the current time range, to the same one at a configurable time in the past, such as: "this hour with the same hour yesterday, or a week ago..."

Upon selecting this option, your data-table will now include two new tabs:

• A Previous Period tab showing you the TopN time-series for the same time range in the past
• A Comparison Summary tab, outlining in an sortable fashion such useful insights such as: 
  • previous current rank for the time series vs previous rank,  
  • rank variation for the time series
  • and the percentage of change (sortable) in the selected metrics between Previous and Current periods

From any of these 3 tabs, you will then be able to select any time series and look at a combined line chart displaying both the current and previous period for the selected time series.

Switching between Current Period and Previous Period will display the full set of time series for either periods - always leveraging the same convention:

• Plain Line:        Current Period
• Dashed Line:   Previous Period

Now go ahead, play around with the feature and let us know what you think of it.
As always, you'll always find in Kentik Product Management a friendly ear to suggest improvements to this feature (and any other for that matter), do let us know!

Kentik’s Data Explorer is enriched with additional query options which influence how time series will be visualized and calculated. These options are located in the Advanced section of the query panel as shown on the screenshot below

The Query execution performed against the Kentik Data Engine consist of the two steps:

1. Initial flow aggregation into samples - Flow Aggregation which is performed on the configurable time Window. For all the query (filter) matching flow’s the summation of bytes or packets will be performed, grouped (broken down) by all selected Flow Dimension. The resulting flow aggregation samples will always represent an average Bit Rate (Packet Rate) across all the summed flows, because the sum of all bytes/packets is divided by the Flow Aggregation Window.
2. Construction of the Time Series - From all the Flow Aggregation samples returned from the initial aggregation, the Time Series will be constructed for selected Top N series by volume. Time Series Buckets are constructed by aggregating multiple Flow Aggregation samples using the selected Aggregation function.

Additional Data Explorer Time Series query options consist of the following 3 options:

• Flow Aggregation Window: This configures the time window which is used for the initial Flow Aggregation into samples. Available options are 1, 5, 10, 20 minutes for Full Dataseries and 1, 6, 12 hours for Fast data series. If the Auto is selected, the behavior will be the same as the usual Data Explorer behavior so far. current Further limitations apply:
  • 1 minute available for up to 3 day wide queries
  • 5 minutes available for up to 14 day wide queries
  • 10 minutes available for up to 30 day wide queries
  • 20 minutes available for up to 60 day wide queries
  • no limitation for the use of Fast dataseries.
• Time Series Bucket Width: Time Window used for the construction of the Time Series from Flow Aggregation Samples. The available options are 10 min, 20 min, 30 min, 1h, 6h, 12h, 1 day, 1 week.
  • This configuration option only takes effect if the Time Series Bucket Aggregation is not "None", 
  • The value always has to be larger than the selected Flow Aggregation Window.
  • The option “Auto” always maps to 1h buckets.
• Time Series Bucket Aggregation:This option defines the function which will be performed while aggregating multiple Flow Aggregation Samples into Time Series Buckets. The available options are:
  • None - does not perform any aggregation to the Time Series Buckets, so only Flow Aggregation Samples are shown
  • Average - Calculates the average value of the Flow Aggregation samples in each Time Series Bucket
  • Count - Counts number of Flow Aggregation sample occurrences within the Time Series Bucket
  • Sum - Calculates the sum of all the Flow Aggregation samples in each Time Series Bucket
  • Maximum - Selects the Maximum value of the Flow Aggregation samples in each Time Series Bucket
  • Minimum - Selects the Minimum value of the Flow Aggregation samples in each Time Series Bucket

Connectivity Costs is one of the workflows in the Edge module that seems to be the most successful with our users. It allows users to centralize the financial modeling of their Transit, Peering and IX contracts in one place, and track costs for each of these throughout the month, and every past month.
For a refresher on this workflow, take a minute to read this blog post from 2021 on Why and How to Track Connectivity Costs.

Today, we add quite a few highly anticipated features to it, read on!

Landing page calendar drawer now collapsible

The drawer with the invoice and contract expiry schedule is now collapsible, which was a common request from users whose contract dates are always the same: it can now be hidden, and the preference will stick for each user.

Dynamic Interface Selection

In the previous iteration of Connectivity Costs, interfaces has to be manually selected and added to the relevant Cost Group in order to bind them to a contract and financial model. For networks with a vast footprint, these external facing interfaces keep coming in and out of the picture, and users demanded to be able to manage this part of the configuration more efficiently. This is now done: from the already popular Capacity Planning workflow, we have ported the Dynamic and Static Interface Selector module into Connectivity Costs.

In addition, we've improved it significantly to make things even easier to maintain - amongst others, users can now assign interfaces in a cost group based on:

• Interface Name and Description regular expressions
• Device Names, including regular expressions (comes in handy if a portion of your device names is Site or Function related - think edge-01_ams01 for an Edge Router in Amsterdam)
• Device Labels, allowing you to exclusively focus on specifically tagged devices
• Last but not least: users can now choose between Logical Only interfaces, Physical Only interfaces, or both - for instance, depending on which of the physical or logical part of an Aggregate logical interface your SNMP accounting is going to come from

Interface or Group level additional costs expressed in any currency

Users can add Cost Group level (monthly or yearly) costs to their cost groups, but can add Interface Level costs as well. Until now, the currency of the parent Cost Group would be used to evaluate these additional costs into the daily computations.

With this release, users can now select the currency for both. Here's an example of where it comes in handy: optional interface charges are often used to document cross-connects. Our customers also tend to use local providers in whichever area they are sourcing bandwidth from. This often results in Bandwidth as documented in the Cost Group financial model being expressed in the local currency, and data-center cross connects (usually from a global data-center contract) expressed in another currency. Problem solved !

While at it, we made the Interface-Level UI for adding/editing costs more usable, see for yourself - in the example below, the cost group is in dollars and the interface charges for the cross connect in Euros. 

New Computation Methods

Based on feedback from our users, we also added a few new computation method flavors as to how our engine should compute monthly invoices. To the existing Peak of Sums and Sum of Peaks we added two new models:

• Peak of Sums (Multiple interface sets)
• Peak of Sums (Max of In and Out)

And to make sure our users select the right one when configuring a cost group, we also gave better in-UI explanations of how each computation method works in the back-scenes:

What comes next ?

In future quarters, we are going to both extend the workflow to deal with non-edge costs, and also create functionality for users who have documented their costs into the workflow to be able to price any slice of their traffic.

Stay tuned, and in the meantime do let us know what you'd like to see next in this workflow !