5 years ago

April 2017 Update

Ultimate Exit Release #1

Fasten your seat-belts, this one is a big deal. It’s the first release within a bigger plan for end-to-end visibility of your traffic, which is a holy grail objective of flow data reconciliation. What do we mean by “end-to-end visibility”? It means an easy way to figure out what volumes of traffic are flowing in and out of your network, from any source to any destination network.

A great example of this is assessing potential peer or transit prospects. How many times have you had to toggle between multiple spreadsheets that contain only approximations of traffic to or from various ASNs, getting bogged down in hacked, convoluted excel formulas, all in order to guess the ROI of what should be a simple decision?

What about trying to figure out how much traffic from a peer is being routed locally versus over more costly long-haul links? You need to able to figure out precisely at the site and device level — and at the interface level in the future — the traffic flowing between network entry and exit points.

It turns out that the sophistication of flow consolidation and reconciliation needed to achieve this task is beyond home-grown tools, data infrastructure, and software engineering capacities of many network engineering teams. And for good reason. It’s a hard problem.

Voila! New Exit Dimensions in Kentik Detect

Introducing two newly added destination dimensions (fanfare, please):

Ultimate Exit Site
Ultimate Exit Device

How do you use these? Let’s say you are a transit provider. You move packets from content providers to eyeball ISPs, and carry them over a costly global backbone. You want to look at the traffic you’re exchanging with one of the major content providers like Google, and see where it comes in, and where it comes out of your network.

Let’s further assume that you run a well organized network, so you indicate within your Interface description nomenclatures any interconnections with Google. This means you can easily include these interconnects with a simple filter. For example:

BTW, if you know that you’re going to be looking at these often, you can also make yourself a nice Saved Filter (see below) and just apply it any time you need it.

Then you can use that saved filter in any Data Explorer query you’re working on.

So here’s what you want to look at, in sequence:

The site where the traffic enters the network.
The site where the traffic leaves the network.
The next-hop Network.
Which eyeball network it is terminating at, i.e. Destination AS.

Using Kentik Detect’s handy new dimensions you can now answer this question with the following query:

For a useful visualization, select the Sankey display type:

Looking at the generated Sankey diagram (above), you can now instantly see what traffic is flowing between the entry Site and the Ultimate Exit site, and which eyeball networks are reached. What you would typically do at this point is look at where transport is the most expensive or least performant between your Entry Site and Ultimate Exit site and optimize for either of them.

In the above Sankey chart, you can see that you’re shipping a lot of traffic from Frankfurt to Marseilles. So a few questions come to mind that can be explored further in Kentik Detect:

Should you track Google’s ability to PNI in Marseilles and save yourself some Frankfurt→Marseilles transport costs?
Do you want to review your prices for transport for London→Marseilles based on how much of that capacity is consumed by your Google PNI?
What portion of the private links between Frankfurt and Marseilles is going to those Google PNIs, and therefore what’s the real ROI you’re getting from these links?

You can’t even start this ROI exploration when you’re stuck in spreadsheet hell. Stay tuned, because there’s a lot more coming over the next few months in this arena.

Custom Dimensions Update

Our Custom Dimension infrastructure has been upgraded, allowing us to upgrade our default provisioning rules:

Max Custom Dimensions per customer account: increased from 5 to 10.
Max characters per dimension: increased from 12 to 128.
Max populators: upgraded from 5000 per dimension to 10,000 overall across all dimensions (no additional per-dimension limit).

User Based Filtering [PREVIEW]

Every now and then we will preview an upcoming feature. We also believe that occasionally there is value in releasing an early/crude version of a feature-set in order to get early feedback from our users, which we can then use to quickly iterate until we arrive at the feature that users really want. In the case of User-Based Filtering (see Knowledge Base article), we are previewing here a feature that we have decided to introduce as an early release.

Kentik Detect currently supports two different user levels: Member and Admin. User-Based Filtering allows an Admin user to apply a user filter that restricts the data available to a Member user. The underlying idea is for Admins to be able to grant (very) granular rights on what specific Members are allowed to see and/or query.

Admin users can set up a user filter on the Users page (Admin » Users).

A user-based filter is composed like a filter in the Filters pane of the Data Explorer sidebar. Once a user filter is associated with a given user, these filters are systematically appended (ANDed) with any query run by that user, including:

Data Explorer queries via Kentik Portal UI
SQL queries from the SQL Query explorer or via PGSQL connections
API queries

One use case example is allowing only certain users to query flows from backbone routers, as shown in the following screenshot:

Another example, shown below, allows certain users to query only flows for CUSTOMER interfaces on ‘Ashburn DC3’ and ‘Ashburn DC4’:

As explained above, we have released the minimum amount of functionality for this feature, and hope to leverage the feedback of interested users to iterate it.

Some open questions we have for this feature include:

Should filtered users be made aware in the UI that they are being filtered? In the current version of this feature, the user wouldn’t know.
If filtered users are made aware, should we indicate a permanently locked filter setting in the Data Explorer?
Should we let users know they are being administratively filtered, but not indicate what the filter constraints are?
Should the display of filtering information be administratively configurable at the user level?
How do we mention or indicate user filtering in the API and SQL? For example, when a user submits a SQL query, should we return a modified version of the submitted requests with the appended filtering in its SQL form?

Please let us know your feedback on support@kentik.com. Is this a useful feature that you would like to rely on? What should the next iteration look like?

Sampling Rate

This is one for the nerdier users out there. As you may know, our ingest platform includes smart ways of re-sampling flows exported by your devices to match your contracted FPS. We’ve been improving this functionality quite a lot recently. Our goal is to resample accurately and keep the resampling-bound distortion as close to zero as possible.

In order to keep our engineering work accurate, we actually had to add Sampling Rate to our available dimensions, metrics, and filters, as shown in the images below:

Available Dimensions:

Available Metrics:

Available Filters:

This could come in handy on your end when debugging potential flow sampling misconfigurations.

Extra Data Explorer Niceties

As we see our customer’s usage of the Data Explorer evolve we often throw in additional convenience features that we think will streamline the overall user experience. This time around, we’ve added a couple of convenience tweaks, both of which are geared towards making queries return faster by allowing users to optionally skip certain processes.

Disable Total: You can now disable computation of Total over a metric if you already know you aren’t interested in looking at the total value for your breakdown. This saves processing time on our mid-layer, resulting in faster return of query results.

Disable Hostname lookups: You can also now disable Hostname lookups directly from the Data Explorer query panel, which reduces query response time because IPs won’t need to be reverse DNSed before returning the results of an IP/CIDR breakdown.

With reverse DNS enabled:

With reverse DNS disabled:

Alerting Update

Syslog Alert Notification Channel

We’ve just added the capability for you to ship alert notifications to good ole Syslog infrastructure. This has been a recurring ask since we’ve released v3 of our Anomaly Detection and Alerting platform. Your voice has been heard! Syslog alerting works in the same way than the JSON Webhook feature does, which is by offering a new type of notification channel, aptly named “Syslog.”

When configuring a threshold in an Alert Policy (Alerting → Alert Policies → edit a policy), you will notice that in addition to the existing Email and JSON webhook options a new entry has been added to the Create Notification Channel button. You can tune all of the config knobs when you create the channel, including Port, UDP/TCP transport, Syslog Severity, and Syslog Facility.

New Alerting dimensions and filters

We’ve just added new support in our Alert Policies for:

IPV6 (for Dimensions as well as Filters)
inet_family (for Dimensions as well as Filters – this is to select IPv4 vs IPv6)

5 years ago

March 2017 Update

Major Alerting v3 Updates

Custom dimensions are now supported in Alerting

Anomaly detection users can now leverage all the profiling power of Kentik’s Alerts capabilities with their own Custom Dimensions. What this means is that baselining and thresholding are now available on user-defined custom dimensions – like location, service name, customer ID, or any other way you’d like to support meaningfully slicing traffic.

A simple use case could be a jump in bits/s for traffic you have classified as “Transit” via custom dimensions. Or a drop in bits/s for traffic you have classified as “Settlement-Free Peering.” Or even major new traffic destinations on a per-application basis.

Alerting JSON webhook triggers

A lot of our anomaly detection users have been asking us to add means to trigger homegrown REST endpoints when alerts are firing, primarily to allow integration to in-house tools and workflow systems.

If you are one of these, your voices have been heard! Whether you want to integrate Kentik’s Anomaly Detection capabilities into your existing monitoring systems or trigger your own form of remediation, this is now possible.

You can now set up a Notification Channel that corresponds to a webhook URL which can be posted to. The Channel will receive all of the relevant JSON data context for you to code against.

Route Traffic Analytics

Route Traffic analysis is the fruit of a hackathon we held earlier this year at Kentik.

You may have heard about studies finding it isn’t uncommon for a given network to have over 95% of its traffic delivered by a minuscule number of routes.

The reason behind these studies is that the FIB capacity of low-end black box L3 switching gear is limited to around 30K prefixes. If you can find a way to live with only 30K routes in FIB and a default route to cover the rest, you don’t need to purchase very expensive routing gear that has a FIB capacity in the millions of routes. The operational question is which 30K routes?

The Route Traffic Analysis feature, under Analysis → Route Traffic, precisely answers this question.

Feature overview

Accessed from the Analytics menu, Route Traffic Analytics feature provides insight into the number and percentage of traffic flows correlated to the number and percentage of routes, plus Mbps per analyzed tranche of routes. The summary view provides both histogram and tabular data views.

Conveniently, the histogram on top of the table will display stops for p95th, p90th, p80th for Traffic and Routes on its X and Y axises.

A listing of the top 1000 routes by traffic density, which provides more details per routes:

Export to CSV of top routes, which could be used to configure routers:

A quick calculation of average and max Mbps per route:

New Packet Size, Interface Capacity Dimensions

Packet Size Dimension

In our constant effort to bring more and more dimensions for our users to slice and dice from, we have just added Packet Size and Packet Size_100 grouping dimensions and filters to our Data Explorer and Dashboards.

The Packet Size_100 dimension segments packet size statistics in buckets of multiples of 100 Bytes, well suited for Comparison Bar Charts.

Interface Capacity Dimension

Interface Capacity has also been added to flow grouping dimensions and filtering in the Data Explorer and Dashboards.

This allows our users to display a graph of all 10Gig links, another of all 20gig links, etc, so customers can eyeball hot links or capacity issues per link type.

This dimension will come in handy when going through a capacity management exercise in your network: it is well paired with a table view, in which you could for instance list your topX 10Gbps interfaces by order of traffic, as displayed in the screenshot below:

With reports using the Interface Capacity dimension, you can now answer questions such as: “How is traffic versus capacity for the 1Gbps, 10Gbps, 20Gbps, 30Gbps, 40Gbps, 100Gbps interfaces on our sites? Are any of them maxed out?”

To illustrate the above, we have created a ‘Capacity Management‘ Preset Dashboard readily usable for this purpose, load it directly from the Dashboards Library section:

SNMP / Interface Overrides

This capability lets users manually set interface level information that is usually polled via SNMP.

Our Knowledge Base entry for Interfaces has been updated with this feature.
The associated API reference is available here in our Knowledge base, and here in our sandbox, within the /device endpoint

The main use cases for this new features are:

Providing query-able interface info on a Router/Switch device when SNMP is not enabled.
Providing query-able interface info on nProbe hosts as SNMP isn’t available for these by default.

The implementation of this feature can be seen in the Device → Interface screen.

Hovering on an interface line will present options to override an interface, as shown below:

Navigating to the Edit button will bring up an in-place edit panel for this interface:

Upon saving, override fields of the interface will be displayed with an orange triangle in the bottom left corner, as in the example here:

…and hovering over the aforementioned orange triangle will display the initial value when there is one:

An additional handy toggle in the interface table’s header allows you to filter it to only view interfaces with an override:

New User Profile Settings

User Profile settings have been updated to allow enabling or disabling of history, default time-zone and DNS lookups. Settings are in the “User Information” table found by clicking on the username at the upper right of the navigation top bar.

Disabling history in the User Information panel sets the Historical Overlay switch (shown below) to off by default in the Data Explorer. This shortens query response time as data points for the selected number of days of history don’t have to be fetched anymore:

Disabling DNS lookups will also reduce query time, as Hostnames for displayed IPs in the Data Explorer query result table won’t have to be fetched before returning the result. Depending on how many IP addresses are being resolved, disabling lookup can greatly speed any graphs or queries returning IP addresses.

Default landing page

A newly added option in User Information is the ability to configure a landing page, which is the page that will show by default upon login.

The landing page can either be a Dashboard, a Saved View, or your the Alert Summary page if you are a user of our anomaly detection feature-set.

Miscellaneous Additions

We now display distinct flow types for NetFlow v9 and IPFIX on the device listing page.
Alerting learning mode default is now +6 days.

6 years ago

Flow Type Auto-Detection

Users no longer have to indicate to Kentik what flow type they are sending (e.g. NetFlow, sFlow, IPFIX) – from now on, Flow Type isn’t specified anymore at device creation time and will be auto-detected by the Kentik Detect Ingest point itself. In the Admin Device List, the “Flow” column now indicates what flow type we are receiving and auto-detecting from each device.

6 years ago

January 2017 Update

Data Explorer improvements

Data Explorer Pivot to Dashboard

Every now and then, the simplest feature unveils a world of possibilities. The new ability to “pivot” a row in the Data Explorer is a great example.

Clicking on the menu at the right of a row in the Data Explorer and selecting “Pivot” opens a (configurable) dashboard showing many different views of the chosen row of data based on different combinations of dimensions and metric.

This pivot feature allows rapid and comprehensive data exploration, reducing the need to manually construct a series of several ad-hoc views in the Data Explorer, for example when trying to identify “why this unexplained bump over this traffic graph occurred.”

For instance, if I am suspicious of traffic sourced in the Netherlands going to a specific IP address, here’s what I would do, taking advantage of the pivot feature:

Below, we see a dashboard that decomposes this NL → dest. IP traffic into multiple different dimensions, without making me go through the trouble of building a unique dashboard.

The pivot feature makes new paths of investigation practical that wouldn’t otherwise have been explored due to the time required to build such a dashboard, and the interruption building a dashboard causes to the investigation workflow.

The pivot feature is discussed in this Knowledge Base entry.

Data Explorer Side-bar Overhaul and Saved Views

As you’ve probably noticed, we revamped the UI of Data Explorer’s Query sidebar to further streamline its appearance.

At the same time, we’ve also added the ability to Create, Edit, and Save Views. Where you previously needed to rebuild your favorite queries in Data Explorer, you can now save them and go back to them to refine them or even share them.

The full documentation on Saved Views is available in our knowledge base under this article.

Saved Views come with an overhauled Data Explorer menu allowing quick access to them.

A new Saved Views Library section has been started, allowing users to share Saved Views within the same company, or even leverage Kentik’s library of pre-existing views.

This marks the initial steps towards a community driven initiative that will be started in the future for Kentik users to share their recipes on Dashboards, Views, Alerting policies.

Directly from the Data Explorer, look for the Save and **Load **controls at the top. With these, no more starting all over from scratch when improving on your (or your co-users’) existing visualizations. Conveniently load them and save them anytime.

Here’s a quick display of what the new Saved Views Library looks like:

Stay tuned and watch this community concept trickle down into further areas of the Kentik Detect Portal in the future.

Further IPv6 support in Data Explorer

Kentik has fully supported storage and querying of IPv6 for some time, and we are steadily adding support for IPv6 in any place where addresses or prefixes are used.

IPv6 Next-Hop flow dimension

Next-hop IP dimension in explorer and dashboards now supports IPv6 on top of the existing IPv4, as displayed in the Data Explorer Dimension selector below. Note that different CIDR thresholds can be set independently for IPv4 and IPv6.

IPv6 Source/Dest prefixes dimension

Metrics support for IPv6 added to explorer and dashboards: Unique src/dst prefix, Unique SRC/DST ASN, and Unique src/dst IP now support ipv6.

Alerting feature update

Alerting is now fully documented in our Knowledge Base; feel free to swing by and get a more detailed view of what it offers!

Additionally, Alerting now supports Route Prefix and Length (Prefix/LEN) both as a Dimension and in Filters.

API v5 updates

APIv5 documentation has been entirely updated, and is now available to our users at the following locations:

v5 API for administration of Kentik Detect Objects	here
v5 Query API to pull data from Kentik Detect Engine	here
v5 API sandbox / tester	here

Additionally, an API functionality to return a URL to open an API call in browser (authenticated) has also been added.

Important note: The current plan is to shut down former API versions (namely v1 and v4) on May 5th.

Miscellaneous

ICMP code and type for v9/IPFIX is now supported. It is overloaded into the IP DST PORT values based on NetFlow v5 ICMP encoding.

6 years ago

December 2016 Update

A10 Integration with Anomaly Detection

On top of the already offered RTBH mitigation method, our Anomaly Detection system now supports integration with A10’s Thunder TPS Series mitigation hardware.

What this means is that if you already own or plan on acquiring such appliances, you can leverage all of Kentik Detect’s powerful Anomaly Detection system and couple it with A10 for mitigation.

To configure the Kentik end of an A10 TPS mitigation platform for use within policies, go to the Mitigation menu under the Alerts and click on the +Create Mitigation Platform, as shown in the screenshots below:

Matrix visualization

Matrix visualizations are described at length in this Knowledge Base article.

A “Matrix view” is now available in the Data Explorer, find it amongst the already existing Display Types section of the query side panel.

Here are a few examples of uses cases for Matrix Views:

Transit providers might want to look at Top 10 Source ASNs vs Destination ASNs matrix of traffic. This might be a good way of trying to identify strategic content or eyeball prospects to engage in the future.
Building a matrix of cross-PoP traffic for capacity planning purposes.
Looking at PPS between different farms of servers or even between top talkers in a Datacenter setup…

Alerting v3 Updates

Minimum look-back for baselining

More details on Look-back Alert Policy settings in this Knowledge Base entry.

You can now use the Minimum Look-back setting to specify the minimum number of hours or days that baseline data collection is performed before a baseline is made available for comparison by alerting policies.

In-policy creation of notification channels

More details on Alert Notification Channels in this Knowledge Base entry.

You now create a new notification channel from directly within the threshold notification-add function.

Dashboard editing overhaul

Knowledge Base entries detailing Dashboard usage and creation are located here.

The dashboard layout infrastructure has been redesigned to improve speed and ease of use. This comes with a streamlined user experience as part of our constant effort to streamline usability of our most used features.

Subscriptions, aka Scheduled Exports

Exports and scheduled reports have been redesigned for ease of use. Here’s an example of the overhauled Email Subscription experience:

And the Export feature in Dashboards and Views updated experience:

BGP Status within device screen

You can now tell the state of your BGP sessions in the Device List table in the Devices page, which is found under the Admin menu.

6 years ago

November 2016 Update

Tags feature update

Tagging now supports regex for device names and interface fields, and supports IPv6.
As a reminder, a comprehensive table references all types of inputs for all of the available Tag Fields, it is located here.

For instance, if your interfaces always include consistent descriptions, you could potentially match said interface descriptions on either ‘PNI’ or ‘Peer’ or ‘customer’ and tag all the matches as ‘Peering’ to then be able to filter them in or out of any Data Explorer query.

Prettified JSON output to describe API calls

You can now see the API calls in Data Explorer as prettified JSON, making it much easier for your users to identify the fields at play in your API calls.

The idea here is to further simplify the task of integrating with the Kentik API under the following methodology:

Building a satisfactory View, tweaking it until it shows exactly what you are after
Exploring the resulting JSON
Building an integration

To describe the underlying API call of a given view, proceed as illustrated below – starts with clicking the Hamburger Menu icon on the top-right side of a view

Peering Analytics IPv6

Peering analytics now supports IPv6 as well as showing the full path on mouseover.

6 years ago

September/October 2016 Update

In graph zooming

Data Explorer now supports graph highlighting and click and drag operation on the timeline to zoom in to the selected timeframe. Following on to a horizontal click and drag, the side query panel will automatically update its “Time” fields and a zoomed-in graph will be spawned.

New ‘Table’ visualization type

A table view has been added to the existing display types on top of the existing chart types.

Beyond their basic appearance, Table views are highly customizable in terms of columns they display and can amongst others allow to build computed fields in a very comprehensive manner. These advanced options for the table widget are available by clicking on the hamburger menu at the top right end of the table component.

Table View Options details can be found in our Knowledge Base under this article.

Custom Dimensions

Custom Dimensions correspond to user defined Flow Enrichment Custom Columns. What that means in practice is that users are now able to programmatically enrich their flow data with columns (5 custom dimensions are allowed per account) that can be grouped-by, summed, max’ed, etc.

Unlike Tags or Saved Filters, Custom Dimensions only affect the flows being considered in a request. Custom Dimensions provide an efficient way of breaking down your visualizations by your own business contextualized data groupings.

Values for each custom dimension can be set via:

Kentik Detect portal UI
or programmatically via our API (link to Custom Dimensions in our API Sandbox here)

Interesting examples of how one would use Custom Dimensions include:

Marking different types of customers: populating a_ customer_type _Custom Dimension based on the IP Ranges within which the customers are hosted.
Marking arbitrary tiers of cheap to expensive destinations or sources by relying on source or destination ASNs.
Marking Peering (paid, free), Transit, and IX traffic based on matching interface description.

The extensive documentation around Custom Dimensions can be found here in our KB

Saved Filters

The extensive documentation around Saved Filters is located here in our KB

Saved Filters are a new addition to Kentik Detect’s take on how to slice and dice data in the Data Explorer in an increasingly quick and convenient manner.

Remember the days where you needed to build a complex filter from scratch when going back to the Data Explorer screen? Those days are over. With Saved Filters, you can conveniently save filters you use on a regular basis and call them from the Data Explorer Filter section any time.

Here’s what filters look like now: If you were to build that filter of destination French ISPs every time you create a query, it could be quite a chore… here’s what your filter would look like:

Saved Filters to the rescue. Now you can now save that filter by clicking on the disk icon at the top of the filter group, and re-use it sometime later.

Voila! You can now invoke this filter any time directly from the filter screen:

Double-click the filter’s name and you can invoke its opposite, in this case filter on any destination ASN but the French ones you previously listed!

Additionally, Saved Filters are shareable between users at a company level, as well as Kentik offers common pre-set filters. To get a view of all filters, just hop on to Admin → Saved Filters:

In a future release, Saved Filters will adopt a library view to facilitate collaboration and sharing, so stay tuned!

6 years ago

August 2016 Update

IPv4 and IPv6 CIDR grouping/breakdown

When selecting dimensions in Data Explorer, users now can configure separate aggregation/grouping levels for IPv4 or IPv6, from a single location in the Query Side Panel:

Kentik support for nProbe

Detailed steps to get setup with nProbe and Kentik are detailed over in this Knowledge Base section.

nProbe now allows you to export flow data to Kentik Detect’s Flow Data platform, unveiling a whole new array of host-level traffic and performance info.While previously limited to flow-data from your networking gear, Kentik now brings server/data-center level metrics to the powerful performance analysis tools it already offers (Custom Dimensions, Tagging, Filtering…).

With nProbe able to send flow data to the Kentik Detect big data platform, the realm of query-able metrics now extends to network performance for such devices, including:

Retransmits/s, %Retransmits,
Out of Order/s, %Out of Order,
Fragments/s, %Fragments,
RTT/2 Client latency
RTT/2 Server latency
RTT/2 Application latency

This first set of performance metrics paves the way for the future addition of application-specific Dimensions to enrich flows exported from servers.
The screenshot below shows how registration of nProbe hosts happens in the Kentik Detect Portal:

Multiple time series

The multiple time series view options will soon enough be detailed in this section of our knowledge base.

Currently in its Beta stage, this feature now allows you to combine multiple graphs in the Data Explorer (and dashboards) into one single, comprehensively configurable representation. This can prove handy when trying to establish causality between different observed phenomena.

Below is a simplistic view of Destination and Source traffic broken down by ASNs: