Early this year we launched Kentik Market Intelligence to spell out the internet interconnection ecosystem for you. It crunches large amounts of public BGP routing data, and scores and ranks all networks in any market based on the size of their customer base.

With this new release, we've added a significant feature set: Kentik Market Intelligence Insights. 

KMI Insights? Tell me more...

Insights are the relevant news items around the networks and markets you care about. When crunching the billions of BGP dump entries multiple times a day, our platform now identifies interesting insights that get contextualized in the KMI UI, coloring them with the networks involved and the market they are observed in.

Sample insights can be: <This provider> added <this transit customer> within <this geography> for instance, or any network adding new providers, as well as rank changes and changes in routing announcements between two networks.

Each insight comes with a certain amount of mandatory attributes: 

• A Customer Network: as identified in the Provider/Customer ASN relation we crunch with every run
• A Provider Network: as identified in the Provider/Customer ASN relation we crunch with every run
• A Market: the market this insight applies to
• Insights type:  as explained before, our engine classifies insights based on their nature (see list in the screenshot above)
• Magnitude: an arbitrary 1-5 value that helps order these insights from most important to least important, which is most often correlated with the change in prefix score, either plus or minus, between two networks

All these attributes have two purposes: helping the Kentik Market Intelligence UI display them in context (see further down this announcement), and allowing users to slide and dice these insights and tailor them to any subset they'd like to keep track of.

Contextualized insights

A new "Top insights" panel now appears on the right side of the landing page, it can be collapsed/expanded using the top right [insights] toggle.

The landing page will display unfiltered insights, ordered by magnitude, and this list of insights will by default follow the market filter used on the landing ranking screen.

This side panel can be configured to include or exclude specific types of insights, or to extend the period covered by the insights.

When navigating to any Network Details screen, the insights right-side panel will be displayed on the "Overview" tab for this network, and filtered to focus on the investigated network.

Track Networks and Markets that matter to you !

A while back, we introduced Observation Deck, which is a place for every user to compose their landing page with the areas of network visibility that specifically matter to them, based on widgets from specific workflows and areas of the product.

Kentik Market Intelligence now has its first widget and it's powerful!

In the future, we will add more KMI-related widgets to the Observation Deck so that you can embed widgets showing rankings and visualizations from KMI.

We hope you'll enjoy this update as much as we do. Please send us your feedback on it, we'd love to hear what you think!

Kentik added support for 6PE technology and BGP IPv6 Labeled-Unicast address family to be used for Flow enrichment.

The IPv6 Provider Edge (6PE) is the technology that enables isolated IPv6 networks to communicate using MPLS LSPs over an IPv4 MPLS backbone network.

The diagram below demonstrates the typical scenario for the use of the 6PE technology. CE routers which are at the border of the IPv6 islands, advertise their IPv6 routes to the 6PE routers of the MPLS network. These PE routers are the only dual-stack routers, which support both IPv4 and IPv6. The 6PE router advertise the received IPv6 routes to other 6PE routers using MP-BGP session over IPv6 Labeled-unicast address family. These route have:

• Original IPv6 route received from CE
• Inner MPLS label value, which would be used in the 6PE router’s data-plane to encapsulate packets toward IPv6 island’s networks.
• Next-hop with the IPv6-mapped IPv4 address which is in the form of “::FFFF:<IPv4-address>”.
  • The mapped IPv4 address is the address of the advertising 6PE router.
  • It determines the Outer MPLS label which will encapsulate IPv6 packets inside the MPLS network

To perform enrichment based on the 6PE information, Kentik’s user should establish the BGP session with the IPv6 Labeled-unicast AF between their 6PE router and Kentik. Based on the received IPv6 LU routes the Kentik’s ingest layer would be able to enrich the Flow’s received from those 6PE routers. All “standard BGP” dimensions would be populated, but more specifically:

• The “Next-hop IP” dimension will be populated with the next-hop IPv4-mapped IPv6 address from the received route
• Based on the IPv4-mapped address, the Kentik ingest would identify the next-hop 6PE router and populate “Ultimate-Exit Device” dimension and based on that, the other “Ultimate-Exit” dimensions.

Additionally, to instruct the use of Labeled-unicast routes, a user needs to select additional configuration for 6PE routers in the Kentik.  At the Settings → Devices page, select the 6PE router device and “Edit device”, then at the BGP Tab, at the “BGP route selection” drop-down menu select the option  “VPN table, fallback to Labeled-Unicast table, fallback to Unicast table”.

The example of the Data Explorer output with the 6PE BGP next-hop dimensions is shown below:

Kentik has added a new configuration option, which determines how the BGP routes are selected for flow enrichment process. To make the whole process clear enough we should start with the basics.

BGP sessions

BGP session between customer’s router and Kentik can be established over:

• IPv4
• IPv6

Since these are “Multiprotocol BGP” sessions, for each of the sessions, it is possible to enable multiple Address Families, for example: Unicast, Multicast, Labeled-unicast, L3VPN, Flowspec, etc.

These Address Families are defined with AFI (Address Family Identifiers) and SAFI (Subsequent Address Family Identifiers) attributes. They are regulated by IANA and the exact values can be found on the following links:

• IANA AFI numbers: https://www.iana.org/assignments/address-family-numbers/address-family-numbers.xhtml
• IANA SAFI numbers: https://www.iana.org/assignments/safi-namespace/safi-namespace.xhtml

The Kentik side of the BGP peering with customer’s devices will be enabled with the Unicast, Labeled-Unicast and L3VPN families by default. For the BGP “IPvX” session from the Kentik side will have the following AFs enabled:

• “IPvX” unicast
• IPv4 and IPv6 labeled-unicast
• “IPvX” L3VPN - (IPv6 L3VPN address family is not used)

Received routes from each of these address families are stored in the separate route table, which is check during the Flow enrichment process.

NOTE: IPv6 VPN routes are received, but not used for the enrichment

The Flowspec address family will be enabled only if the customer explicitly enable it in the device configuration on the Kentik portal.

BGP attributes enrichment process

Assignment of the “Route Prefix/LEN” dimension

The assignment of the Src and Dst Route Prefix is the following:

• Src and Dst Route Prefix dimensions are first populated from the Flow information using Src and Dst Mask field from Flows - if applicable.
• Src and Dst Route Prefix will be overwritten further in the ingest processing if there is a matched BGP route.
• The way to know if the Src or Dst Route Prefix is coming from flow or BGP is by observing other BGP route attributes:
  • if the Route prefix originates from the flow information the dimension “Next-hop AS Number” will be “0 - -Reserved AS-,ZZ” and the dimension “AS Path” will be empty.
  • if the Route prefix is overwritten by the BGP information, the BGP related dimensions such as “Next-hop AS Number” and “AS path” will be populated

VRF metadata collection

As the part of the SNMP interface discovery process, Kentik SaaS or Kentik kproxy will perform the VRF discovery and interface association. This information about the VRFs is collected over SNMP using MPLS-L3VPN-STD-MIB, if the device supports it. The devices from Cisco and Juniper Networks support this MIB. We have also developed support for for Nokia’s proprietary MIBs.

For each VRF, Kentik collects:

• Name
• Description
• Route Distinguisher (RD)
• Route Target (RT)
• Interface association

BGP route matching process

The enrichment of the BGP/Route related Flow dimensions is performed as a result of matching the Flow’s IP address against the BGP route received from customer’s device over BGP sessions. The default behavior of the matching process is the following:

• Flow’s Src interface is checked if it is assigned to the VRF.
  • If the source interface is in the VRF, flow’s Dst IP address is looked-up against the BGP VPNv4 routes with the RD associated with the source interface’s VRF:
    • If there is a route match, the route will be assigned to the flow
    • If there is no match, or there is no BGP VPNv4 table at all, or even no L3VPN AF established as part of the BGP peering, the match will not be found and BGP route dimensions are not populated.
  • If the source interface is not in the VRF, flow’s Dst IP address is looked-up against the “global” BGP table containing Unicast IPv4/IPv6 AF routes.
• The same process is performed for flow’s source IP address route lookup, based on the destination interface association with the VRF.

BGP route selection configuration

To address some additional scenario’s that we have seen in the customer’s network, Kentik added the configurable option to influence the BGP route selection process related to which BGP routes will be used for matching process.

This configuration is available at the Settings → Devices → Edit Device dialog → BGP Tab.

At the dialog, there is a new drop down menu called “BGP Route Selection” with the following three options:

• VPN table for VRF interface, Unicast table for non-VRF interface (default option)
• VPN table, fallback to Unicast table
• VPN table, fallback to Labeled-Unicast table, fallback to Unicast table

The following table describes the behavior of each configuration option:

Kentik now supports SNMP collection of the CPU and Memory utilization for Huawei devices.

The Kentik’s ingest/kproxy will determine that the discovered device is Huawei device by searching for the “huawei” string in the well-known SNMP sysDescr OID.

The Kproxy would monitor the following number of device’s “Components”, with the use of the SNMP OIDs described below:

• Iterate over ENTITY-MIB to get the descriptions of all the Components and their indexes using OID .1.3.6.1.2.1.47.1.1.1.1.2
• Iterate over HUAWEI-ENTITY-EXTENT-MIBto get CPU and memory utilization for each component:
  • CPU utilization in percent, using OID: .1.3.6.1.4.1.2011.5.25.31.1.1.1.1.5
  • Memory utilization in percent, using OID: .1.3.6.1.4.1.2011.5.25.31.1.1.1.1.7
  • Uptime of the component, using OID: .1.3.6.1.4.1.2011.5.25.31.1.1.1.1.10

Kentik will ignore any components which have both CPU and Memory utilization equal to 0.

The support is available in kproxy starting from version v7.36.0. The example of the Data Explorer query is shown below:

Kentik now supports SNMP collection of the CPU and Memory utilization for F5 BIG-IP devices. Some of the F5 BIG-IP devices do not support flow export features, which means that the collection of the SNMP metrics would require use of Kentik's kproxy with the “bootstrap_devices” option.

The general behavior of the Kentik’s SaaS ingest or Kentik’s Kproxy is that the SNMP metrics collection would only start once the flows are received from the certain device. This behavior can be limiting in some cases and can be changed by the use of the Kproxy’s Bootstrap devices feature. This feature is enable with the use of the-bootstrap_devices command line argument. This argument should provide the comma-separated list of devices’ IDs. For those devices, the Kproxy will start the SNMP metrics collection immediately, without waiting to receive devices’ flows. More information about the Kproxy CLI arguments can be found in our Knowledge Base article: https://kb.kentik.com/v0/Bd04.htm#Bd04-kproxy_CLI_Reference

The Kentik’s Kproxy will determine that the discovered device is F5 BIG-IP by looking for the “big-ip” string in the well-known SNMP sysDescr OID.

The Kproxy would monitor the following 4 Device “Components”, with the use of the SNMP OIDs described below:

• Name “Global”:
  • MemoryTotal [bytes] - OID Name: sysGlobalHostMemTotal, OID: .1.3.6.1.4.1.3375.2.1.1.2.20.2.0
  • MemoryUsed [bytes] - OID Name: sysGlobalHostMemUsed, OID: .1.3.6.1.4.1.3375.2.1.1.2.20.3.0
  • CPU [percentage] - OID Name: sysGlobalHostCpuUsageRatio5m, OID: .1.3.6.1.4.1.3375.2.1.1.2.20.37.0
• Name “TMM”:
  • MemoryTotal [bytes]- OID Name: sysStatMemoryTotal, OID: .1.3.6.1.4.1.3375.2.1.1.2.1.44.0
  • MemoryUsed [bytes] - OID Name: sysStatMemoryUsed, OID: .1.3.6.1.4.1.3375.2.1.1.2.1.45.0
  • CPU - value will be 0
• Name “Other”:
  • MemoryTotal [bytes] - OID Name: sysGlobalHostOtherMemoryTotal, OID: .1.3.6.1.4.1.3375.2.1.1.2.20.44.0
  • MemoryUsed [bytes] - OID Name: sysGlobalHostOtherMemoryUsed, OID: .1.3.6.1.4.1.3375.2.1.1.2.20.45.0
  • CPU - value will be 0
• Name “Swap”:
  • MemoryTotal [bytes]- OID Name: sysGlobalHostSwapTotal, OID: .1.3.6.1.4.1.3375.2.1.1.2.20.46.0
  • MemoryUsed [bytes] - OID Name: sysGlobalHostSwapUsed, OID: .1.3.6.1.4.1.3375.2.1.1.2.20.47.0
  • CPU - value will be 0

For each component  MemoryFree and MemoryUtilization are calculated from collected MemoryTotal and MemoryUsed metrics. Each component use standard Uptime which is collected at the device level from sysUpTime OID.

The support is available in kproxy starting from version v7.36.0. The example of the Data Explorer query is shown below:

The important characteristics of the Kentik Platform are rich API capabilities and the supporting SDKs. For the last two years Kentik has supported the development of the community Python SDK which is based on the Kentik’s APIs. This SDK enables our customers to use Kentik’s Platform APIs natively in the Python programming language, with Python Objects and Methods instead of dealing with the details of the API syntax.

The Community Python SDK is available on GitHub: https://github.com/kentik/community_sdk_python.

Just over a month ago, we released a new version 1.0.0. Until this version, the community Python SDK supported objects and methods that are exposed within Kentik’s REST API v5. With this new release, the support has been extended to some of the endpoints of our new gRPC-based Kentik API v6, specifically supporting Synthetics monitoring and Cloud Export configuration.

Important note on breaking changes

As it is already mentioned, Kentik’s API v6 is natively a gRPC API, but it also supports the REST access. The community Python SDK is using the Kentik API v6 directly over gRPC. To accommodate communication with the Kentik backend using both REST-based Kentik API v5 and gRPC-based Kentik API v6, the necessary change has been introduced that would require a change of your existing Python scripts and programs.

In most of the cases, you would initialize the KentikAPI object with the constrictor that is using the api_url argument, for example:

from kentik_api import KentikAPI

client = KentikAPI(api_url=KentikAPI.API_URL_US, auth_email=email, auth_token=token)

The api_url argument would expect the URL to the Kentik’s API endpoint, which would be in the form: https://api.kentik.com or https://api.kentik.eu. However, the endpoint that is used for the Kentik’s API v6 is in the form of the host, for example: grpc.api.kentik.com.

For this reason and to be able to configure API access information with the single parameter, it was decided that api_url argument of the KentikAPI constructor should be replaced with api_host argument. The argument is expected to contain only the fully qualified hostname of the server hosting the target Kentik API instance (the default value is KentikAPI.API_HOST_US which is equal to api.kentik.com). Consequently, the Class variable KentikAPI.API_URL_US has been replaced with KentikAPI.API_HOST_US and KentikAPI.API_URL_EU with KentikAPI.API_HOST_EU

To summarize, if you upgrade the version of your Python SDK to 1.0.0 or later, you will need for adjust the initialization of the KentikAPI to use the changed argument, for example:

from kentik_api import KentikAPI

client = KentikAPI(api_host=KentikAPI.API_HOST_US, auth_email=email, auth_token=token)

Installation

You can easily install the latest version of the Python SDK using pip, for example:

$ python3 -m pip install kentik-api

Let us know what you think about our Python SDK and feel free to submit any contributions or issues over GitHub. Happy coding!

The list of supported devices from which Kentik can collect CPU and Memory utilization is growing. Kentik now supports SNMP collection of the CPU and Memory utilization for Palo Alto Network devices. 

Palo Alto devices are using the standard HOST-RESOURCES-MIB, for CPU and Memory usage. The Kentik’s ingest/kproxy will determine that the discovered device is from Palo Alto by looking for the “palo alto” string in the well-known SNMP sysDescr OID.

For CPU:

• Component name is provided in the OID: hrDeviceDescr: .1.3.6.1.2.1.25.3.2.1.3.index
• CPU utilization is provided in the OID: hrProcessorLoad: .1.3.6.1.2.1.25.3.3.1.2.index

For Memory:

• Component name is provided in the OID: hrDeviceDescr: .1.3.6.1.2.1.25.3.2.1.3.index
• Memory utilization is provided from the SNMP table: hrStorageTable: .1.3.6.1.2.1.25.2.3

The support is available in kproxy starting from version v7.36.0. The example of the Data Explorer query is shown below:

Kentik now supports collection of the NetFlow and IPFIX fields for source/destination VLAN, which we previously not collected from the received flows. 

The related VLAN fields are shown in tables below:

NetFlow v9 VLAN fields

Resource: https://www.cisco.com/en/US/technologies/tk648/tk362/technologies_white_paper09186a00800a3db9.html

IPFIX VLAN fields

Resource: https://www.iana.org/assignments/ipfix/ipfix.xhtml

These two fields are collected from NetFlow/IPFIX protocols and stored in the Kentik’s Source VLAN and Destination VLAN dimensions.

The support is available in kproxy starting from version v7.36.0. The example of the Data Explorer query is shown below:

We continue developing new features for the connectivity checker in order to ensure better user experience for our customers

• It’s now possible to run an ad-hoc test without creating a report first. This test can be saved as a report for later use.
• New source and destination types. You can run a test between subnets, network interfaces and instances. All of the source and destination types are searchable using their names.

• You can start a connectivity test directly from the object on the topology, with the source being automatically pre-filled.

• Direct link to the AWS console is added on under details of impacted objects for easier failed connectivity test troubleshooting.

Data Explorer allows you to drill into your traffic flows for better understanding of what is going on in your network. Data Explorer uses different dimensions and filtering to show you the flows of interest. 

ENI Entity name and ENI Entity type were added to the list of supported dimensions on AWS. This allows users to see and filter traffic to and from AWS services such as Load Balancers and VPC endpoints.