This month the cloud product team delivered a ton of improvements: tightening up our maps, making our backend systems more resilient, and improving Azure onboarding. We released an alpha version of Kentik Kube, a product designed to help networkers awash in a sea of containers find solid ground, a variety of map improvements, support for VPC endpoint interfaces and gateways, and the ability to retain VPC Flow Logs inside of S3 buckets indefinitely. And we delivered the first of two projects designed to make agentless cloud ingest a reality. We are actively seeking design partners for Kentik Kube. Please contact us if you’d like to help guide our vision for this product early on. We welcome your input! Read on for all the details.

Kentik Kube: Network visibility for Kubernetes clusters

One thing we believe at Kentik is that everything that runs over the network should be observable — from containers to clouds, and everything in between. And with that in mind, whenever we become aware of a new visibility gap making operators’ lives more difficult, we aim to bridge it.

Enter Kubernetes. In 2020, we released an agent that could be used to export network flows from Kubernetes. And this month, we’ve released Kentik Kube to visualize this data.

Kentik Kube is a new module of the Kentik Network Observability Cloud that helps cloud and infrastructure engineers gain detailed network traffic and performance visibility both inside and among their Kubernetes clusters to quickly detect and solve network problems.

As the industry has adopted Kubernetes as a de facto standard for workload scheduling, teams that support these environments are discovering that their network and Kubernetes monitoring tools can’t help them answer critical questions because they:

• Are unaware of traffic patterns within Kubernetes or do not create traffic logs for these environments
• Do not take into account the network models implemented by Kubernetes
• Do not fuse Kubernetes metadata into traffic data.

We built Kentik Kube to provide this visibility for these teams. Our solution supports cloud-managed Kubernetes clusters (AKS, EKS, and GKS) and on-prem, self-managed clusters using the most widely used cloud implemented network models.

Kentik Kube helps by supporting the following use cases:

Network performance: Discover which services and pods are experiencing network delays so you can troubleshoot and fix problems faster. Identify service misconfigurations without capturing packets. Configure alert policies to proactively find high latency impacting nodes, pods, workloads or services.

Top talkers: Identify clients/requesters consuming your Kubernetes services so you can track down problematic connections. Discover oversubscribed microservices so you can adjust scaling, configure node affinity, etc. Know exactly who was talking to which pod, and when.

Policy validation: Ensure that your network reality matches your design. See which pods, namespaces and services are speaking with each other to ensure that your configured policy is working as expected.

Total infrastructure visualization: Know which pods are deployed on which nodes — even historically. See which pods and services are communicating with non-Kubernetes infrastructure or the internet. View your network from container to cloud.

AWS Flow Logs in-bucket file retention

Kentik has always allowed our AWS customers to retain their flow logs in their buckets. However, we ran into problems with customers who wanted to keep their flow logs around for long periods of time. We’ve made some significant changes and are happy to report that customers can now keep their flows around as long as they like — without any adverse effects on their Kentik subscriptions.

VPC endpoint support and ENI gateways

This month we’ve begun our journey towards full support of VPC endpoints in Kentik Cloud. Our first volley of product improvements includes giving customers the ability to group and filter flows based on these gateway IDs in the Data Explorer.

We support both Gateway-style VPC Endpoints as well as Interface VPC endpoints (AWS PrivateLink). However, despite their similar names, these are very different things under the hood. VPC Endpoint Gateways act as gateways to Amazon services, keeping this traffic inside your VPC and off the public internet. Using these endpoints can save money by keeping VPC network egress costs down — making identifying when traffic isn’t using these gateways a key use case. These gateways aren’t exposed as elastic network interfaces and thus can be difficult to track traffic using other network solutions. Kentik is the only solution on the market that gives you this capability.

AWS PrivateLink allows AWS and their customers to configure their services for consumption without forcing traffic to flow over the public internet. Over 128 different AWS services are available behind these so-called “interface endpoints” while just a small few are available behind VPC endpoint gateways. Interface Endpoints serve to reduce cost and can serve a security function in that sensitive information can be shared between AWS customers privately. These endpoints are exposed in a more standard fashion inside of VPCs using Elastic Network Interfaces as the gateways. The major difference in how this impacts Kentik users is that these interfaces are grouped in the Interface Type dimensions and specific interfaces are identified in the ENI ID dimension while the Gateway-style endpoints are grouped into the Gateway Type dimensions and the specifics are found under Gateway ID dimension.

S3 bucket ingest

We took a major step forward this month in our promise to offer agentless ingest to AWS customers. Instead of having Kentik reach into customer buckets to retrieve flow logs, we can now support customers who wish to write their flow logs directly into an S3 bucket that Kentik manages and maintains. This allows customers to write flow logs more flexibly (via direct VPC write, lambda copies, S3 actions, etc.) and also allows customers to avoid giving Kentik permissions into their S3 buckets. Next up on our agentless agenda is to provide a method for companies to post their metadata to a similar endpoint for enrichment and mapping visualizations. Stay tuned!

Map improvements

Following our September release of the Weather Map, we’ve continued to iterate on our Kentik Maps product. This month we’ve added a bunch of important fixes and improvements.

We’ve improved handling overlapping lines in AWS. Previous versions of Kentik Map would allow lines connecting close together objects to overlap. We’ve added some logic to detect this condition and give these objects a bit of breathing room, making it easier to visually determine the path connecting the objects and select them with your mouse.

We’ve improved visualizing site-to-site VPN connections. Prior versions of Kentik Map for AWS kept the customer gateways in the middle of the screen. We moved customer gateways into the “On Prem” box where they rightfully belonged. This helps clean up the interconnection section of the map considerably. We have more work coming here this quarter to make the map experience even easier to use for companies with lots of interconnections.

RPKI validation in BGP Monitor

We’ve added optional RPKI validation to the BGP Monitor test. When enabled (also the default) we will query the RPKI database and report the RPKI status in the result set.

The Actions menu in both Dashboards and Saved View now contains all the actions that Data Explorer would offer in this menu. This has been a long time requirement from our customers and we are happy to get this delivered.

Additional features include:

• The ability to copy a saved view into a dashboard
• The ability to generate a report subscription directly from Dashboards and Saved Views
• Show API call on saved views
• And many others

The Edge > Connectivity Costs workflow supports history.

While only details for the ongoing month were previously visible in the Connectivity Cost UI, the new release will allow users to navigate cost details month-by-month.

Pictures being worth a thousand words, see for yourself in the screenshots below ! Also you can take a look at our KB article on the Connectivity Costs workflow.

In the back scenes, Kentik's OTT Service Tracking workflow leverages an engine, aptly named True Origin. This engine correlates DNS Query/Response data with network telemetry to provide a scalable, affordable DPI-lite type of service.
You can learn more about it here.

While this is something that we rarely spend time talking about, our OTT detection engine, a broadband provider’s favorite, keeps on learning and evolving as time goes by, and the amount of DNS hostname patterns that it is able to detect the underlying service for keeps on increasing constantly.

Over the month of October 2021, the True Origin engine has learned to classify an additional ~1,000 new hostname patterns to match flow with DNS queries. This month, its improvements were centered around Vietnam, Denmark, Sweden and Germany.

As one of our customers’ go-to modules in Kentik Portal, the Library was due for a facelift. Some of Kentik’s customers have as many as 500 dashboards and saved views. We decided to make improvements to handle large numbers of dashboards and saved views.

See the product update that details the highlights of this new version.

Some of these include:

• User manageable Categories to classify content
• Drag and drop Categories
• Guided Mode dashboard inputs straight from the Library
• Favorites and Recently Used are back!
• Multiple contextual actions to do more directly from the Library
• Ability to create report subscriptions directly from the Library context menu
• Keyboard shortcuts

We know that many of our users are fond of Kentik’s dashboarding capabilities. A quick survey of our user population shows that some users save as many as 500 dashboards or views. Taking this to heart, we’ve entirely redesigned and optimized the Library module. 

Read on for a summary of the great new things we’ve done.

Shortcuts, shortcuts, shortcuts

First, we wanted to make sure the Library is easier to access. For that, we have added keyboard shortcuts to quickly summon the Library from anywhere in the portal.

Try SHIFT+? to see a list of available shortcuts. There you’ll see you can bring up the Library by simply hitting SHIFT+L.

User set categories

A top request from users is to be able to create and manage categories in the Library. Now you can create a category on the spot! To do this, click on the top right Add Category icon on the Categorized Views screen.

You can also now rename categories using the triple-dot icon to the right of categories:

Content can be moved between categories by using drag-and-drop. You can auto-open categories using drag-hover, scrolling up and down when dragging a category above and below the fold.

Content that hasn’t been placed in a category will sit in the Uncategorized Views section of the main Library screen, waiting for you to find a new category home for them.

If shortcuts are your thing, we even added a very useful shortcut. When SHIFT-clicking on a category’s expand or collapse arrow, you will now be able to expand or collapse all categories in the Library screen.

Be more productive

Further study of some of our sample users’ portal behavior shows that workflows often gravitate around frequently eyeballing a discrete set of dashboards and saved views, so we’ve made improvements to help.

Favorites and recently viewed panes

The right side of the Library screen is now dedicated to two always-on Favorites and Recently Viewed panes.

You can favorite (or un-favorite via the star icon toggles) any dashboard or saved view from anywhere in Kentik Portal, whether from the Library or in the central content section. You can also favorite in the always-on panels and directly from dashboards and saved views.

Beyond that, you can also favorite Kentik Presets content from the second tab of the Library screen!

As you visualize more dashboards and saved views, your recently viewed items will stack up in the namesake Library panel.

Quicker access to Guided Mode dashboards

Guided Mode dashboards are a long-time favorite, and you can now directly input the pivot value from the Library screen.

The Guided Mode input will also be available directly from within both the Favorites and Recently Viewed panels.

Contextual actions galore

A handful of actions can now be taken directly from the Library entry of dashboards and saved views. These will reduce the number of clicks needed to visualize, and allow you to immediately select the necessary action:

If you are a My Kentik portal tenant, you can now directly export and add saved views to dashboards, clone, create subscriptions and even view directly. You too are one click away from the Library screen!

Custom HTTP Headers for v2 Webhook Notification

The custom webhook notification method in v2 notifications (/v4/settings/notifications) now supports the ability to customize the HTTP headers and values sent with the request, in addition to the request body.

Among other uses, this allows users to provide authorization credentials for API endpoints that require it.

Additional v2 Notification Methods

Notifications v2 now supports all of the notification methods that were supported in notifications v1, along with a few new ones like Microsoft Teams, VictorOps and xmatters.

In addition, with customizable HTTP headers and request body templates in the Custom Webhook method, it should now be possible to do one-off integrations with virtually any third party API.

NOTE: Some notification methods are not yet available to select as destinations for Synthetics notifications. Template updates are required for these methods to properly present the different data fields associated with Syn notifications.

v2 Notification Support for v4 DDoS Policies

v4 DDoS policies now support the selection of both v1 and v2 notification methods as destinations for alert notifications. In the thresholds section of the policy configuration, users will now see both v1 and v2 methods shown in the drop-down list.

Each available notification channel is labeled with the notification method type, though we do not distinguish between v1 and v2 types since these are not user-facing designations. We’ve also temporarily removed the link to the v1 notifications configuration page until we have migrated all v1 methods to v2.

Native v4 UI Forms for Mitigation Configuration

Mitigation platforms and methods are now configured via a native v4 UI form. The new form combines platform and method configuration onto a single page with a better UX that shows which methods are associated with each platform.

The new form also removes the limitation on configuring both RTBH and flowspec mitigation methods on the same router.

Ratio-based Thresholds

We’ve added an additional threshold type for DDoS policies, which allows the user to compare two different metrics that are measured by the policy. Along with this, we’ve added some additional metrics that measure separate inbound and outbound packets/sec and bits/sec rates. The metrics that are compared in a ratio-based threshold must be metrics that are configured as primary or secondary metrics for the policy.

Some use cases where ratio-based thresholds can be useful:

• DDoS policies: Comparing bits/sec in to bits/sec out can make it very easy to detect attacks for content / server destinations, since these resources almost always have a much greater traffic volume out than in. If this ratio reverses, it can be indicative of an attack and ratio-based detection doesn’t require knowledge of the actual traffic volumes.
• Peering policy violation: Many settlement-free peering agreements are based on exchanging traffic with the other party at 1:1 ratio. Setting a ratio-based threshold on a policy that looks at interface traffic in / out can detect possible violation of agreement terms by the other party.

Ratio-based policy thresholds allow the ratio to be compared in both directions (i.e. A:B and B:A) or one direction only. In the both directions case, an optional margin parameter effectively lets the user define a “band” of acceptable ratios, with values above or below the band triggering the threshold condition.

The months of August and September 2021 are synonymous of a large feature dump in the Cloud section ! While there's too much in it for a comprehensive summary, read the full details below to get your monthly fix of Cloud features.

Security Group & Network ACL Visibility in Kentik Map

Further burnishing our credentials as the cloud network engineers’ tool of choice for troubleshooting connectivity issues in AWS, we’ve just added a new sidebar feature to the Kentik Map, Security Groups & Network ACLs.

This sidebar enhancement enables network engineers to find traffic that is currently being dropped by AWS security groups or network ACLs applied to the selected VPC or subnet. The component analyzes the selected VPC or subnet for denied traffic into or out of the network environment and then crawls through the company’s AWS metadata to allow users to determine exactly what traffic has been dropped. The component also helps users understand which security group or network ACL policies caused the traffic to be dropped.

The system works by running a query of the flow logs to or from the selected VPC or subnet to find any traffic that had been marked by AWS as REJECTED. It then analyzes the direction of the traffic to provide an at-a-glance view of these traffic flows, as well as a convenient method for searching through the traffic to find a particular source or destination.

If a user wants to find more information about why particular traffic was dropped, they only need to click on the row to open an analysis window:

The system highlights rows that contributed to the specific traffic being dropped, making it easy to determine what policy needs to be updated and even which rule could be modified in order to rectify a misconfiguration.

Users can also view these access control policies directly from within the map — a very cumbersome task using only the AWS console and/or CLI. Kentik Cloud users now need only click on View Security Groups or View Network ACLs buttons in the sidebar and the system will open up a dialog showing exactly which policies are applied to the selected object and allow the user to browse the rules associated with each policy.

Support for New AWS Dimensions

Several months ago, AWS introduced support for the following dimensions in AWS flow logs:

• Source/Destination Packet Address: Network traffic is often encapsulated — think of NAT gateways and GRE. AWS surfaced this dimension to help users determine the original source or destination of traffic and gain a deeper understanding of how traffic flows through a cloud environment.
• Source/Destination AWS Service: Traces traffic to or from AWS services, even if the traffic is tunneled. This new dimension maps the traffic based on the packet address.
• Traffic Path: This dimension shows the path that egress traffic takes towards its destination.
• Flow Direction: Marks the direction of traffic from source to destination as ingress or egress.

ENI Tagging and Dimensions

Flows are generated from network interfaces that attach infrastructure to the network. In AWS parlance, these interfaces are called ENIs (elastic network interfaces). Mapping flows based on ENIs provides an opportunity to add new dimensions to group and filter by ENI type, as well as group or filter traffic by source and destination ENI. These new dimensions allow our users to construct super-precise flow queries that don’t double count traffic to or from instances, through gateway and load balancers as well as special infrastructure like Lambdas. This is an important advantage for Kentik Cloud users.

Cloud-Native Views for Kentik Map

We also created a more welcoming experience in the Kentik Map for cloud-native/cloud-only customers. Our previous version of the map assumed that users always had an on-prem network (or would soon be adding one). The result was that the cloud infrastructure was tucked away in the Cloud Block, while the large on-prem block remained a bare focal point on the map.

No longer! Now, when single cloud users without an on-prem network register their clouds in Kentik, the map will open up either directly in their cloud’s most appropriate view — and multi-cloud users without an on-prem network will be presented with a new multi-cloud view in the center of the map. If and when users decide to add on-prem network devices to Kentik, their experience will go back to what we are used to today (an on-prem centric view of the Kentik map).

Improvements in Sidebar Traffic Queries

Did you know that sites don’t need to be directly connected to each other in order to show traffic lines in the Kentik Map? Several quarters ago, we introduced a feature called “Draw Links Using…” which enabled users to select an option to draw links based on BGP Ultimate Exit as well as Site IP addresses configured in the site architecture dialog. This enables “island” networks (networks without a backbone) or SD-WAN networks to configure their sites and easily run traffic queries between sites.

These lines are drawn by queries using new dimensions called Source/Dest Site by IP and Site Type by IP. Because we’d heard that some new business was based on this, we’ve responded by adding these dimensions into the sidebar for convenient analysis in the map.

Another quick but important usability improvement was to create a new sidebar section titled “Details.” This prevents map objects (subnets, VPCs, gateways) with lots of metadata from making the sidebar unusable.

Azure Updates

A major improvement we’ve added for Azure is the ability for companies that centralize the collection of NSG flow logs into a single storage account to create “metadata-only” exports for resource groups within the same region. To make this work, simply disable the slider called “Enable Flow Logs for this Export” on any resource groups that don’t have their own storage account associated.

We’ve also implemented some improvements to our Azure services based on customer feedback as well as added infrastructure resiliency and backend code improvements. Stay tuned for more improvements this and next quarter as we continue to round out our cloud offerings.

New Cloud Tour in Demo Mode

We’ve added a sixth tour to Kentik’s Demo Mode, which walks users through a troubleshooting scenario involving connectivity problems between AWS resources and an on-premise database. The new tour highlights the difficulty of conducting this kind of troubleshooting in complex cloud environments with existing tools, and makes very clear Kentik’s strength in helping solve these issues.

New Weather Map

This month we are excited to announce beta availability for our new Weather Map — a new core feature of Kentik Maps.

Our new Weather Map shows network engineers how their network looks so that network architectures and the current traffic patterns can be understood at a glance. This feature was one of the most requested enhancements to Kentik Maps since we went live, and we’ve only begun to scratch the surface in terms of what we plan to do here.

Today, the Weather Map is simple. It renders a company’s sites over a geo-political map, using the customer’s configured site addresses to translate to latitude and longitude coordinates. We also cluster groups of sites within the same region to declutter the map; as users zoom towards these clusters, the cluster breaks open, revealing the sites positions on the map below. Between sites (and clusters) of sites, we’ll draw links using the connected interfaces so customers can view their backbone network utilization and click on links for easy traffic analysis.

We’ve got an amazing roadmap of features coming out for Kentik maps this quarter, so stay tuned for future updates to Weather Map, AWS map and site maps in Q4.

Historical Queries

Another great new feature enhancement is our ability to rewind the clock and show users how their AWS network (and associated traffic) looked in the past, using historical metadata.

When we launched the Kentik Map for AWS, we began with a metadata service that only stored metadata describing the current state of the user’s network. However, if a user adjusted the time window to find specific flows, we assumed that the AWS architecture was the same during the specified query window as it was when the query was actually run. We knew this would eventually require historical support, which took time to design and implement.

However, that day is here! Users can change the to/from dates in the Kentik Map and we will update the map to show the user what the environment looked like during that time. If we took multiple “snapshots” of metadata during the specified time, we will show the most current we have for the time window.

This means that if traffic used to flow through a gateway that was subsequently deleted, we’ll show that gateway on the map. If traffic entered a subnet that only existed for a day or an hour — we’ll draw that subnet on the map.

Clickable Lines in the Kentik Map for AWS

We’ve added the ability to click on a line within AWS and get instantaneous traffic details for the line! In prior versions of the Map for AWS, users could only click on Map elements such as Subnets, Gateways, etc. Understanding and analyzing traffic between elements was left as an exercise for the user to construct queries using the Data Explorer. Now users can click on lines between subnets (“Show Connections”), lines between gateways, and lines to and from internet ASNs.

NAT Gateways and Transit Gateways

We also improved upon the way that the Kentik Map rendered traffic to and from gateway objects. Previous versions of the Kentik Map couldn’t determine the amount of traffic entering a subnet from a gateway. Now that we’ve switched our flow enrichment over to using network interfaces rather than only IP addresses, we can indeed show traffic from this infrastructure entering your customer’s environments.

Lots of additions in August and September 2021 in the Synthetic Monitoring department.
Almost every area of the product got new additions: from additional test frequencies (sub-second for the win !) to Page Load Test enhancements to Synthetic Test based dashboards, there's a lot for everyone, see for yourself !

More Test Frequency Options

We now offer more test frequency options. We have added 2 minute, 10 minute, 15 minute and 30 minute options to the existing (1 second, 15 second, 1 minute and 5 minute) options when configuring tests.

Ignore TLS Option

Network teams told us that they often don’t care about catching bad certificate issues and would rather “bypass” these failures to continue testing network performance. To achieve this, we have added support for ignoring TLS errors and exposed the option to configure this through the advanced settings for HTTP/API test types and Page Load test types.

Standard Deviation-based Custom Thresholds

Previously, custom test thresholds were based on static values only. While this worked well on the surface, static thresholds are limited in practical use. Specifically, static thresholds cannot be used in mesh-type tests because each pair of agents would likely need a different value. Similarly, with multi-agent tests to a single target, if the agents are in different geographic locations, the expected latency for each is different, and it isn’t possible to specify a single value that satisfies them all. For all those reasons, we have added the option to specify thresholds for latency (including HTTP latency and DNS resolution time) and jitter that are based on a multiple of the baseline measurement.

By default, the new algorithm computes a baseline for latency and jitter measurements. It uses that to determine whether to mark a particular measurement as a warning (if it is more than 1.5x the baseline) or critical (if it is more than 3x the baseline). Similarly, for packet loss we default to warning when there is any packet loss (that is >0%) and to critical if it is higher than 50%.

DNS Performance Dashboard Preset Tab

To build on our commitment to include tests to common infrastructure services that network teams rely on, we now include a new DNS Performance tab as a preset on the Synthetics Performance Dashboard.

This tab is built using the “DNS density grid” type of synthetic dashboard and features availability and uptime checks to 7 of the top/common DNS service providers from 15 geographically distributed global agents. Where available, two IPv4 and two IPv6 DNS servers are tested for each service. This allows customers to quickly rule out (or in) any DNS failures they see from “general issues” with the specific provider.

SaaS Performance Tab Migrated to HTTP GETs

The SaaS Performance tab on the performance dashboard was the very first preset tab when we launched Synthetics back in October 2020. At that time, this was a basic “hostname” test which is essentially just a ping and trace. Since then we have introduced much better tests that are capable of testing both HTTP layer and network layer performance, so in the spirit of “upgrading” to the latest, we have migrated all our existing SaaS performance tests to the HTTP or API test type.

Page Load Asset Validation

Our browser Page Load tests (only supported on app agents) are enhanced with “Asset Validation.”

• Users have the option of specifying one or more CSS selectors during the test configuration.

• The agent/test will look for those CSS selectors in the web page assets that it loads as part of the tests and if it finds them all it will indicate a PASS, but even if one of them isn’t present in the results, it will return a FAIL. Note that as with other results in the table, a grey color indicates that this metric is not used in computing test health (which ultimately drives alerts and notifications).

• To figure out how many loaded, and which selectors did not load at what points of time, the user can click through the “View Details” button to see a time-series chart. It shows the total returned selectors relative to the total configured, and by hovering over the chart, shows which ones were found and which were not found.

Dashboard Widgets for Aggregated Test Results

One of the common use cases for Synthetics is to benchmark one’s own performance against competitors as well as to compare and contrast one’s services’ performance from different parts of the world. Customers will often configure multiple tests (one set for their services and one for competitors, or tests in different parts of the world, labeled by regions) and will want to see an aggregated view. To help, we are leveraging the Kentik dashboards (Library).

There are two possible widget types:

• Table view: Great for comparing the performance of your service between different regions of the world from where your users may be accessing the service
• Gauge view: Great for comparing the overall average performance of your service (globally) relative to that of one or more of your competitors

When creating a custom dashboard widget with Synthetic data, users now have the option to choose a “Panel Type,” which may be either “Single Test” (the existing type) or the new “Multiple Test” type, which aggregates data across multiple tests. When the “Multiple Test” option is selected, the user has the option to specify the “Test Type,” “Display Type” and “Metric” that they want to aggregate data on. Optionally they can also specify agent labels that will be used to group results by in the table.

Results are displayed either as a table with each row showing aggregated values for the selected metric grouped by agent labels (user configurable).

Or as gauges showing the average metric:

These widgets can be combined with the existing widget types to produce some useful dashboards.

Notifications for BGP Monitor

We now have notifications available for BGP tests via email and other notification channels.