GeoIP is one of the critical enrichments provided by our ingest layer to your Network or Cloud telemetry.

As our experience with GeoIP data has matured, we have realized that GeoIP datasets from commercial providers are most often strongly accurate for broadband/mobile providers; only moderately accurate for Content Providers, and not accurate at all for Backbone providers.

We have just rolled out an update to the Kentik platform that constantly augments our GeoIP records leveraging an extendable selection of datasets.

State of the GeoIP union

Originally, GeoIP was used to map source_IP and destination_IP from Netflow/sflow data in our unified, enriched flow record data structure, making City, Region and Country available as Flow Dimensions for querying.

Later in the life of the Kentik platform, these GeoIP mappings got used in the portal, under the hood, in a large number of areas, such as:

• In Synthetic testing; we compute the distance of a test based on the sum of the distances for all hops in the traceroute of a Network Test. If GeoIP is off for certain hops along the way, the distance will be incorrect, affecting our inference of the end-to-end latency when we compare it to the latency values obtained during the test.
• In Kentik Market Intelligence; we rank network providers in any market based on the amount of IP prefix space they get from their customer base (the more IPs, the higher the score). This means that the weight of a network can be overplayed or downplayed in the case of incorrect GeoIP data.

How did we fix it?

We initially relied on a simple system that takes in GeoIP data daily from our providers and reloads it into our ingest layer to constantly fetch the provider's updates, and apply them as soon as available.

When you as a customer would notice inaccuracies, we'd relay the evidence to our provider and they would surface them once blessed by their experts in a future update. We were not satisfied with the end-to-end time to satisfy customer requests for relocation so we built an override layer system to which we could feed both:

• Manually, by entering our own overrides
• Programmatically, by using additional external trusted datasets

We landed on a modular and layered architecture below, that does the following with each daily run:

1. fetch the provider GeoIP Dataset
2. overlay our own overrides based on their respective precedence/priority
3. generate a resulting GeoIP custom Dataset
4. swap the dataset on the fly on our ingest memory datastore as part of the daily job

Leveraging your SNMP data

Network Devices exporting flow data to Kentik can have SNMP enabled on them and fed to our ingest. One of the MIBs polled by our SNMP service is the interface MIB. With each poll, we get all interfaces for the network element and the configured IPs on these interfaces.

As part of our enrichments, our customers also declare their network elements in Sites, so they can later query telemetry data by site. The bonus is that users submit an address that we translate in Geocoded data so we can place sites on a map.

Every day, we scan the entire partitions of IP addresses learned on each device via SNMP, and for each public IP address, we associate it to the Geocoded data of the site, which contains the network element, which contains the public IP address.

This is a somewhat unique dataset that no GeoIP provider out there has at their disposal. We now leverage it daily as a "Layer" of overrides superseding this base GeoIP dataset from our provider. 

Benefits for our customers

As of now, if you ever notice an inaccurate GeoIP mapping when using Kentik, we have reduced the time to correction by as much as 15x: while the typical back and forth process with the GeoIP provider would potentially take 15 days (we vet, then the GeoIP provider vets, then slots it to their next synchronous release).

Today, we are able to add an override to our GeoIP system immediately after we have vetted the evidence you submit, cutting this turnaround time to one day or less.

For critical updates, we can even have the engine recompute a complete map on demand.

As of today, our public link sharing capabilities got extended to now include dashboards.

As displayed in this recent update, you can try the feature on a dashboard with the new Share console that appears when you click on the top-right share button:

The new Share console as displayed from a dashboard will look like so, where you can easily change the human-readable URL, rope in email recipients to share it with (including an explanatory message)

And here's a sample example of a publicly shared dashboard:

Why don't you play around with the feature and show us some of your most eye-candy public dashboards ?

We've enhanced the synthetic library widgets to include a scrubbable health status bar! 

While previously the widgets displayed results of just the last test run executed, this enhancement allows you to scroll back and forth to review the results historically on the widget itself without having to jump into the test page itself. This will aid in reducing the Mean Time To Identification (MTTI) of network or application issues.

The health status bar is available for all single test synthetic widgets.

We've expanded the capabilities of the Kentik Synthetic Application agent to support all test types. Application agents can now execute all Network based tests (including Network Mesh & Grid and Autonomous Tests) and DNS based tests in addition to the Web tests (HTTP, Page Load and Transaction Test) they already supported. 

BGP Monitor tests are not applicable since these use BGP Vantage Points and not synthetic agents.

Look for the list of Private or Global App Agents in the agent selection modal at test creation.

Find the full list of supported Global App Agents on the Agent Management page

Use the agents to run Network tests like the agent to agent test seen below.

Today, we just released a significant update to the way Kentik users will share the Kentik network observability experience, inside and outside of the company.

We have taken a set of horizontal features and have grouped them into consistent UX patterns for sharing Kentik content, whether these are reports, public snippets or even subscriptions. Read on!

What’s new?

With this update, we’ve centralized the Sharing functionality and streamlined the way it is presented on all screens of the UI/UX. As explained earlier, this creates a more holistic approach to sharing data from Kentik inside and outside.

This includes multiple areas of work, detailed in the sections below.

A redesigned, streamlined Share button on every screen

Extending the subscription capabilities to many more screens

Historically, subscriptions were only available for dashboards, we wanted to do more to meet Kentik users “where they live”.
We extended subscriptions to a discrete set of screens/workflows, with the intent of furthering this list in the future. Subscriptions are available immediately for the following items (we followed our users and their feature requests to make a determination on which to work on first):

• Dashboards
  We’ve added a few under the hood goodies such as the ability to treat carriage returns properly in subscription email messages, as well as offering another previously unavailable frequency: “Last Day of the Month” (useful to vet connectivity invoices at the end of the month)
  
  
• Saved Views
  Subscriptions were not available for Saved Views previously. They are now.
  
  
• Connectivity Costs
  We’re offering here a compelling subset of functionality, 4 types of subscriptions can be configured: Summary (corresponding to the summary screen), Provider(s), Site(s) and Connectivity Type(s).
  Each one of the non summary subscriptions allows Admins to configure which set of Providers/Sites/Connectivity Types to configure these subscriptions for to bundle in one delivery.
  
  
• Capacity Planning
  As for Connectivity costs, users can now subscribe to a summary (aka front page) or a subset of Capacity Plans.

Extending the CSV export functionalities to many more screens/workflows

A significant number of non-technical users have requested (via FRs) to be able to download CSV data for some of the most used workflows, and be able to massage it further into their own reporting.

Additional CSV exports available

First and foremost, CSV report download is now available in these workflows/screens

• Saved views

• Connectivity Costs

  • All connectivity costs
  • Per provider
  • Per site
  • Per connectivity type

• Capacity Planning

  • All plans details
  • Any number of plans together in a single CSV

• Kentik Market Intelligence

  • Rankings CSV
  • Customer and Provider CSV for any given network

To further streamline the experience, all reports download actions are now located in the exact same location, regardless to the screen the user is on, that is within the [Action] button as it initially was for Data Explorer.

CSV downloads now available for subscriptions

Not only have these CSV capabilities been extended, but they are now also available in subscriptions, which answers multiple Feature Requests we had had on Capacity Planning and Connectivity Costs.

Below is an example of a Connectivity Cost report being sent 

• for Transit and Paid Private Peering
• As a CSV spreadsheet
• The Last day of the month

Multiple convenience UX changes to make this feature set work better

Settings + relocating Public Link Shares

While initially in the Navigation panel, Public Shares have now been moved together to the Settings screen, in their own “Share” panel to the right, bundled together as a cohesive functional unit:

Subscriptions & Sharing features in Library

1. Library now gives a clear indication of when a given Saved View or Dashboard is involved in a subscription: the underlying goal is for admins to know before modifying a view that other users may be relying on it.
   
   
   
2. Users can also now both subscribe and unsubscribe directly to a Saved View or Dashboard from the library by using the contextual actions:
   

Once Public Sharing is finalized for Saved Views and Dashboards (currently ongoing), a similar marker will be displayed.

Redesigned Report Subscriptions management screen

As displayed in the previous screenshots, the Report Subscriptions UI can now be accessed directly from the Sharing Modal, as displayed below:

The Report Subscriptions management screen has been significantly updated and now offers all the convenience features that our usual list screens offer, such as the search panel on the left.

Amongst the very useful items, Users can now quickly identify using filters:

• Subscriptions that they are recipients of
• Subscriptions that have non-kentik account emails in them (aka Shared Externally)

These two situations being potential data exfiltration threats, the UI allows Kentik Admins to identify them easily.

Subscription Auto-Sense at share time

As users ideally want to limit the amount of email subscriptions being created around the same screen/workflow/report, the sharing modal performs some amount of magic:

If a subscription already exist when the user hits the [Share] button and goes to the Subscription tab, the Subscription Title entry will offer to either

• Modify an existing subscription
  OR
• To create a new one

Sample share screen displaying the choice between "Create New subscription" vs "Edit Existing Subscription"

Observation Deck is our vision for a new focal point of the Kentik platform. It brings together all of our products & modules (current and future), in one place, so customers can get a complete and tailored view of their network and infrastructure. Every new customer will get a curated, default Observation Deck at the end of their initial onboarding. Observation Deck can then be configured to meet their exact needs.

Now customers with Azure Cloud infrastructures can add widgets to their Observation Deck and the Kentik Cloud landing page has been extended to show Azure views as well.

Note the addition of “Cloud Provider Filters” at the top right. This allows to visualize either a multi-cloud environment (“All”), or a specific Cloud. Traffic/flow widgets will filter data by cloud provider if selected.

Azure regions are now shown on the Weathermap.

For a multi-cloud environment, here is how this looks, including the Cloud backbone traffic.

Setting up monitoring of Cloud environments with Kentik can be difficult, especially when users run large multi-account environments. Ensuring that each account has the proper roles with the necessary permissions for Kentik to retrieve metadata and flow logs can be challenging to maintain at scale. That’s where our Cloud Config status comes in. It quickly analyses complex environments to ensure that all accounts are configured as required for Kentik to monitor Cloud networks. This capability is now available for Azure in addition to AWS.

We're excited to announce that our initial launch of Synthetic Transaction Monitoring (STM) is now GA!

Transaction tests expand on our existing suite of web/app layer testing capabilities. While an HTTP/API test tests the availability of the front door (hosting web server) of an application and a Page Load test gives you the ability to test the performance of how a website loads, a Transaction test goes much deeper and allows you to simulate a user's journey as they interact with an application, like logging into an email application or checking out a shopping cart or searching for a stock ticker symbol.

For instance, an e-commerce site has many user workflows which enable a customer to login, search through selections, select payment and delivery options and finalize a transaction. Delays or errors in any of these steps can impact the customer experience and lead to lost revenue. STM will enable you to benchmark the performance of each of these transaction stages and troubleshoot specific performance issues.

Steps to perform STM in Kentik:

1. Mimic the actions of your users as they would log into your application and perform a transaction.
2. Records these actions using the Chrome DevTools Recorder.
3. Open your Kentik account and create a new Transaction Test.
4. Paste the script exported from Chrome DevTools Recorder.
5. Select the agents (private or public) that you’d like to run the test from.
6. Run the test and analyze results.

Here is an example of a Transaction recorded on Chrome DevTools.

We can now export the Transaction script as a Puppeteer script and paste it into new Kentik Transaction script as below 

You can select any of our public Application agents to test from, and/or easily deploy your own private Application agents which we can supply (for Docker, x86 and ARM). STM tests can be set as both automatic and periodic on time intervals.

Presentation of Synthetic Test Monitoring Results

Results are presented on a timeline that shows transaction completion time. Performance is measured against dynamically calculated baselines and lags in performance are colored - orange is a warning, red is critical. 

Selecting a point on the line will indicate total completion time and the rolling standard deviation baseline. 

Screenshots captured during the transaction process provide insight into the script execution flow and aid in troubleshooting. 

The Waterfall tab shows the load order and load duration of each element in the Document Object Model (DOM) of every page visited.

Analyzing the results give us insights into how our users experience the performance of an application in real time from different geographies allowing us to proactively respond to user experience issues before they are reported by real users. When used in conjunction with Kentik's suite of network tests we can identify the root cause of performance issues as originating in the network or application stack within minutes.

Kentik Market Intelligence gets a couple useful improvements added.

KMI will now let you export rankings and customer/provider lists as a CSV file.
Additionally, we've added a "density chart" on top of the KMI landing page before the ranking table.

The  KMI landing page has been updated with what we call a Density Chart: the usual table-chart we all know and love did not really convey the idea of the actual distance between consecutive networks in the ranking. This new widget has been added on top of the ranking table and places Networks on the line based on how far they are from each other in terms of KMI score.
(check out this article out if you want to understand how KMI score are computed)

Based on popular demand from our KMI customers, we've added a functionality to download the CSV data from both the landing page, and any given network's detail page: