Kentik Data Engine (KDE) ingests and enriches flow records from routers, switches, and firewalls, as well as flow logs from cloud providers. This unaggregated data on individual flows is the basis for our comprehensive analytics functionality, including charts, graphs, tables, and dashboards. While those views are extremely useful, we’ve also given customers the option to directly view, filter, and export the raw underlying data in the Raw Flow Viewer (Analytics » Raw Flow; see the Raw Flow article in our Knowledge Base). We’ve now extended this functionality with Raw Flow Views for dashboard panels.

A Raw Flow View allows raw flow output to be embedded into a dashboard. Panels based on Raw Flow Views can inherit the devices, filters, and time range specified for the dashboard on which they live. Such panels are particularly useful on dashboards that are linked to Kentik alert policies, providing more traffic detail when investigating an alarm.

You can add a raw flow panel to a dashboard with a few easy steps:

1. Create or Edit a dashboard. When you click the Edit Mode button, you’ll see Raw Flow View as one of the options available in the Add A Dashboard Panel pane.

2. Click the Add button in the Raw Flow View card, which will open a configuration window where you can customize your query and get a preview of raw flows.

3. Click the Add Dashboard Panel button to add the panel to the dashboard.

For further information about the Raw Flow View, please contact our Customer Success team.

Virtual routing and forwarding (VRF) is a technology that allows multiple routing table instances to co-exist within the same router at the same time. Because Internet service providers (ISPs) often take advantage of VRFs to create separate virtual private networks (VPNs) for customers, the technology is also referred to as VPN routing and forwarding. With VRF support in Kentik Detect, you no longer need to manually map interface names and descriptions to VRF names and IDs (which are hard to read, troubleshoot, and support). Instead, flow data is enriched with VRF identifiers as it’s ingested into the KDE, enabling the use of VRF attributes to filter or segment network traffic in your Kentik queries.

The first phase of our VRF implementation includes support for Cisco L3VPN, Cisco VRF-lite, and Juniper L3VPN. As shown in the screenshot below, there are eight new dimensions associated with VRF support: source and destination VRF Name, VRF Route Distinguisher, VRF Route Target, and VRF Extended Route Distinguisher.

Our new VRF functionality enables multiple use cases:

• An enterprise network can verify that VRF-lite network partitions are functioning correctly (e.g. to ensure there is no traffic leaking).
• An infrastructure/network planner can see inbound or outbound traffic at the Provider Edge (PE) segmented by VRFs.
• A network operator can see all traffic associated with a specific Route Distinguisher (RD) or verify the names of the VRFs that are associated with a specific RD.
• A network operator can get alerts for changes (e.g. increase/decrease) in traffic volume per customer using VRF IDs to distinguish customers at the PE

The screenshot below shows a Sankey graph and table with all of the details about how VRFs map to interfaces on network devices. With this view, network teams can accelerate troubleshooting and easily answer questions about how traffic maps to VRFs.

As shown below, the new VRF dimensions are also supported in Alert Policies.

As we extend our VRF capabilities going forward we’ll be able to provide an even richer set of insights for analytics and visibility, including deeper integration with per-VRF BGP routing data and Kentik’s existing Ultimate Exit feature. For more information, please see the listing of VRF dimensions in our Knowledge Base, or contact our Customer Success team.

Kentik now supports the full set of fields from the NetFlow Templates that are supported by Palo Alto Networks firewalls. This first phase of our PAN integration adds huge value for Kentik customers who use PAN firewalls, providing single-pane-of-glass network visibility that now includes firewall policies and events. Beyond standard flow-record fields such as IPs, protocols, and interfaces, you’ll also now have visibility into data including user IDs, application names, and more.

You can see this new functionality in Kentik Detect’s Data Explorer. First, make sure that your PAN firewall is included in the devices selected in the Devices pane in the Explorer sidebar (shown at right). Then click on the Group-by Dimensions selector in the Query pane to open the Group-by Dimensions dialog. Scrolling down, you’ll now find (as shown below) a dimension category for Palo Alto Networks Firewall. Available PAN dimensions include:

• Source Dimensions: Post-NAT Transport Port, Post-NAT Address
• Destination Dimensions: Post-NAT Transport Port, Post-NAT Address
• Non-Directional Dimensions: ICMP Type, Flow ID, Application ID, User ID, Firewall Event, Direction

Once you select the dimensions of interest, you can order them as you like and run the query. Moreover, you can combine other data (i.e. Geo) and filter on specific firewall events to answer questions like, “Is this a region-specific problem or a general problem?” The following screenshot, for example, shows a Geo HeatMap chart of traffic filtered by a Firewall Event value of Flow Denied (the filter setting is shown in the overlaid inset).

Bringing PAN firewall events into Kentik Detect flow records empowers enterprise network teams with application visibility and enables additional security use cases, such as:

• Forensic investigation of current and past threat activity.
• Real-time verification that firewall policies are not over-or under-blocking applications.
• Cross-correlation between source countries and firewall events with a map view.

We’ll continue executing additional phases of integration, aiming for the best user experience (e.g. UI improvement and alerting integration). For more information, please see the Kentik Knowledge Base topic on Palo Alto Networks Firewall dimensions, or contact our Customer Success team.

The secret of our speedy integration with Palo Alto Networks firewalls is our new Universal Data Records architecture. With Universal Data Records, we’ve made it even easier to take advantage of the Kentik Data Engine (KDE) data store’s ability to store, unify, and query disparate data types, mapping its flexible schema to an even wider set of traffic sources, and so to bring data integration (i.e. vendor, protocol, etc.) faster to the customers for actionable insights. This approach has many advantages, like storing vendor-specific flow fields, more capacity for Custom Dimensions, and even the ability to store non-flow records that don’t contain standard flow fields like IP addresses. That makes it much faster for us to expand the types of data sources ingested into KDE, enabling visibility into a wider range of customer networks and infrastructure.

Using Palo Alto Networks firewalls as an example, with Universal Data Records we can now accept all of the fields included in PAN NetFlow Templates. While most of those fields are IANA IPFIX standard, we also include two vendor-specific fields, App-ID and User-ID (see below), that we previously couldn’t have ingested or stored.

Flow data that identifies applications and users is extremely valuable, and with Universal Data Records our customers can now take full advantage of this data to get a complete end-to-end picture of network activity.

This December 2018, we revamped the export of chart and table information from Data Explorer. The labeling in the Export submenu (from the drop-down Options menu at the top right of the chart display area) is now more intuitive, offering the following export options:

• Chart + Legend: Export, as a single PDF, both the visualization and the results table.
• Chart Image: Export, as either bitmap (PNG) or vector (SVG), just the visualization.
• Data: Export, as CSV, the data for either the visualization or the results table.

If Data Explorer is currently displaying the results of a compound (multi-axis) query, then in addition to the options listed above the Export submenu will include a Series Data option (as shown below) from which you can choose to export either the visualization (as PNG) or the results table (as CSV) associated with each individual axis of the query results.

Kentik Detect lets you apply dimension-based filtering in many locations throughout the portal, including Library dashboards, Data Explorer, alert policies, the analytics pages, and even the user admin page (where you can filter the traffic that’s visible to Member users). Filtering is applied with the Filtering Options dialog, which we’ve now redesigned in several different ways - read on!

First, we’ve restructured the dialog so that there are no longer separate working areas for ad hoc filters (defined in the dialog itself) and saved filters (Kentik presets or previously saved “company” filters). Instead, as demonstrated by the initial dialog state shown in the screenshot below, there is a single Filter Groups pane where you configure filter groups containing both types of filters.

Consolidating these working areas allows us to make a more fundamental improvement, which is to change the logic used in compound filters (filters built from multiple filter groups). It used to be that all filter groups with saved filters were first ANDed together and then ANDed with the combination of all filter groups with ad hoc filters. But now you can mix and match saved and ad hoc filters in the same filter group, either at a single level or nested, and groups can be either ANDed or ORed together.

In the filter group below, for example, you can see four distinct filters: two single-condition filters at top (source country and destination URL), then a saved filter (MYNETWORK_IN), and then a nested group that excludes traffic from two source cities.

To implement these new capabilities we’ve made some changes to filtering controls, notably the addition of the Add Saved Filter button to each filter group. As shown below, we’ve also made it easy to check the individual components of a saved filter. Just click the expand icon (right-facing triangle) to the left of a filter’s name to reveal a list of its parts.

Another improvement is that you can now convert a saved filter to an ad hoc filter, allowing you to build new filters from saved filters rather than starting from scratch. To do so, click the saved filter’s Customize button, which you can see at the top right in the screenshot above.

The enhanced flexibility that we’ve built into our new filtering UI now enables you to zero in on the precise result you need. For a complete explanation of the new filtering controls, see the Filter Groups Interface article in our Knowledge Base, or ask the Kentik Customer Success team at support@kentik.com.

Raw flow refers to actual data fields (normalized from NetFlow, sFlow, etc.) that Kentik Detect stores for each ingested flow record. We’ve long had a Raw Flow viewer in the portal at Analytics » Raw Flow, allowing you to see flow data for specified devices over a specified time range. Now you can go directly from a query in Data Explorer to a view of that query in the Raw Flow viewer.

Start by clicking the Options menu icon (hamburger) at the upper right of the chart display area in Data Explorer. At the bottom of the resulting drop-down menu you’ll now find Explore Raw Flows (see screenshot at right). Choosing this item will take you to a Raw Flow viewer that has the sidebar controls set to display flow records for the traffic that you were viewing in Data Explorer.

Alerting is one of the core features of Kentik Detect, and we’ve now made it even more useful to customers who operate very large-scale networks:

• The maximum number of active policies per company has been increased from 60 to 100.
• The maximum number of keys evaluated per policy has been increased from 300 to 1,000.

The increase in these limits reflects additional work we’ve done to better define what our alerting processes can handle and to optimize the system accordingly. This will allow customers to track and detect even more potentially anomalous network behavior. Stay tuned for additional scaling improvements in this area moving forward.

As we work to make Kentik Detect more powerful we’re also aware that features are most valuable to users when they’re easy and straightforward to use, as well as to customize for individual needs. The improvements in the next couple of sections are designed with that in mind.

Redesigned Dimension Selector

Working with dimensions for both group-by and filtering is an integral part of defining the queries used throughout Kentik Detect. We’ve taken a fresh look at how customers access and select dimensions, and made the following improvements in the dimension selectors for both group-by and filtering:

• Dimension categories – Dimensions are now organized into more intuitive categories, such as Network & Traffic Topology, IP & BGP Routing, and Cloud.

• Directional columns – Additionally, dimensions are now organized into Source, Destination, and Non-Directional columns to make them easier to locate. This also means that the Source and Destination versions of related dimensions are always located next to each other in the same row.

• New dimensions for clouds – The newly-added categories include provider-specific categories for AWS and GCP dimensions.

• Bi-directional filtering – In the dimension selector for Ad-Hoc Filter Groups, we’ve also added a Source or Destination column to simplify filtering on the same value in two related dimensions.

Custom Color Palettes

Kentik customers have always recognized UI design as one of our major differentiators. The ability to customize aspects of that design, tailoring it to individual needs and preferences, makes it even more powerful. Our latest step in this direction is the new Visualizations tab, which you’ll find on the My Profile page (access from the drop-down menu at far right in the main portal navbar).

As shown below, you can tune five different color settings that determine how visualizations are rendered:

• Theme – Toggle between the Standard Theme with a light background or the Dark Theme.
• Labels – Customize the color of labels by choosing from a popup color palette or entering a hex number.
• Overlays – Change the color of the Total and Historical overlays that are used on time-series visualizations.
• Quantitative – Choose a color theme for all chart types that display quantitative data such as Stacked Area Chart, 100% Stacked Area Chart, Stacked Column Chart, and Bar Chart.
• Qualitative – Choose a color theme for all chart types that show qualitative data such as Line Chart, Pie Chart, and Sunburst.

When changing the Quantitative and Qualitative color palettes, you can preview the effect on different view types by choosing a view type from the drop-down menu at upper right of the preview visualization. Once you save the changes, the new palette will be applied to all of your existing (and future) visualizations of that type (Quantitative or Qualitative) throughout the portal.

Finally, for Qualitative and Qualitative settings, you can create custom color palettes by turning on the Use custom values switch.

For more information, please contact the Kentik Customer Success team at support@kentik.com.

Beyond new features, we’ve continued our ongoing work on refining the utility and performance of Kentik Detect. The following enhancements cover areas that you can see as well as areas that are under the hood.

Enhanced Mitigation States and Controls

Anomaly detection, alerting, and mitigation, which are among the core features of Kentik Detect, sometimes involve complex situations like multiple mitigation actions and overlapping alarms. To better handle these scenarios we’ve simplified our state machine model. Updates include:

• Take manual control – Users can now assert manual control over mitigations that were originally triggered automatically. To support this change, we’ve created a separate set of manual mitigation states that parallel the states used in automated mitigation.
• Easier mitigation deletion – When deleting a mitigation, users now don’t have to additionally clear the mitigation on the mitigation appliance or wait for state transition to occur.
• Mitigation escalation – When an alarm escalates (starts as Minor and becomes Major), mitigations will now escalate in parallel. That means users can now associate a particular mitigation method with the minor threshold and a different method with the major threshold.

In addition to these backend changes, the UI for mitigation actions in the Active Alarms table (Alerting » Active) has been changed to provide more flexible and granular control. Play and Stop icons have now been replaced with context dependent icons and tool tips that reflect the current mitigation state.

Selective Interface Classification

A number of customers requested that we allow Interface Classification rules to be applied to some devices and not others. As shown below, the IF settings in the Add Rule dialog now include two new controls that enable you to tailor sets of Included Devices and Excluded Devices that govern application of the rule.

These whitelists and blacklists will also be displayed, as shown below, in the Rules List on the main classification page (Admin » Interface Classification).

For more information, please see the Rule IF Settings topic in the Kentik Knowledge Base or contact the Kentik Customer Success team at support@kentik.com.

Query Engine Improvements

Kentik Data Engine is the backend where your network traffic data is collected and enriched, and from which it is pulled at query run-time. Recent enhancements enable Kentik Detect to support ad-hoc queries over longer time ranges with much higher cardinality. For example, we can see source/destination IP pairs as a time series over a time range of 90 or more days. As requested by some customers, Data Explorer’s Table view can also now display much deeper results — up to 50,000 rows — for queries on certain group-by dimensions when the metric type is Total. Additional changes include improved performance for queries that filter on long lists of IP addresses.