Kentik now supports the collection of the NetFlow and IPFIX timestamp fields (starting kproxy v7.38.0). The additional 3 fields are available to be viewed in the Raw Flow Viewer:

• Flow Start - Timestamp of the flow start
• Flow End - Timestamp of the flow end
• Duration - Calculated duration of the flow as Flow Start - Flow End

NetFlow v5 and v9

In the case of the NetFlow v5 and v9, Start and End Flow timestamps are calculated using System Uptime and Unix Seconds fields from the NetFlow header and the following fields from Flow records:

IPFIX

In the case of the IPFIX, the timestamps are determined in the following two ways:

1. Using flowStartSysUpTime, flowEndSysUpTime and systemInitTimeMilliseconds fields:

   ID	Name	Type	Description	Units
   21	flowEndSysUpTime	unsigned32	The relative timestamp of the last packet of this Flow. It indicates the number of milliseconds since the last (re-)initialization of the IPFIX Device (sysUpTime). sysUpTime can be calculated from systemInitTimeMilliseconds.	milliseconds
   22	flowStartSysUpTime	unsigned32	The relative timestamp of the first packet of this Flow. It indicates the number of milliseconds since the last (re-)initialization of the IPFIX Device (sysUpTime). sysUpTime can be calculated from systemInitTimeMilliseconds.	milliseconds

   Both fields in the description refer to the additional field which is equivalent to the SysUpTime:

   ID	Name	Type	Description	Units
   160	systemInitTimeMilliseconds	dateTimeMilliseconds	The absolute timestamp of the last (re-)initialization of the IPFIX Device.	milliseconds

   
   

2. Using some of the IPFIX-specific fields for the flow start and flow end timestamps. Not all device’s IPFIX implementation used these fields, but if they are present, they are preferred and used:

   ID	Name	Type	Description	Units
   150	flowStartSeconds	dateTimeSeconds	The absolute timestamp of the first packet of this Flow.	seconds
   151	flowEndSeconds	dateTimeSeconds	The absolute timestamp of the last packet of this Flow.	seconds
   152	flowStartMilliseconds	dateTimeMilliseconds	The absolute timestamp of the first packet of this Flow.	milliseconds
   153	flowEndMilliseconds	dateTimeMilliseconds	The absolute timestamp of the last packet of this Flow.	milliseconds
   154	flowStartMicroseconds	dateTimeMicroseconds	The absolute timestamp of the first packet of this Flow.	microseconds
   155	flowEndMicroseconds	dateTimeMicroseconds	The absolute timestamp of the last packet of this Flow.	microseconds
   156	flowStartNanoseconds	dateTimeNanoseconds	The absolute timestamp of the first packet of this Flow.	nanoseconds
   157	flowEndNanoseconds	dateTimeNanoseconds	The absolute timestamp of the last packet of this Flow.	nanoseconds
   158	flowStartDeltaMicroseconds	unsigned32	This is a relative timestamp only valid within the scope of a single IPFIX Message. It contains the negative time offset of the first observed packet of this Flow relative to the export time specified in the IPFIX Message Header.	microseconds
   159	flowEndDeltaMicroseconds	unsigned32	This is a relative timestamp only valid within the scope of a single IPFIX Message. It contains the negative time offset of the last observed packet of this Flow relative to the export time specified in the IPFIX Message Header.	microseconds

sFlow

sFlow is packet sampling technology, which means that there is no flow, hence no flow start or flow end time.

sFlow also does not contain any timestamp, so the behavior is the following:

• Flow Start: save the time of arrival/processing of the sFlow packet
• Flow End - same as Flow Start
• Duration - 0

Raw Flow Viewer

In the Raw Flow Viewer, these additional flow timestamps need to be selected as Dimensions to be shown in the output, as shown below:

We released a new configuration knob that allows customers to change the sampling rate for AWS and Azure on their own without contacting Kentik team.

That will allow customers to consume flow logs at the preferred rate fitting into the licensing strategy, assigning priority for certain types of traffic and being flexible by changing the sampling rate at any time and separately for each flow log exporter.

Licensing will be enforced after the sampling, so customers can use heavier sampling in some cases, and saving the licensed FPS for the another S3 buckets containing flow logs.

There is a slight difference in available options for AWS and Azure.

AWS flow log sampling

Historically Kentik was supporting a “legacy” mode of sampling where for the large files with flow logs we were randomly picking 10,000 flow records per file in S3 bucket and ingesting only those records into Kentik Data Engine. Since the number of the flow in a file can vary this was considered an “adaptive sampling” where larger files were getting more heavily sampled comparing to the smaller files. Another option was no sampling i.e.  all the records were consumed from the file.

Moving forward we now support 3 options for AWS:

• Legacy sampling - random 10,000 flow records per file.
• Sampling rate - where user can provide the sampling rate in 1:N format (meaning 1 out N records to be picked up for an ingest into Kentik Data Engine), where N should be between 2 and 2000.
• Unsampled - all the records in a flow log file will be taken into ingest. Effectively that is the same as sampling rate 1:1.

Sampling rate can be configured when new flow log file is added, or changed for the existing exporter.

Azure flow log sampling

Flow log exporters for Azure before this release were supporting only Unsampled mode, where all the flows from the flow log file were processed by the Kentik Data Engine.

Since for some situations full flow log visibility might be not required, we added sampling knob that allows users to configure sampling rate 1:N format (meaning 1 out N records to be picked up for an ingest into Kentik Data Engine), where N should be between 2 and 2000.

Kentik now supports collection of the NetFlow and IPFIX fields for position 3 MPLS Label in the label stack , which we previously not collected from the received flows. The related fields is shown in tables below:

NetFlow v9 VLAN fields:

Resource: https://www.cisco.com/en/US/technologies/tk648/tk362/technologies_white_paper09186a00800a3db9.html

IPFIX VLAN fields:

Resource: https://www.iana.org/assignments/ipfix/ipfix.xhtml

This field is collected from NetFlow/IPFIX protocols and stored in the Kentik’s MPLS Label 3 and MPLS Label 3 EXP dimensions.

The support is available in kproxy starting from version v7.37.0. The example of the Data Explorer query is shown below:

Kentik added support for 6PE technology and BGP IPv6 Labeled-Unicast address family to be used for Flow enrichment.

The IPv6 Provider Edge (6PE) is the technology that enables isolated IPv6 networks to communicate using MPLS LSPs over an IPv4 MPLS backbone network.

The diagram below demonstrates the typical scenario for the use of the 6PE technology. CE routers which are at the border of the IPv6 islands, advertise their IPv6 routes to the 6PE routers of the MPLS network. These PE routers are the only dual-stack routers, which support both IPv4 and IPv6. The 6PE router advertise the received IPv6 routes to other 6PE routers using MP-BGP session over IPv6 Labeled-unicast address family. These route have:

• Original IPv6 route received from CE
• Inner MPLS label value, which would be used in the 6PE router’s data-plane to encapsulate packets toward IPv6 island’s networks.
• Next-hop with the IPv6-mapped IPv4 address which is in the form of “::FFFF:<IPv4-address>”.
  • The mapped IPv4 address is the address of the advertising 6PE router.
  • It determines the Outer MPLS label which will encapsulate IPv6 packets inside the MPLS network

To perform enrichment based on the 6PE information, Kentik’s user should establish the BGP session with the IPv6 Labeled-unicast AF between their 6PE router and Kentik. Based on the received IPv6 LU routes the Kentik’s ingest layer would be able to enrich the Flow’s received from those 6PE routers. All “standard BGP” dimensions would be populated, but more specifically:

• The “Next-hop IP” dimension will be populated with the next-hop IPv4-mapped IPv6 address from the received route
• Based on the IPv4-mapped address, the Kentik ingest would identify the next-hop 6PE router and populate “Ultimate-Exit Device” dimension and based on that, the other “Ultimate-Exit” dimensions.

Additionally, to instruct the use of Labeled-unicast routes, a user needs to select additional configuration for 6PE routers in the Kentik.  At the Settings → Devices page, select the 6PE router device and “Edit device”, then at the BGP Tab, at the “BGP route selection” drop-down menu select the option  “VPN table, fallback to Labeled-Unicast table, fallback to Unicast table”.

The example of the Data Explorer output with the 6PE BGP next-hop dimensions is shown below:

Kentik has added a new configuration option, which determines how the BGP routes are selected for flow enrichment process. To make the whole process clear enough we should start with the basics.

BGP sessions

BGP session between customer’s router and Kentik can be established over:

• IPv4
• IPv6

Since these are “Multiprotocol BGP” sessions, for each of the sessions, it is possible to enable multiple Address Families, for example: Unicast, Multicast, Labeled-unicast, L3VPN, Flowspec, etc.

These Address Families are defined with AFI (Address Family Identifiers) and SAFI (Subsequent Address Family Identifiers) attributes. They are regulated by IANA and the exact values can be found on the following links:

• IANA AFI numbers: https://www.iana.org/assignments/address-family-numbers/address-family-numbers.xhtml
• IANA SAFI numbers: https://www.iana.org/assignments/safi-namespace/safi-namespace.xhtml

The Kentik side of the BGP peering with customer’s devices will be enabled with the Unicast, Labeled-Unicast and L3VPN families by default. For the BGP “IPvX” session from the Kentik side will have the following AFs enabled:

• “IPvX” unicast
• IPv4 and IPv6 labeled-unicast
• “IPvX” L3VPN - (IPv6 L3VPN address family is not used)

Received routes from each of these address families are stored in the separate route table, which is check during the Flow enrichment process.

NOTE: IPv6 VPN routes are received, but not used for the enrichment

The Flowspec address family will be enabled only if the customer explicitly enable it in the device configuration on the Kentik portal.

BGP attributes enrichment process

Assignment of the “Route Prefix/LEN” dimension

The assignment of the Src and Dst Route Prefix is the following:

• Src and Dst Route Prefix dimensions are first populated from the Flow information using Src and Dst Mask field from Flows - if applicable.
• Src and Dst Route Prefix will be overwritten further in the ingest processing if there is a matched BGP route.
• The way to know if the Src or Dst Route Prefix is coming from flow or BGP is by observing other BGP route attributes:
  • if the Route prefix originates from the flow information the dimension “Next-hop AS Number” will be “0 - -Reserved AS-,ZZ” and the dimension “AS Path” will be empty.
  • if the Route prefix is overwritten by the BGP information, the BGP related dimensions such as “Next-hop AS Number” and “AS path” will be populated

VRF metadata collection

As the part of the SNMP interface discovery process, Kentik SaaS or Kentik kproxy will perform the VRF discovery and interface association. This information about the VRFs is collected over SNMP using MPLS-L3VPN-STD-MIB, if the device supports it. The devices from Cisco and Juniper Networks support this MIB. We have also developed support for for Nokia’s proprietary MIBs.

For each VRF, Kentik collects:

• Name
• Description
• Route Distinguisher (RD)
• Route Target (RT)
• Interface association

BGP route matching process

The enrichment of the BGP/Route related Flow dimensions is performed as a result of matching the Flow’s IP address against the BGP route received from customer’s device over BGP sessions. The default behavior of the matching process is the following:

• Flow’s Src interface is checked if it is assigned to the VRF.
  • If the source interface is in the VRF, flow’s Dst IP address is looked-up against the BGP VPNv4 routes with the RD associated with the source interface’s VRF:
    • If there is a route match, the route will be assigned to the flow
    • If there is no match, or there is no BGP VPNv4 table at all, or even no L3VPN AF established as part of the BGP peering, the match will not be found and BGP route dimensions are not populated.
  • If the source interface is not in the VRF, flow’s Dst IP address is looked-up against the “global” BGP table containing Unicast IPv4/IPv6 AF routes.
• The same process is performed for flow’s source IP address route lookup, based on the destination interface association with the VRF.

BGP route selection configuration

To address some additional scenario’s that we have seen in the customer’s network, Kentik added the configurable option to influence the BGP route selection process related to which BGP routes will be used for matching process.

This configuration is available at the Settings → Devices → Edit Device dialog → BGP Tab.

At the dialog, there is a new drop down menu called “BGP Route Selection” with the following three options:

• VPN table for VRF interface, Unicast table for non-VRF interface (default option)
• VPN table, fallback to Unicast table
• VPN table, fallback to Labeled-Unicast table, fallback to Unicast table

The following table describes the behavior of each configuration option:

Kentik now supports collection of the NetFlow and IPFIX fields for source/destination VLAN, which we previously not collected from the received flows. 

The related VLAN fields are shown in tables below:

NetFlow v9 VLAN fields

Resource: https://www.cisco.com/en/US/technologies/tk648/tk362/technologies_white_paper09186a00800a3db9.html

IPFIX VLAN fields

Resource: https://www.iana.org/assignments/ipfix/ipfix.xhtml

These two fields are collected from NetFlow/IPFIX protocols and stored in the Kentik’s Source VLAN and Destination VLAN dimensions.

The support is available in kproxy starting from version v7.36.0. The example of the Data Explorer query is shown below: