HyperScale Custom Dimensions & API
Custom Dimensions provide a mechanism by which Kentik Detect customers can effectively add custom columns — queryable dimensions — to the main tables of the Kentik Data Engine (see Main Table Schema). Our developers have worked up a major upgrade to our Custom Dimensions engine, which we’re calling Hyperscale Custom Dimensions (HSCD). This new approach increases the capacity and agility of Custom Dimensions by eliminating many constraints of the legacy system, particularly with regard to the cardinality of values.
Legacy, UI driven Custom Dimensions | Hyperscale Custom Dimensions (HSCD) |
---|---|
Maximum of 10,000 populators over 12 Custom Dimensions | ~100,000s of populators for shared Kentik SaaS setup ~Millions of populators for single-tenant Kentik On Prem setup |
~20 min target ingest-to-flow latency | <5 min target ingest-to-flow latency |
HSCD Availability and Use
While HSCD is already fully functional, our engineering team is currently completing final refinements in preparation for General Availability release. In the meantime, we encourage you to get familiar with it to see what it can do. So we’re launching HSCD in preview mode and putting it into the hands of customers who request it from Customer Success (support@kentik.com).
Note that to define HSCD dimensions you’ll use our new HSCD API. Instead of exposing direct REST endpoints like we do for our query and admin APIs, we’re making HSCD available via an Open Source client library, written in Python, that we’ve published on GitHub. We will also soon add the API itself to our API Tester and publish API documentation.
New Custom Dimension UI
Once we activate the new feature for you, you’ll be able to use the Python client to push Custom Dimension mappings, and you’ll see some changes to the controls in your Custom Dimensions dialogs (see screenshot below). The new UI allows:
- Paging when there is a large number of populators.
- Searching for Custom Dimension values across a large number of populators.
Stay tuned to this space as more details will follow soon about the powerful features that are enabled by the increased scale made possible by HSCD.