Building a Secure Modern Data Platform for Quality of Service, and Customer Insight.
TalkTalk is the UK's leading value for money consumer and B2B telecoms provider. Their mission is to deliver affordable, reliable, simple, and fair services for everyone.
In 2016, TalkTalk engaged endjin to investigate a very specific problem: "we're seeing unexplained spikes in our network telemetry data". Every day over 1 million set top box devices generate 10 GB of diagnostic data, and although the network operations team could spot unusual patterns, data analysis was a labourious, manual process that often exceeded the SLAs to resolve such issues. Endjin ran a series of data science experiments to establish if an anomaly detection model could identify known anomalies in historic data.
These experiments indicated that the raw telemetry needed to be processed in order to improve the signal to noise ratio, after which, the anomaly detection algorithm could be applied effectively to each stream of related events to identify anomalous activity. The results could be aggregated, denormalized, and correlated to show relationships between different anomalous events and higher level patterns across periods of unusual activity.
Endjin's experience & processes enabled us to move on from simply collecting data, to asking business questions, and gaining valuable insights, which allowed us to focus on the right business decisions.
Microsoft Azure data services were used to productionise the end-to-end solution - orchestrating the data movement and transformation with Azure Data Factory, hosting and running the model in Azure Machine Learning, and displaying a series of interactive and dynamic exploratory dashboards in Power BI that gave the team on-demand access to the latest data. With the Anomaly Detection Platform in place, the customer insight team could understand unexplained issues and outages, identify patterns of unusual behaviour, carry out proactive support and maintenance, improve customer experience, and reduce customer churn by making customer quality investments based on reliable data insights.
The Modern Data Platform blueprint helped us navigate the complexities of security & governance as we migrated data to Azure.
In 2017, building upon the Anomaly Detection Platform, endjin were asked to create a Modern Data Platform, based on Azure Data Lake Store to store and analyze vast amounts of hitherto untapped Radius based customer networking telemetry to enable TalkTalk to gain deeper insights about customer behaviour, quality of service, customer experience and churn prediction, while designing for data security and GDPR considerations.
