Tubi Time Series KPIs Outliers Detection
Welcome to the world of Tubi — where the excitement of ad-supported video-on-demand never stops!
Introduction
As one of the world’s largest streaming services, Tubi continues to experience steady growth. We like to stay on top of emerging trends that are essential to our success.
It’s the reason why the Tubi Data Science Team developed a new alerting system that detects outliers and trends in our Key Performance Indicators (KPIs). We know how monitoring KPI metrics can be a daunting task. We also know how easy it is to get lost in the numbers and overlook significant events. That’s why we created a system that is flexible enough to account for diverse segments while also accurate enough to increase true positive rates and decrease false positive rates.
Are you excited, yet?
If you are, let’s dive in together and find out more on how the Data Science Team developed this new system while overcoming some major challenges along the way.
Challenges and Requirements
With the steady, organic, evolution of Tubi’s service, we observe a general trend of upward growth across a majority of our metrics. Tubi’s usage shows seasonal trends, including fluctuations in engagement on weekends, weekdays, and holidays. As a result, we had to make proper adjustments to avoid any potential misinterpretation of data. Without careful adjustments, it is possible for a drop in metrics to be overlooked amid the overall growth trend or falsely flagged due to weekly fluctuations.
Another challenge we faced was having to work with a diverse range of metrics. This is because some of our metrics are percentages constrained between 0 and 1 (e.g., conversion and retention), while the other metrics have no upper bound (e.g., view time and the number of visitors). The behavior of these metrics can be heavily influenced by audience type and platform size. We had to find a solution to help us account for the diverse trends and behaviors.
The solution we developed involved segmenting our metrics into distinct time series representations based on audience category and platform combinations. This allowed us to take a more detailed look at each metric and identify issues with greater accuracy. By breaking the metrics down into different cuts based on various criteria, we were also able to remove any inherent bias in the metrics.
To remove trends and seasonality from each segment, we utilized the powerful seasonal_decompose function from statsmodels. This function is a straightforward approach to decomposing a time series into its core components because it helps us to identify and remove trends and seasonality from each segment’s time series.
Two visual examples are as follows:
The above visualizations illustrate the impact of decomposing a time series, which results in the removal of most seasonal spikes and dips while preserving the overall characteristics of the time series.
Our new KPI alerting system was subject to a rigorous set of requirements. These include:
- Coverage of a prioritized list of metrics and user segments.
- Detection of anomalies with high precision.
- Ability to prioritize alerts.
- Inclusion of both drops and spikes in the alerting system.
The reliability and effectiveness of the system hinged on our ability to maximize true positives and minimize false positives. Given these requirements and challenges, the team had to implement a system flexible enough to account for all diverse segments, and accurate enough to increase true positive rates and decrease false positive rates.
So, how did we develop the new system?
We divided our methodology into two specific implementations:
- Methodology Part 1 — Point anomaly detection
- Methodology Part 2 — Trend detection
Methodology Part 1 — Point anomaly detection
Point anomalies are observations that significantly differ from the typical pattern of the time series. To detect them in Tubi KPIs, we utilized the Matrix Profiling (MP) algorithm because it receives a time series as input. It also calculates a score for each point in the series that indicates how different the value is compared to others. To perform MP, we utilized the widely adopted open-source STUMPY package.
We chose MP for point anomaly detection for the following reasons:
- Simplicity and ease of implementation — MP is straightforward to implement and can be easily adopted through standard libraries.
- Explainability — MP is a great choice for use cases where interpretability is essential.
- Space efficiency and speed — Most MP algorithms require little memory and are highly parallelizable. This results in fast computation times.
- Anytime algorithms — MP can be computed in an anytime fashion. This allows for fast real-time usage without specific training data/windows.
- Maintainability — MP results can be stored efficiently. Maintenance is relatively pain-free due to the straightforward implementation.
- Unsupervised method — MP does not require pre-labeled data points or subsequent training/testing phases.
Now, let’s break down the MP algorithm into specific steps.
Matrix Profile Algorithm
A Matrix Profile is a data structure that was introduced in 2016 by multiple researchers across various universities.
A Matrix Profile Algorithm is defined as:
A vector that stores the (z-normalized) Euclidean distance between any subsequence within a time series and its nearest neighbor.
When using Matrix Profile, a time series is divided into consecutive subsequences of a fixed length that are compared to each other using Euclidean distance, or other distance calculations.
This comparison is performed on a sliding basis until every possible combination is covered.
The results of these comparisons are stored in the Matrix Profile.
When computing the Matrix Profile, only the distance from the nearest neighbor is kept for each subsequence. This score represents the overall similarity or dissimilarity of the subsequence in comparison to the rest of the time series.
To interpret the results of the Matrix Profile, it’s important to understand that low values indicate potential motifs or patterns in the time series. In other words, these low values represent subsequent pairs that are highly similar to each other and may indicate that there is a recurring pattern in the time series. High values, on the other hand, may signal possible outliers or anomalous events in the time series.
It’s worth noting that while the Matrix Profile provides a powerful tool for analyzing time series data, it’s important to consider the context of the data and the specific problem being addressed. Visual inspection of the time series and the corresponding Matrix Profile can help in understanding the patterns and anomalies in the data.
For more information on the Matrix Profile algorithm, please refer to the official stumpy documentation.
Matrix Profile Implementation
Now that you understand how the matrix profile works, how did we actually implement it?
For each segment, the final point anomaly detection method works as follows:
- The time series data is preprocessed to remove both trend and seasonality.
- The preprocessed data is fed into the Matrix Profiling function in three different versions to improve the robustness of the following results:
- Raw version — No changes are made to the time series data before it is analyzed.
- Moving Block Bootstrapping version — The time series is split into smaller segments that are randomly shuffled to create new sequences for analysis to reduce the impact of any trends or patterns in the data.
- Random Window Partitioning version — The time series is split into smaller overlapping windows of a fixed length, and a random subset of the windows is selected for analysis to capture the local structure of the data and reduce the impact of any trends or patterns.
3. The weekly percentage change is calculated for each point.
4. The final anomaly score for each point is calculated as the sum of the
individual MP results, multiplied by the weekly change.
5. Any score above a threshold is labeled as an anomaly and logged in a
table.
Our Matrix Profile implementation is designed to detect anomalies in time series data, with a focus on optimizing the mean degree of daily drop.
We achieved this by testing our implementation using a range of flexible parameters and looping through different parameter combinations to maximize the detection of historical outliers. The parameters are:
- KPI days — The length of the input time series. We want to cover enough days to include as much signal as possible while reducing unnecessary noise.
- MP window — The length of the time series snippets that will be compared to define the anomaly score.
- C-factor — A multiplier factor to adjust the threshold. By multiplying the threshold by a C-factor, we can make the algorithm sensitive to potential anomalies.
- A higher value of “C” will result in a higher threshold that makes it difficult for the algorithm to detect anomalies.
- A lower value of “C” will result in a lower threshold that makes it easier to detect potential anomalies.
Our goal was to find the best combination of parameters to achieve the highest mean degree of daily drop. We also performed visual inspections of the results to ensure the accuracy of the anomaly detection.
Outlier Detection With Matrix Profiling
Methodology Part 2 — Trend detection
In some cases, a metric may not exhibit sudden, alarming changes but instead may experience a gradual and consistent decline. To identify such cases, we utilized a technique called Moving Average Convergence/Divergence (MACD).
While MACD is typically used in stock market analysis to track trends and develop buy/sell signals, we opted to use it for its applicability to any time series data and its ease of interpretation.
The logic behind MACD is described by the following flowchart:
- The time series data is preprocessed to remove both trend and seasonality.
- An Exponentially Weighted Moving Average (EWMA) function is applied to the data using two different parameters. One parameter is for a slow window and the other parameter is for a fast window. This helps to identify trends in the data at different time scales.
- The fast trend is subtracted from the slow trend and a smoothing filter is applied to get the MACD Signal.
- When the input is mostly going upwards, the signal is positive.
- When the input is mostly going downward, the signal is negative. - The result from Step 2 is subtracted from the result of Step 3 to get the MACD histogram. This histogram helps us to detect gradual changes in the time series data.
Using the MACD values, we can pinpoint a positive momentum when the trend is above zero, and a negative when the trend is below zero. The points where the signal crosses the zero line indicate a change in the trend, particularly when the trend goes from positive to negative.
An example of MACD applied to Tubi’s streaming hours. The first plot shows the daily metric with additional green and red markers when the MACD signal goes from negative to positive and vice versa. The second plot contains the fast and slow trends, plus the MACD histogram.
Putting it all together…
We’ve covered two ways to detect outliers in our time series data: Matrix Profiling and MACD.
By combining the two approaches, we get a better picture of our data and can detect a wider range of anomalies. We display the results of the analyses on a dashboard that’s accessible to stakeholders. The dashboard provides updates on our KPIs and includes visualizations that help us to spot any abnormalities or outliers. By leveraging both Matrix Profiling and MACD in our anomaly detection process, we can proactively identify issues and take corrective actions before they become major problems.
Our new system has been up and running for a few weeks now, and we’re working on adding new features. As Tubi continues to grow, we expect to rely on this system, even more, to monitor the health of our business over time.
If you’re passionate about data science and interested in working on projects like this, let us know! We’re always looking for talented data scientists to join our Tubi Data Science Team. To find out more, check out Tubi’s careers page to see what positions are available.
Many thanks to Davide Totaro for his contribution to this project.
