11  Self study - Session 2

In this session, we will revisit Chapter 5 and Chapter 6.

In Chapter 5, we have explored different methods to handle missing data in time series and have seen that the choice of imputation method can significantly affect the results.

The data set we will rely on in this session is the synthetic time series from the previous section which simulates weekly production data of a manufacturing plant over 20 years. Unfortunately, some data points are missing due to frequent outages in the data collection system and sometimes we observe huge outliers due to temporary errors in the counting process.

Your task is to impute the missing values and handle the outliers appropriately to prepare the data for a 20 years jubilee management report.

Again, you can download the data set from here.

Additionally, here are some useful helper functions:

import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import pandas as pd

def plot_production_data(
    df,
    orig_data_column,
    gap_column,
    imp_column=None,
):
    """
    Plots production data with and without gaps using seaborn.
    Parameters:
      df (DataFrame): DataFrame containing the production data.
      orig_data_column (str): Column name for the original data.
      gap_column (str): Column name for the data with gaps.
      imp_column (str): Column name for the imputed data.
    """
    fig, axes = plt.subplots(3 if imp_column else 2, 1, figsize=(12, 10))

    # Original data
    sns.lineplot(x=df.index, y=df[orig_data_column], ax=axes[0], color="grey", label="Original")
    axes[0].set_title("Original Data")
    axes[0].set_ylabel("pieces")

    # With gaps
    sns.lineplot(x=df.index, y=df[gap_column], ax=axes[1], color="black", label="With Gaps")
    axes[1].set_title("Data with Gaps")
    axes[1].set_ylabel("pieces")

    if imp_column is not None:
        dff = df[[gap_column, imp_column]].copy()
        dff.loc[~df[gap_column].isna(), imp_column] = None
        sns.lineplot(x=dff.index, y=dff[imp_column], ax=axes[2], color="blue", label="Imputed")
        axes[2].set_title("Imputed Data")
        axes[2].set_ylabel("pieces")

    plt.tight_layout()
    plt.show()


def seasonal_decompose_seaborn(decomposed_ts, title):
    """
    Plots a decomposed time series using seaborn.
    Parameters:
      decomposed_ts (DecomposeResult): Result of STL or seasonal_decompose.
      title (str): Title for the plot.
    """
    fig, axes = plt.subplots(4, 1, figsize=(12, 10))
    sns.lineplot(x=decomposed_ts.observed.index, y=decomposed_ts.observed.values, ax=axes[0])
    axes[0].set_title("Observed")
    axes[0].set_ylabel("pieces")

    sns.lineplot(x=decomposed_ts.trend.index, y=decomposed_ts.trend.values, ax=axes[1])
    axes[1].set_title("Trend")
    axes[1].set_ylabel("pieces")

    sns.lineplot(x=decomposed_ts.seasonal.index, y=decomposed_ts.seasonal.values, ax=axes[2])
    axes[2].set_title("Seasonal")
    axes[2].set_ylabel("pieces")

    sns.lineplot(x=decomposed_ts.resid.index, y=decomposed_ts.resid.values, ax=axes[3])
    axes[3].set_title("Residual")
    axes[3].set_ylabel("pieces")

    plt.suptitle(title)
    plt.tight_layout()
    plt.show()

Exercise 11.1 (Load the data with gaps and get rid of outliers)  

  • Download and load the dataset. It has three columns: Date, pieces and pieces_gaps. The column pieces contains the full data set without gaps, while pieces_gaps contains the same data set with several missing values (NaNs). The period covered by the data set is from 2005-01-01 to 2024-12-31 in weekly frequency.
  • Visualize both time series (pieces and pieces_gaps) using the provided plot_production_data function.
  • Smooth the pieces_gaps time series using a rolling window with appropriate window size and decide whether it is better to use mean or median as aggregation function. Discuss your choice. Ideally the resulting smoothed time series should have the same gaps as the original pieces_gaps time series.
  • Visualize the resulting smoothed time series using seaborn.

Let us try to impute the missing values in a second step.

Exercise 11.2 (Impute the missing values with known methods)  

  • Use the smoothed time series with gaps from the previous exercise as input.
  • Experiment with different imputation methods discussed in Chapter 5.
  • Visualize the results using the provided plot_production_data function.
  • Discuss the results.

Of course, we have only seen the tip of the iceberg when it comes to imputation methods for time series. Especially for seasonal data, more advanced methods such as STL can be very effective. We have had a look at STL in Chapter 6, where we decomposed a time series into its components. Now we will use it for imputation to not lose the seasonal structure of the data while filling the gaps.

Exercise 11.3 (Impute the missing values with STL)  

  • Use the smoothed time series with gaps from the first exercise as input.
  • The STL needs data that is complete, i.e., without missing values. Therefore, first impute the missing values using a simple method of your choice (e.g., linear interpolation).
  • Then, apply the STL decomposition to the imputed time series.
  • Visualize the decomposed components (trend, seasonal, residual) using the provided seasonal_decompose_seaborn function.
  • The seasonal part of the decomposition can now be used to deseasonalize the smoothed time series with gaps.
  • Now that we have a deseasonalized time series with missing values, we can again impute the missing values using a simple method of your choice.
  • Finally, reseasonalize the imputed deseasonalized time series by adding back the seasonal component obtained from the STL decomposition.
  • Visualize the final imputed time series using the provided plot_production_data function.
  • Discuss the results.

Great job, you have provided the graphs in time for the management report and the board liked the results. In the meanwhile your colleagues have found the original data set without gaps on some old dusty server and would like to know the number of outliers in the data set.

Exercise 11.4 (Count the number of outliers in the original data set)  

  • Use the original data set without gaps (pieces column) as input.
  • Use an appropriate method to identify the outliers in the data set.
  • Visualize the outliers and the original data set using seaborn.
  • Count the number of detected outliers.
  • Discuss the results.