Companies often struggle to answer the question “what is an acceptable level of forecast accuracy?” The search for a realistic answer can create angst due to the myriad of benchmarking data available coupled with the lack of information regarding the data collection procedures employed and the specific formulas used to calculate the results. Even if agreeable benchmarking statistics can be determined, company managers have a tendency to discount these data because they are not considered representative of business and channel characteristics within their respective industries.

In the search for appropriate benchmark statistics for forecast accuracy, it is prescribed that companies undertake an internal benchmarking effort. Such an analysis of company data can provide valuable insights for establishing an acceptable level of forecast accuracy that are not likely to be gained by using external forecast accuracy benchmarks. Moreover, it is hard to argue with internal benchmarks because they are based on actual historical company data. Analysis of company data also offers opportunities toward developing an understanding of product and customer demand patterns that can provide the basis for customer segmentation based on the value of the each segment to the firm and the apparent stability of that product/customer data.

A protocol which utilizes a decomposition approach for internal benchmarking was proposed during the February 2006 Forecasting Summit. This decomposition approach comprises the teasing out of seasonality, trend and level components, thereby leaving only noise. Assessing such noise, an approximation of stability for individual products/customers, is calculated by way of coefficient of variation (CV) and linked to an estimation of mean absolute percent error (MAPE). Data analyses would begin with examination of whether data is seasonal and/or untrended, which can be done in a spreadsheet format, following which the data would then be de-seasonalized and untrended accordingly. The remaining data stream would comprise level and noise components. Calculating the standard deviation and comparing it to the average of the level data stream provides the Coefficient of Variation (CV) and offers an approximate measure of stability; a CV approaching zero suggests a more stable data set. By employing this methodology, an analyst can categorize product/customer data in regards to stability, recognizing that those more stable data streams should be more forecastable than those less stable.

While not a panacea, this approach can help managers to better understand their data and segment it more appropriately in the course of applying forecasting techniques and planning resources to manage each data time series.

On a broader note, forecastability should be viewed as more than just forecast accuracy. Issues concerning cost and margin per product should be included alongside discussions about forecast accuracy. Forecast accuracy also should be viewed as an intermediate statistic in the course of attaining an overriding business objective. It is therefore critical to think about what the company wants to achieve as its ultimate objective for the forecasting endeavor. Goals such as customer service, customer satisfaction, and profitability are likely objectives in support of Sales and Operations Planning. And in supporting the Sales and Operations Planning process, forecasting plays a truly strategic role as a distinct process that underlies company decision-making.

For more on this topic, you may view the presentation given by Dr. Kahn at the Forecasting Summit conference.

About the Author
Dr. Kenneth B. Kahn is an Associate Professor of Marketing in the Department of Marketing and Logistics at the University of Tennessee. His teaching and research interests include product development, product management, demand forecasting and interdepartmental integration. Dr. Kahn is co-founding Director of the University of Tennessee’s Sales Forecasting Management Forum. He is the author of the upcoming book, New Product Forecasting: An Applied Approach (M.E. Sharpe, 2006). Dr. Kahn is a frequent contributor at the Forecasting Summit.  

Business Forecast Systems in cooperation with the International Institute of Forecasters