Probabilistic demand forecasting

Probabilistic demand forecasts are a must-have whenever it comes to inventory optimization. Demand forecasts are said to be integrated over the lead time, when forecast values match the total demand over the lead time, as opposed to a classic forecasting perspective, where forecasts are periodic (typically per day, per week or per month) and agnostic of the lead times. Lokad’s forecasting engine delivers integrated probabilistic demand forecasts that take probabilistic lead times as an input. Lokad’s forecasting engine natively supports the various statistical patterns found in business data such as seasonality, trends and product lifecycles. Distortion of demand such as stock-outs or promotions is also taken into account. Lokad’s forecasts represent the expected probabilities for every single unit of future demand. In this section, we detail the syntax used for computing Lokad’s integrated demand forecasts.

General syntax

The forecasting engine has a function dedicated to probabilistic forecasts. The syntax is the following:

T.Demand = forecast.demand(
  category: T.C1, T.C2, T.C3, T.C4
  hierarchy: T.H1, T.H2, T.H3, T.H4
  label: T.PlainText
  demandStartDate: T.LaunchDate
  demandEndDate: T.EndDate
  horizon: T.Leadtime
  offset: 0
  present: (max(Orders.Date) by 1) + 1
  demandDate: Orders.Date
  demandValue: Orders.Quantity
  // 'SO' for stock-outs
  censoredDemandDate: SO.StartDate, SO.EndDate
  inflatedDemandDate: TVAds.StartDate, TVAds.EndDate
  // 'Promos' for promotions
  promotionDate: Promos.StartDate, Promos.EndDate
  promotionDiscount: Promos.Discount
  promotionCategory: Promos.Type)

Unlike regular functions, call functions have named arguments instead of positional arguments. These named arguments are more suitable for complex functions, because they make the source code much more readable - at the expense of limited extra verbosity. These arguments behave just like regular function arguments, thus they are permitted for Envision expressions.

The function returns a vector Demand that is of type ranvar. Ranvars are an advanced data type that represents functions $p: \mathbb{Z} \to \mathbb{R}$. More specifically, the forecasting engine returns ranvars, that is, distributions that are non-negative and have a mass equal to 1. In the present case, $p(k)$ represents the probability associated with a demand of $k$ units. Each item - in the Envision sense - becomes associated with its own demand ranvar.

The full forecast.demand syntax includes many arguments, however, only four of them are mandatory:

• present: a scalar date value
• demandDate: a date vector with an item affinity
• demandValue: a number vector with an item affinity
• horizon: a ranvar

The present value is the date intended as the first day to be forecast, following the assumption that data is complete up to the day before. Some businesses may be closed on Sundays for example, and if the most recent date found in the dataset is a Saturday, there is an ambiguity as to whether the forecast should start on Sunday or Monday. In the illustrative syntax above, we use max(Orders.Date) + 1, assuming that orders are observed every day, and that the input data is fresh from the day before.

The demandDate and demandValue are expected to belong to the same table which exhibits an item affinity, that is expect [Id] in Envision terminology. The dates demonstrate when the demand was observed in the past. The values represent the scale of the demand - which is typically counted in units or eaches. Fractional demand values are not supported. This table contains the demand history being forecast by the forecasting engine. Ideally, the history length should be made as long as possible, although in practice there are limited benefits to exceeding 5 years’ worth of demand. The forecasting engine accommodates both short and long demand history alike, when this history is long, older data points simply fade into statistical irrelevance.

The horizon represents the probabilistic lead time to be used when forecasting the demand. While the lead time is treated as an input when computing an integrated demand forecast, the lead time is typically also a forecast in itself. The forecasting engine offers the possibility to forecast lead times. The lead time forecast is decoupled from the demand forecast itself, because this offers the possibility to perform ad-hoc adjustments on the lead time ranvar, before feeding them into the forecasting engine.

Beyond these mandatory arguments, the accuracy of the forecasts can be greatly improved by providing more data to the forecasting engine. The following sections explain this in more detail.

Formal definition

In this section, we briefly detail the formal definition of the statistical operation performed by the forecasting engine when computing an integrated demand forecast.

Let $d(t)$ be the demand function at the day $t$. Let the integrated demand $D$, associated with the random variable $\Lambda$ representing the lead times, be defined as follows:

$$D : (d,\Lambda,t_1) \mapsto \sum_{j=1}^\infty {\mathbf{P}[\Lambda=j]} \left( \sum_{i=1}^{j} d(t_i) \right)$$

where $\mathbf{P}[\Lambda=j]$ represents the probability for the lead time random variable $\Lambda$ to be equal to $j$. The demand is qualified as integrated, because it is summed over a probabilistic lead time.

If $t_0$ represents the present date, then the demand is known - because observed - until the day $t_0$ but unknown afterwards. The purpose of the forecasting engine is to compute $\hat{D}(d, \Lambda,t_1)$, a probabilistic estimate of this future demand $D(d, \Lambda,t_1)$, expressed as a probability distribution (ranvar) of all possible demand scenarios.

Categories, hierarchy and label

Categories, hierarchy and plain-text labels play a very similar role from the forecasting engine perspective: they help the forecasting engine cope with sparse historical data.

See Forecasting with categories and a hierarchy.

New product forecasting

From a forecasting perspective, a new product is a product that has not yet been sold. This represents a rather specific forecasting challenge because, by definition, there is no historical data associated with this new product. Our forecasting engine supports new product forecasting through the demandStartDate argument. When historical start dates are known, it is advised to provide this information to the forecasting engine as it contributes to improving the forecasting accuracy, for both new and old products.

The demandStartDate argument expects a date to be provided for each item. This date is intended to represent the first day when the demand becomes effective for this item. This date is in the past for items that have already been sold, and it remains in the future for items that are yet to be launched.

There are two distinct benefits to providing the demandStartDate argument. Obviously, the first benefit consists of forecasting new items. In this case, it is usually also important to specify the offset argument. Indeed, if the offset is kept at zero - its default value - then the period covered by the forecast might not overlap the active period for the item.

Example. Today is July 1st. The forecasting horizon is a Dirac delta ranvar at 7 days; that is a constant lead time of 7 days. Product A is launched on July 15th - its start date. If the forecast is carried out today, then the forecast probability distribution for Product A is a Dirac delta ranvar at zero because, the horizon ends prior to the start date of Product A. In order to forecast the first week of demand for Product A, the offset for Product A should be set at 14 days.

The second benefit of specifying the demandStartDate is to increase the forecasting accuracy for all items, and not just the items that are yet to be launched. Observing the first unit sold on the start date for a given item is not the same thing as observing the first unit sold six months after its launch. While the former case hints at steady upcoming sales, the latter hints at a very limited demand of only a handful units per year. The forecasting engine leverages the demandStartDate argument to refine the demand forecasts for all the items.

Censored and inflated demand

The intent is to forecast the demand. Yet, frequently, historical data only approximates the real demand, thus creating distortions (willingly or not). For example, historical data might be represented by historical sales. However, in the case of stock-outs, sales volumes drop while the demand itself might remain steady. Lokad’s forecasting engine is natively designed to take care of these distortions, and this is the very purpose of both the censoredDemandDate and inflatedDemandDate arguments. Both arguments expect a date vector of item affinity, that is (Id, Date) in Envision terminology.

When a date for a given item is marked as censored through the censoredDemandDate argument, the forecasting engine will assume that the demand is higher or equal to the observed value. The engine makes no assumptions on how high the demand would have been on this particular day as this value cannot ever be known to us. Yet, by pinpointing the bias, the engine can roll out entire classes of optimization tailored around this case. In practice, the most common occurrence for censored demand are stock-outs as observed through sales data that does not capture information on prospects who silently walk away while goods are missing.

Similarly, demand can be inflated. The inflatedDemandDate argument offers the possibility to pinpoint the dates and the items where the demand should be considered as lower or equal to the observed demand. Again, the real demand remains unknowable, but pinpointing the bias is already very helpful to the forecasting engine. In practice, demand is inflated when there are temporary non-recurrent market boosts: for example, the exceptional victory of a local sports team in a national championship may impact very favorably on the sales of local supermarkets for a few days.

The two arguments inflatedDemandDate and censoredDemandDate can take one or two vectors as input. If two date vectors are provided, then, the pairs (start, end) are treated as inclusive segments, with the first date being the start of the segment, and the second date being the end of the segment. If only one date vector is provided, then, segments are considered to be 1-day long; the dates flag the exact days to be considered as inflated or censored.

If demand censorship or inflation are recurrent - per year, per week, etc. - then, there is no need to mark the demand as such, as the forecasting engine handles such patterns automatically.

Forecasting promotions

The forecasting engine offers a native support for promotions. Providing data about promotions is optional. However, when promotions data is provided, promotions are expected to be specified both in the past and in the future. At the very least, the argument promotionDate can be provided alone. The argument promotionDate follows the same usage pattern as censoredDemandDate: when a single date vector is provided, promotional periods are considered to be 1-day long, if two dates are provided, the first vector represents inclusive start dates, while the second represents inclusive end dates.

The promotionDiscount argument is optional, and can be provided in order to help the forecasting engine gain insights about the intensity of a given promotion. A number vector is expected for this argument, and the forecasting engine treats this data as ordinal values: the greater the discount, the greater the expected promotional impact. In practice, it is the forecasting engine that computes the expected demand uplift based on uplifts observed for past promotions.

The promotionCategory argument is also optional, and can be provided as a classification of promotional events. When provided, this argument is leveraged by the forecasting engine to test the affinities between promotional events, and to detect whether events marked within the same category achieve similar demand uplifts. This argument is very similar in spirit to the category argument, except that it is applied to promotions instead of being applied to items.

Caveat lector. Promotions are notoriously difficult to forecast even with excellent historical data. Lokad’s experience indicates that most companies do not have high accuracy promotional data readily available. That being said, such data can be obtained through careful preparation at the later stages of a project. As rule of thumb, gathering promotional data that is good enough to actually improve a forecast’s accuracy requires significant amounts of effort. Feeding the forecasting engine with approximate promotional data only decreases the resulting accuracy.

When promotions data is provided, the periods relating to promotional activity should typically not be flagged through the argument inflatedDemandDate. Flagging a period through both arguments promotionDate and inflatedDemandDate has a subtle semantic: it indicates that the promotional uplift has been inflated beyond what would be reasonably expected from a promotion, and the promotion itself would be considered as biased.