Fixing the Models that COVID Broke
Predictive models rely on the principle that the patterns and behaviours of the past will most likely repeat in the future. The more stable and repeatable the pattern, the better.
If every year the sale of hotdogs and burgers go up in August, then a model will likely advise a supermarket to stock extra hotdogs this summer. So, what happens when in 2020, to paraphrase Yeats; “All changed, changed utterly: A terrible beauty is born.”
The pandemic has completely changed the way we live and work. Lockdowns, travel bans, the complete shutdown of certain sectors of business, physical distancing, widespread furloughs … all of these things have changed how we shop, where we work from, how and if we travel, how we behave and all of that translates to changes in patterns in data, rendering many pre-COVID-19 models fragile, if not completely useless.
Anyone working with models won’t be surprised to find that shifts in the data feeding into models built before the crisis has sent some models off the deep end. Models are expected to degrade over time – no model lasts forever, and models regularly need re-building as shifts in the market, environment or customer behaviour happens.
With COVID-19, this has just happened more quickly and more radically than before. Before throwing out all models/analytics and wringing your hands in despair, we’ve put together a few tips below on how you can ‘fix’ the models COVID-19 broke.
- Evaluate, triage, plan, act: To determine what needs to be prioritised a business might look at how a model is used, whether a model is material in key tasks such as building financial reports, to what degree COVID-19 has had an impact (not all models will be affected equally) and how direct that impact is on critical business. Other considerations include regulatory focus or the potential of upstream models to “contaminate” other critical models. By performing model-triage you can focus limited analytics resources on the most critical models and focus attention on where it’s most needed and will give highest returns. As with any analytical work, and especially during these times, it’s vital the business is involved in these decisions as they are closely linked to potential shifts in business strategy.
- Identify if a shift/new trend is temporary and likely to die out post-COVID-19 (such as a surge in demand for flour as people took up baking during lockdown) or likely to be more long-lasting or ‘the new normal’ (such as customers move to online shopping). This will shape your business strategies as well as your approach to analytics.
- If a model is not vital to business activities during these times, and is performing poorly, consider switching it off/pausing until the situation stabilises.
- Override or switch-off auto-replenish if a model automatically orders stock and predictions are wildly off. Return to manual orders for the time being.
- There are various approaches you can take to improving model performance such as: try tuning a models’ parameters, re-train the model on more recent data or limit data to only post-COVID-19 time periods.
- Investigate using a different model – SPSS has an Expert Modeler function that allows the tool to choose the optimal time series model based on the data. Other tools such as IBM Modeller have a function to auto-select the best algorithm.
- Identify future demand drivers and include them in the models. This could mean including disease and lockdown information in the model – separate forecasts can be created, and the outputs added to existing models, or use past case numbers/lockdown events as features in the model.
- Revisit the time period for predictions: if a model has been forecasting for longer time periods ie on a quarterly basis, it might be useful to build another model that forecast on a weekly or daily basis. Short term forecasting can allow more rapid reaction to changing events.
- Re-evaluate the importance of your model inputs: retail has seen a massive shift to online shopping, if a model previously relied on footfall figures, including new/previously unused digital data sources such as mobile-app usage could be more useful
- Beware one-size-fits-all modelling: whereas previously a model may have been used to forecast at a group level ie sales for Europe, considerable differences now exist between countries based on varying government reactions/strategies, the impact of the disease on the population in each area etc – consider that a more regional or granular approach may now be needed.
- Promote agile iterative analytics: sometimes creating a quick solution, test small, fine-tune later can help a company react quickly especially in volatile times. A simple moving average of the sales of the past three days might be enough to allow a rapid response to a shock event that would otherwise not have been caught while waiting for a full re-build of a more complicated predictive model. Having a strong analytics department that is closely tied to the business will allow your company to keep track and respond in an uncertain world.
- Check assumptions made during model build/data engineering steps: we’ve seen unprecedented events this year such as the collapse of the airline, travel and entertainment sectors. If your model relies on certain assumptions such as a minimum number of flights per day, or that crude oil prices would never drop below zero dollars a barrel, and doesn’t allow for the alternatives, that can have tremendous implications for model predictions.
- Widen where you apply analytics: As companies moved to providing services to consumers online, and their workforce moved to remote-working, this has had a massive impact on many companies internal IT infrastructure, with many companies worrying about business continuity. Issues faced were companies realising their systems couldn’t cope with the increase in web traffic, their systems weren’t set up for remote-working or simply they didn’t have enough stock such as laptops for workers. Analytics can be employed to ensure business continuity. Various models such as time series, clustering models and anomaly detection models can be used to model network usage – allowing your IT department scale up systems (easily done if you are already using cloud based technology such as Azure), they can flag anomalous and potentially threatening activity that could result in downtime to your IT systems, even traditional t-tests/ANOVA can be used to analyse worker-satisfaction survey results to monitor how workers are coping during these times.
Being able to gather and illustrate data accurately and communicate in a way that everyone can understand and react to is down to the tool that is deployed, and this is where data analysis software such as SPSS can deliver real business benefit.
Version 1’s experienced consultants are on hand to help you find the best software and license type for your analytical and usage requirements. Contact us to discuss your requirement and identify the best SPSS product for you.