How the pandemic challenges companies that use predictive models

For enterprises using predictive models to forecast consumer behavior, data drift was a major challenge in 2020 due to never-before-seen circumstances related to the pandemic. Organizations were forced to constantly retrain and update their machine learning models, and 12 months later, many are still wrestling with the challenge.

In an interview with VentureBeat, Dan Simion, VP of AI & Analytics for Capgemini North America, said that while companies are in a better position than they were three months into the pandemic, they’re in a different position. While they’re acclimating to the data coming in within the context of this new environment, they can’t draw patterns from the past 12 months of data because behaviors continue to change.

“In the first three months, everyone was more or less trying to make sense of what could be done to use the new data and start building more accurate machine learning models,” Simion said. “Today, the question is: How quickly can we adapt and retrain machine learning models?”

Simion pointed out that models need to be nimble enough to increase accuracy by leveraging new data on the changing behaviors as it comes in. It’s also critical to establish a way to scale this process, he said, because changing machine learning models and adapting to continuously shifting data takes a coordinated effort.

As an example, Simion talked about a multibillion-dollar global consumer packaged goods company in the frozen food sector. Early on in the pandemic, the company, which is a Capgemini customer, had to adjust to trends and behaviors that varied widely depending on specific regions and states. In the first three to four months of the pandemic, when most regions had restrictions in place, frozen food sales went up significantly as customers chose to eat at home. But while some states have since loosened their quarantine rules and the number of frozen food sales has decreased overall, other states have opted for a slower reopening, leading to shifting trends that make it difficult to predict where frozen food sales will ultimately settle.

In another example, Simion says that a Capgemini client in the industrial components space is struggling to anticipate disruptions in the global supply chain. Because of international restrictions and limits, there aren’t many ways to deliver materials and products across countries. The company had emergency reserve supplies stockpiled in warehouses at the three-to-four-month mark of the pandemic, but with that emergency supply gone, limited transportation and supply have made it difficult to build accurate forecasts amid so many variables and constraints.

Simion says the challenges are particularly acute in commerce. One Fortune 500 retailer retaining Capgemini’s services can no longer track and predict certain buying patterns with the precision it did before the pandemic. In a normal year, during Christmas or approaching Back to School season, shoppers would make purchases, and orders of particular items would increase predictably. But that has changed as varying pandemic constraints, including hybrid learning environments and smaller holiday gatherings, impact people and their spending.

“The supply chain was built in a way that would fulfill steady demand. Planning and forecasting based on that prior data was easy and highly accurate,” Simion said. “Now, all of that is changing. [E]ven after 12 months of living in this pandemic-impacted world, we cannot grasp what an accurate predictive model will be.”

A report recently published in Harvard Business Review suggests several remedies for unstable predictive models. To fix these models, companies might look to analogies like past economic shocks for an idea of the future during and after the pandemic. They might also embrace ensemble modeling, which combines predictions from different models to suggest a reasonable range. And they could include local knowledge, as well as aggregated knowledge from a panel of experts on the pandemic and its effects.

“The question is: How quickly can we adapt and retrain ML models? Not only do the models need to be rebuilt or redesigned based on new data, but they also need the right processes to be put into production at a pace that keeps up,” Simion added. “Until there is some sort of stability, it will continue to be difficult for organizations to identify consistent trends.”