Building anti-fragile technology systems

The Covid-19 outbreak which resulted in the government-mandated shutdown has exposed a lot of weakness in the technological systems we use to manage our day to day life. Apart from the negative effects on the economy and the loss of life and livelihood, Covid-19 is a wake-up call for all the people who build and manage large systems and especially those of us building or thinking of building future AI systems, we have to build anti-fragile systems.



A lot of systems that depend on the most sophisticated forecasting technologies of our times broke down when they faced the unexpected reaction from the pandemic. Some of them are listed below

1. Financial anti-fraud systems broke because consumers changed their behaviour. For example, credit card companies often flag a card as possibly stolen if the purchase pattern associated with it suddenly changes. But this rule of thumb doesn’t work well when huge swaths of society start working from home and stop going to restaurants and malls.
2. Logistics models used to predict supply and demand broke when manufacturers, shippers, and consumers changed their behaviour. Trained on last year's data, a model that predicts 1,000 widgets arriving on time next month can’t be trusted anymore.
3. Online services receiving a new surge or plunge in users are rethinking their demand estimation models since earlier models no longer are accurate. source: deeplearning.ai

What does this teach us? The natural reaction would be that we build systems that are more robust or disaster resilient. But what does robustness or disaster resiliency really mean? In the world of AI, it could mean getting more and more data to train models that can handle larger swaths of unexpected input, or in traditional systems engineering, it might be adding more redundancy, building systems based on more flexible constraints on input, etc.

While all these traditional approaches to designing robust and disaster-resilient systems are all well and good, we should be expanding the fundamental assumptions about what it means to have intelligent systems from systems that just recognize patterns from input data to systems that modify their own internal designs to cope adaptively with more chaos in the environment.

Nassim Taleb the father of the philosophy of Anti-fragility defines an anti-fragile system as one that benefits from disorder rather than one that abhors it.

As engineers charged with the responsibility of building future AI, our goal should not be to focus on systems that we can only improve incrementally in a gradual iterative fashion by adding more data and then even more because no matter how much data we get it cannot really protect us from black swan-like events except we build insurance against these kinds of events directly into the root of the systems. Our algorithms should not abhor uncertainty but rather embrace uncertainty and radically self-improve from it.

Intelligence must include adaptation to uncertainty besides pattern recognition, and these should be the basis of systems not just in AI but every other critical large human-technology system from supply chains to medicine, etc.