Can we trust machine learning for decision-making?

• As AI and machine learning gain wider adoption, the issue of trust becomes more important
• Even big tech firms have had problems with AI going wrong
• Certification and quality control systems are being set up and may help build trust

As artificial intelligence (AI) and machine learning (ML) move out of the lab and into real-world business applications, companies are asking “can we trust AI and ML for decision-making?”

It’s a question that’s important not just for large businesses. The government’s AI Activity in UK businesses study in January 2022 showed that even among small firms, 15% had adopted at least one AI technology. The government data suggests that every working day 200 small- and mid-sized firms invest in their first AI application.

AI is being used in a wide range of applications including data analysis (such as demand forecasting), processing natural language (deployed in customer-facing roles such as web chatbots), and computer vision (often used in quality control).

Albert King, Chief Data Officer for Scotland, said in March: “Being trustworthy, ethical [and] inclusive are actually necessary conditions for comfortable adoption of AI technology.”

AI has new types of failure

Computer says ‘no’

The issue of quality control is particularly acute for some types of ML, such as image recognition, that use the digital equivalent of “gut instinct” and can’t provide a human-understandable set of rules that explain their decision.

This is already a problem for recruitment or credit scoring. Under Article 15 of GDPR, anyone who has an important decision taken about them by an algorithm has a right to “meaningful information about the logic involved”.

Big firms employ data scientists to verify the large-scale AI applications they need. But this is unreasonable for SMEs buying off-the-shelf or semi-custom AI-powered projects, and potentially reduces their willingness to try this new technology.

Building trust in machine learning

Testing and quality assurance of AI won’t be compulsory. But even a voluntary system can have impact. Euro NCAP’s programme is voluntary too, but cars that score five stars get a sales boost and those that score poorly have even been withdrawn from sale.

Some jurisdictions are going further and potentially forcing firms to ensure their AI is trustworthy or risk penalties. The EU, for instance, has proposed a legal framework for AI that sets tight requirements for applications deemed high risk. This doesn’t just include obvious categories such as medical devices and power stations, but applications that could infringe personal rights such as recruitment, systems that evaluate workplace performance, and credit scoring.