Trustworthy AI series, part two: explainability, uncertainty, and keeping humans in the loop 

The first Insight in this series explains why trust has become the gating factor in whether AI systems move beyond pilot environments into business-as-usual operation. Even with excellent technical performance, deployment stalls if organisations cannot show control or accountability in real-world conditions. 

This second Insight explores what operational trust requires in practice. Trust comes from visibility. People who operate or govern systems need to understand how AI arrives at recommendations, and how these combine with human insight to enhance decision-making. 

Explainability and uncertainty quantification are central to this. They are the difference between AI remaining an opaque black box or becoming a decision-support tool that people can interrogate, challenge, and rely on in critical operations. 

Trust only endures when those who use and govern systems understand their behaviour, especially under operational conditions. Headline accuracy scores are not enough on their own. In safety-critical environments, confidence comes from being able to explain how recommendations are reached and how these support human judgement under pressure.  Being able to explain the robustness of an output to changing inputs, as well as the associated uncertainty, is also vital. 

The distinction between accuracy and explainability is key for senior decision-makers. If AI-informed decisions cannot be clearly explained, they are difficult to justify to regulators, boards, or frontline operators. 

Time also matters. Many operators need to justify decisions in real time. Without structured explainability and clear representation of uncertainty, AI outputs are difficult to integrate into live decision-making, regardless of how strong the underlying model is. 

Explainability allows stakeholders to see and interrogate how model decisions are formed. Techniques include feature attribution, sensitivity analysis, and interpretable surrogate modelling – simplified explanatory models that approximate the behaviour of more complex systems. These reveal which inputs drive outputs and how outputs change under different conditions. 

This provides validation and assurance. When engineers, operators, and leaders can explore how recommendations are produced, they build a working understanding of where a model is reliable and where caution is required. Explainability becomes a way of calibrating trust, not simply a compliance exercise. 

This framing prevents explainability being viewed as a silver bullet. Transparency does not remove uncertainty or risk. Instead, it makes them visible and manageable.

Explainability and uncertainty quantification help make model behaviour visible. However, it is human oversight determines how that information is used. 

In high-consequence environments, AI systems can inform judgement, but accountability and final decisions must remain with human experts. Operators should be able to challenge outputs, override recommendations, and apply wider contextual knowledge that only humans can have. 

Keeping humans in the loop ensures that decisions are trustworthy and remain defensible when conditions change, when outputs are unexpected, or when consequences extend beyond the model parameters. 

The National Energy System Operator illustrates explainability and uncertainty working together in practice. In reserve-setting and system-balancing operations, decisions are made under time pressure with direct implications for cost, resilience, and supply security. 

In our work with NESO, Smith Institute embedded explainable forecasting and optimisation models directly into operational workflows. These models provided recommendations alongside structured insights on how those recommendations were derived. Uncertainty is quantified and presented with each output, allowing operators to understand risk margins in different scenarios. 

This changed how these tools are used. Operators can interrogate forecasts, explore sensitivities, and see how outputs respond to different assumptions. The models are treated as decision-support with operators retaining control of the final decisions. Trust comes through measurable evidence that can be explored, questioned, and defended in real time. 

Explainability and uncertainty quantification create the conditions for trust, but they do not guarantee it on their own.

In the next insight, we consider the assurance mechanisms that allow trust to be sustained over time, including governance structures and managing model drift as live systems evolve.