Why independent verification is essential to fill in the cracks
Algorithms are fast becoming ubiquitous in replacing manual operations. But are your algorithms doing what you claim they are doing? And if companies believe they have checked over an algorithm, have they done the best they could to verify it? Or have they only seen what they wanted to see?
The Smith Institute has recently been awarded a Queen’s Award for International Trade for its work providing independent quality assurance of high-value auction algorithms in the telecoms industry. But even if you’re not in the telecoms industry, your business, service or regulatory board can likely benefit from the same expertise too.
Ensuring market competition through algorithms
If you’re a mobile service provider, you need radio spectrum like an oil company needs oil. But while spectrum bands are finite and limited by geography and national boundaries, spectrum is essentially endlessly usable. Regulators in each country auction off licenses to use different spectrum bands to mobile providers, and this is a highly complex process.
Auctions involve many overlapping and often conflicting interests, with potentially many thousands of licenses and billions of pounds being spent. There needs to be a completely reliable algorithm to allocate the winners and prices fairly – and to ensure everything is above board.
This is an important issue for telecoms regulators. For them, the auctions aren’t about raising as much money as they can. Not if it means one company leaves with a monopoly on spectrum. The regulators need to act in the consumer’s best interests, encouraging a competitive market without increasing end prices.
Finding ambiguities in the algorithm
For spectrum auctions, the rules of how winners and prices will be calculated are decided in advance and published. But sometimes the rules are ambiguous, a little like board game rules that don’t cover all scenarios and leave players guessing and creating their own interpretations. When this interpretation is left to an algorithm in a highly regulated industry, it can lead to errors and discrepancies that run too high a cost, with both financial and reputational risk.
Over the last 14 years the Smith Institute has provided quality assurance of algorithms for 30 spectrum auctions in 10 countries. Completely independent from the various software providers who built the electronic bidding software, we implemented our own parallel algorithms with the same sets of rules, as laid out by the regulator. We also built a testing robot to explore all the pathways through the algorithms, exposing any areas in which they might go wrong.
On occasion we’ve gone back to the regulators and told them where they need more clarity in their rules. This in turn has influenced the design of their algorithms so they can account for situations in which there are situations that carry the greatest risk of system failure. The risk is mitigated and we’re able to run the tests again to verify the fact.
Proving your algorithm is certified
It’s essential for an algorithm to do what you say it does. This is true even when the cost of being wrong is minor, provided that cost is paid on a regular enough basis. A leak isn’t as dramatic as a dam burst, but it can lose you just as much water over time.
For instance, when it comes to electricity in the energy sector, the results of balancing supply and demand are, at one level, easy to see: either the lights stay on or they don’t. But the process behind the decision making is very complex and the potential for waste is present throughout the balancing act.
Balancing real-time supply and demand is also vital in on-demand services such as Amazon, Uber and Deliveroo. Effective algorithms can make the difference between having the competitive edge or not. If these optimisation algorithms aren’t achieving what they’re meant to do, it can lead to a loss of custom and a negative customer sentiment.
If optimisation algorithms provide your business with real-time decisions or decision support, independent verification can determine whether your decisions are as effective as you believe them to be. If you need an improved mathematical model, the Smith Institute can take you on that journey, along with suggestions for algorithm design and implementation.
The cost of being wrong
Who tells you that your system operates optimally? When there’s a vast number of inputs contributing to a high-level outcome, it’s hard to determine where the outcome – billions of dollars, the security of vital infrastructure, or the reputation of a company – could be at risk. But that risk needs to be uncovered and managed so it can be mitigated. And for that, you want independent verification of your systems and algorithms.
Independent review is especially valuable if there’s a regulator who will hold you to account. It’s an assurance to your business, your clients, customers and other stakeholders that you have done everything in your power to provide the optimal service you intended.
Independent verification can also future proof against potential new regulations. There is legislation being put forward, such as the US’s Algorithm Accountability Act, that require organisations to be transparent about what their algorithms do. If you aren’t certain what your algorithms are doing, you’ll find it difficult to comply with the rules.
Get independent experts on board
What’s the risk appetite of your organisation? Is business-as-usual good enough? The Smith Institute can find the strengths and weaknesses in your algorithm so you can optimise it. We can also derisk it, taking the potential problems away from your desk, and ensuring you and your customers can have confidence in your systems.
We don’t have any preconceptions of what your algorithm should do, so we’re able to ask the difficult questions and find out what it’s actually doing. If you’d like our experts to take a look at your system and figure out what needs to be investigated, simply get in touch.
