15 things I have learned launching AI projects - Part 4

Looking forward to better AI adoption

I’ll close this blog series with 2 reflections on what to do next; because AI adoption, regardless of bubbles and hype, won’t end today. This is how you get and stay ready: adapt, adapt, adapt.

14: AI is creating arms races, sometimes the answer is to redesign the game

One thing I’ve seen repeatedly is that AI doesn’t just change how you do something; it changes the conditions around the thing you’re doing. And when those conditions shift fast enough, you end up in an arms race: one side adopts AI, the other has to respond, and suddenly the old way of doing things doesn’t work for anyone. Recruitment is a good example.

There’s a lot of discussion about using AI in recruitment. If you’d asked me 5 years ago, I’d have said this was an area of interesting research, not one to immediately apply. In fact, in the NHS AI Lab, we did explore the idea. We framed the project as a way to assess bias in shortlisting, while also testing algorithms to shortlist automatically. Ultimately, though, we decided not to proceed: too many ethical headaches, few metrics (a classic question: “how do you know you’ve selected the right candidate?”), and most importantly a problem that wasn’t felt sufficiently impactful; after all, in those times we’d consider ourselves lucky if we got more than a handful of applications for certain roles.

Then things changed, and changed my view of the constraints.

What changed is that with LLMs becoming widely available, candidates have started using them to drive applications. As a result, we’ve been receiving ten times as many applications as we would do before. In a recent campaign that I ran for my previous job at the Department for Work and Pensions, for example, I had 350 – you’ve read it right: THREE HUNDRED AND FIFTY – applicants for a single Head of Product Management role. That makes traditional sifting close to impossible. Or, rather than impossible, at risk of being low quality, because the sifters will get distracted by the noise within those applications. With organisations being overwhelmed, there is a question about how to respond to this use of AI. And whether we like it or not, it is an arms race. So my view has had to pragmatically change: if we can’t keep sifting the way we were, we’ll have to find ways to use AI to respond to AI.

But here’s the thing about arms races: there’s a law of diminishing returns. If you read this blog, you’re probably aware of the Moneyball story, the story of successfully applying data analytics to baseball. The first team to use data analytics was massively successful and gained a huge competitive advantage. The others who came after did the same, and out of that competition, the playing field levelled.

My prediction is that the same will happen with AI in recruitment, and in many other areas. But levelling the playing field isn’t enough. Sometimes the right response isn’t to optimise within the existing rules, it’s to change the rules altogether.

I went to a conference about data and AI in sports where Mihir Bose, former BBC sports correspondent, told an interesting story about how his team lost an important match to a wrong offside call by the referee. What he followed that up with is rather apt: the offside rule, he said, had been designed in a time where there was no video recording, no replay, no VAR. It is a rule that dates from an age where the word of the referee was the final word, and there was no way to question, challenge, or even verify if they were right. Technology has changed the context in which that rule operates. And that context has made its application more controversial, more contentious, more antagonistic, to the point that we ask today: is it still right to have an offside rule designed like this?

That, to me, is the deeper lesson. Recruitment, like offside, was designed for a world that no longer exists. AI hasn’t just changed the tools; it’s changed the conditions. And when the conditions change fundamentally, the answer isn’t always to run the same process faster or with better technology. Sometimes you need to redesign the process itself. And sometimes we don’t even know yet what that redesign looks like.

15: What it means to be an AI adopter today

Being an AI adopter today means understanding two things.

The first is the probabilistic nature of the tools: LLMs may give you a different answer every time. Their predictability is low compared to existing pipelines. The second, though, is that the process you’re applying AI to might not be as deterministic as you think it is; and if it is indeed deterministic, it might carry a measurable error.

Take a simple example. Let’s say we want to design an application that transcribes voicemail automatically. At first you might think: AI won’t give me certainty, it might not understand the numbers, the interpretation of sound will be full of mistakes. You might think you trust a human more. But a human transcribing the same voicemail also makes mistakes: noise on the line, an unfamiliar accent, a bad connection. There is a degree of error that is intrinsic in the process, with or without AI.

The key element is measuring. It is deciding beforehand what you’re measuring against, if it’s a like-for-like replacement.

Being an adopter also means accepting that the technology you use will change. Two years ago I would have sworn on ChatGPT as the leader, looking at adoption data. Now, it seems that Claude is in a position of leadership. In two years’ time things might be different again. We are at a stage in the development of AI where technical expertise, while important, is not as important as curiosity and the ability to experiment.

And that’s where I’ll leave it. AI adoption, for non-technical leaders, starts with becoming effective users of AI by measuring against existing baselines. It starts with understanding that processes need redesigning, not just automating. And above all, it requires the willingness to keep adapting, because the landscape will keep shifting under your feet.

Those are the skills to invest in.

And if you’d like to get updates, please join my weekly newsletter about data.