Same AI Technology. Different Results. Why?

A few months ago I wrote about a Stanford study that found something intriguing: organisations implementing the same AI technology for the same use cases were getting vastly different results. In one organisation, weeks. In another, years.

I wrote at the time that the difference wasn’t primarily AI, it was the organisation around it: its readiness, its processes, its leadership, its willingness to redesign work, absorb iteration and recover from failed attempts. In the Stanford cases, 77% of the hardest challenges sat outside the technical layer, and 61% of successful implementations had already been through at least one failed attempt.

So, we’ve seen evidence two orgs can implement the same AI tech and yet end up with different results. Are we saying technology does not determine the organisational outcome? Of course we are, and this is why the myopic focus on tech first fails us.

Forty years ago Stephen Barley studied the introduction of CT scanning technology into two radiology departments and watched different organisational structures develop around it.

Barley’s argument wasn’t that the technology didn’t matter, quite the opposite. The CT scanners disrupted established roles and relationships between radiologists and technologists. New expertise appeared in different places, existing hierarchies were disturbed, patterns of interaction changed, but the technology itself didn’t prescribe what the organisation should look like. What emerged depended on how people responded to it through the work.

That’s why the title of his 1986 paper is so timeless for us: Technology as an Occasion for Structuring.

An occasion for structuring, not a blueprint for structure.

Now jump forward forty years and replace the CT scanner with AI. We have organisations buying access to many of the same models, often from the same handful of vendors, and getting wildly different results. We tend to go looking for explanations in technology. Better models. Better prompts. Better tooling. Better integrations. Some of that obviously matters, but what if a large part of the difference is what happens to the organisation when the technology enters the work?

Who starts doing something?

Where does expertise move?

Where does judgement move?

Which interactions disappear and which new ones appear?

What happens to an approval that existed because information previously had to travel through a manager?

What happens when someone lower down the hierarchy suddenly has access to analytical capability that previously sat selsewhere?

What happens when the technology itself can participate in the work rather than simply support it?

This is where Barley gets really interesting for me, because AI makes the original question harder. A CT scanner changed the distribution of expertise and therefore the relationships between the humans around it. An intelligent system can do that too, but it can also perform parts of the cognitive work itself. It can synthesise, recommend, generate, monitor, route and increasingly act. We are not only changing the tools available to the existing actors, we are introducing a new kind of actor into the work.

And perhaps that helps explain why I am obsessing over implementation currently.

You cannot completely know the organisational consequence of introducing AI beforehand. You can redesign what you already know needs redesigning, but some consequences only become visible when the system meets the organisation. Roles shift. Workarounds appear. Assumptions break. Judgement turns out to live somewhere nobody had documented. The formal process turns out not to be the process at all.

That isn’t implementation going wrong, some of it is precisely what implementation allows us to discover.

This is where the Stanford findings and Barley’s work start to meet. Stanford showed that similar AI implementations can move at radically different speeds because organisations differ in their ability to sponsor, redesign, iterate and absorb the technology. Barley showed us decades earlier that the organisational consequences of a technology are not only contained within the technology itself.

Perhaps we spend too much time comparing the technology two organisations bought and not enough time comparing what happened to those organisations when they implemented it.

And there is another implication here that I think matters. If some organisational consequences only become visible through implementation, then implementation isn’t just the execution of a design decided beforehand, it is also a way of discovering what the organisation needs to become.

None of this means we shouldn’t redesign upfront. If we already know a workflow is broken, automating it and waiting for something interesting to emerge is plainly ridiculous. Redesign what you already know, but accept that you cannot know everything before intelligent systems begin operating inside the organisation.

Once they do, they start showing you things, and we truly start learning, which is the name of the game.

Where expertise actually lives. Where the documented process differs from the performed one. Which approvals still have a purpose and which are remnants of an older information constraint. Where human judgement is genuinely required. Where authority needs to move. Which roles become more important and which assumptions about them no longer hold.

That is why I think of implementation as a mechanism for organisational discovery.

And this takes Barley somewhere he obviously couldn’t have taken it in 1986. The CT scanner occasioned restructuring among the humans who worked around it. Intelligent systems can become participants in the work around which that restructuring occurs.

We are changing not only the technology through which an organisation operates, but the distribution of intelligence within the organisation itself.

Perhaps that is the real significance of Barley for AI. Technology was already an occasion for structuring. Intelligent systems make implementation an occasion for discovering how the organisation itself must change.

So perhaps the Stanford finding shouldn’t surprise us at all.

Same AI technology. Same use case. Different results.

The technology creates the possibility. Implementation is where the organisation discovers what that possibility does to the work. What it learns, what it captures and what it changes as a result may ultimately determine the outcome.

Forty years later, Barley’s question feels remarkably current.

Only this time, the technology isn’t just changing how humans organise around a machine.

The machine can participate in the work in its own right.

◇

I work with organisations navigating exactly these questions: AI implementation, work redesign and the organisational consequences of intelligent systems.