Incongruent
Experimental podcast edited by Stephen King, co-founder of KK Studio and founder of the Steve K. King AI Academy.
Correspondence email: steve@kk-stud.io.
Incongruent
Cabinet office 2025: The People Factor: Understanding Hidden Risks in Organizational AI Adoption
Spill the tea - we want to hear from you!
The biggest challenges in AI implementation aren't the headline-grabbing risks but the hidden ones stemming from human factors and organizational dynamics. Drawing parallels to aviation safety, successful AI deployment requires understanding how people interact with these tools in complex organizational settings.
• Hidden AI risks often come from human-system interactions rather than technical failures
• Real examples show how AI can create burnout, enable misinformation spread, or amplify existing biases
• Standard AI safety approaches often miss these subtle but critical issues
• The "Adopt, Sustain, Optimize" framework helps track the user journey through AI implementation
• Six categories of hidden risks include quality assurance, task-tool mismatch, and workflow challenges
• Proactive "pre-mortem" approaches are more effective than waiting for problems to emerge
• Human oversight only works when people have expertise, time, and authority to challenge AI outputs
• Successful implementation requires diverse teams, tailored training, and leadership understanding
• Measuring impact should go beyond efficiency to capture quality improvements and risk management
• Building resilient sociotechnical systems means designing for human realities, not just deploying technology
Okay, let's unpack this. What if I told you that the biggest challenges in successfully bringing AI tools into your organization you know, managing the genuinely risky stuff have more in common with, well, aviation safety checks than with, say, a headline about a rogue chatbot. It's not always the dramatic crash. Often it's about the equivalent of a slightly miscalibrated instrument, or maybe communication missed during turbulence.
Speaker 2:That parallel is exactly where the sources we're diving into today begin. They argue that while we focus on the big visible AI risks you know bias, deep fakes, ai hallucinations the most significant problems in real world implementation often stem from far more mundane human and organizational factors, things that remain hidden until they cause an issue.
Speaker 1:And these insights aren't coming from a tech giant or some sci-fi author, are they?
Speaker 2:Correct. No, we're working with a stack of material from the UK Cabinet Office, specifically these detailed guides and resources developed around two key publications the People Factor a human-centered approach to scaling AI tools, and the Mitigating Hidden AI Risks Toolkit.
Speaker 1:Okay, so our mission for this deep dive is basically to strip down these government documents, pull out the actionable insights. We want to understand why getting AI tools right in practice is maybe less about the code and more about the people and processes Exactly. And crucially, how you can proactively identify and maybe head off the risks you probably aren't even looking for yet.
Speaker 2:Precisely, it's about looking past the obvious dangers to build systems that are truly resilient in these messy human environments.
Speaker 1:Okay, let's get into that core problem. Then we're all kind of aware of the high profile AI risks algorithmic bias, deep fakes, ai generating false information they're real. Algorithmic bias, deep fakes, ai generating false information they're real, they're important and, yeah, they grab headlines.
Speaker 2:And the sources don't dismiss those, definitely not. But they pivot pretty quickly to argue that the most common and damaging risks in actual organizational use they come from elsewhere.
Speaker 1:Right.
Speaker 2:Think back to that aviation parallel. The majority of accidents are linked to human factors miscommunication, errors in procedure, fatigue, maybe organizational culture issues, not usually a sudden engine failure. These sources propose AI implementation. Risks are well likely to follow a similar pattern.
Speaker 1:So it's not necessarily the AI failing, but the system around the AI, the people, the processes, the culture that introduces the major vulnerabilities.
Speaker 2:That's the core argument, and these are what the sources term hidden risks, basically unintended consequences that aren't immediately visible or salient. They sort of remain below the surface until they've potentially snowballed.
Speaker 1:Can you give us some examples from the sources, ones that really illustrate this? What does a hidden risk actually look like on the ground?
Speaker 2:Yeah, the sources use several helpful scenarios. Consider Marco His organization automates his routine, seemingly menial tasks. Logically, this should free him up for higher value work.
Speaker 1:Sounds good on paper.
Speaker 2:But the hidden insight those tasks were also cognitive breaks. Now Marco's day is just relentlessly demanding, leading to fatigue, stress and actually reduced productivity. The risk materialized not from the AI's technical function, but from its unexamined impact on his workflow and well-being.
Speaker 1:Wow, okay, that's insidious a benefit that creates invisible costs.
Speaker 2:Totally. Or Nithya, an analyst under intense deadline pressure, she uses a gene AI tool for urgent research. The output looks plausible, seems okay and, crucially, that time pressure prevents her from conducting proper due diligence, like cross-referencing with verified sources, so she shares inaccurate information that then spreads. The risk isn't just the AI might generate errors, but how a human's operating conditions, the pressure, the lack of time interact with the tool to enable the spread of bad info.
Speaker 1:What about Jordan, the researcher, who already has a strong opinion? He prompts the AI reflecting his pre-existing views.
Speaker 2:Yes, that's another good one. He asks the AI to find evidence supporting his hypothesis. The AI, because it's designed to be helpful and provide relevant responses, essentially caters to his perspective Right. The sources point out this can really reinforce confirmation bias, leading decision makers to act on an incomplete or skewed picture. The AI didn't invent the bias, but the interaction amplified the human's existing bias.
Speaker 1:And the Sarah example, the hiring manager using AI to sift applications.
Speaker 2:That one is critical. The tool gives plausible sounding scores, maybe percentages or ranks. Sarah trusts them. Maybe skip some quality checks outlined in policy because she's busy or the scores seem objective OK. While algorithmic bias might exist in the tool, the actual harmful outcome embedding discrimination into recruitment at scale happens because of the human decision to trust the outputs without proper checks and maybe an organizational failure to enforce those checks.
Speaker 1:It's the human system interaction where the risk really hits. These examples really underscore that it's the human and organizational layer where things get well complicated. Score that it's the human and organizational layer where things get well complicated and the sources are quite direct in saying that standard AI safety approaches often miss these issues right.
Speaker 2:They are very direct Technical safeguards, like improving training data or using adversarial testing, red teaming. They're necessary, sure, but they're just not sufficient.
Speaker 1:They won't stop Marco's burnout.
Speaker 2:Exactly or address a manager's misplaced trust or Nithya's time pressure problem.
Speaker 1:And the common phrase human in the loop, just having a person involved in oversight. What do the sources say about that?
Speaker 2:They critique this quite heavily. It's often presented as the solution like a magic bullet, but it's only effective if the human can actually provide effective oversight.
Speaker 1:Like Sarah.
Speaker 2:Right. Like in Sarah's example, people might lack the necessary expertise to judge the AI or sufficient time to do it properly, or maybe the psychological safety or the actual authority to challenge the AI outputs or its use, even if they spot a potential problem. Research mentioned in the documents actually shows even experienced professionals can struggle with effective AI oversight, sometimes even introducing new errors themselves.
Speaker 1:We're just telling people you know, read the terms and conditions and verify outputs yourself.
Speaker 2:Yeah, they liken that to the may contain nuts warning on a food package.
Speaker 1:Yeah.
Speaker 2:It's necessary, legally maybe, but it's insufficient on its own as a safety system. People often don't read them or they forget, or they don't act on disclaimers when they're under pressure or just doing something routinely. It doesn't build real safety culture.
Speaker 1:Okay, here's where it gets really interesting. Then the sources argue there was a critical gap. Plenty of frameworks existed for technical risks for the AI itself, but none specifically for systematically identifying and mitigating these subtle behavioral and organizational hidden risks before they blow up.
Speaker 2:Exactly, and that gap is precisely what their work aims to fill.
Speaker 1:So how do they propose we fill it? They offer this core framework called Adopt, Sustain, Optimize, ASO. What's the thinking behind that structure?
Speaker 2:Well, the fundamental idea is that successful AI deployment isn't just a technical rollout. It's a social and behavioral transition. It's about people changing how they work.
Speaker 1:Okay.
Speaker 2:The ASO framework maps this transition through three distinct stages, really from the user's perspective.
Speaker 1:Adopt being the first stage makes sense.
Speaker 2:Right. The goal here is simple, but absolutely essential Encourage people to start using the tool. An unused tool provides zero benefit, obviously, so this phase focuses on understanding and addressing the barriers to just getting started. Things like do people think it's useful? Is it easy enough to use? Do they feel confident? What are their colleagues doing? Do they trust it? Basic AI literacy stuff too.
Speaker 1:And they provide ways to figure that out.
Speaker 2:Yeah, the supplementary materials even provide example survey questions to help map user profiles and identify those specific barriers effectively.
Speaker 1:Okay, so you get people using it initially. Then comes sustain.
Speaker 2:Right. The goal of sustain is to embed the tool into routine workflow, making sure it's used consistently, long-term, not just tried once and forgotten.
Speaker 1:Making it a habit.
Speaker 2:Exactly. This phase considers concepts like the habit loop needing a trigger to use the tool, receiving some kind of benefit or relief, the reward, and then repeating the action. It's about integrating the AI into people's daily tasks so it becomes second nature.
Speaker 1:And you track that.
Speaker 2:Definitely Metrics are key here. Things like login frequency, interaction patterns the supplementary evaluation guides mention these, Plus user feedback to see if routine use is actually happening and spot any hurdles.
Speaker 1:Got it From trying it once to using it regularly, and that leads us to the third stage Optimize.
Speaker 2:Optimize. This is where the focus really sharpens on high quality, effective and safe use. It directly tackles the hidden risks we talked about earlier.
Speaker 1:Oh OK.
Speaker 2:The goal isn't just that people use the tool consistently, but that they use it well, maximizing the intended benefits while actively avoiding or mitigating those unintended negative consequences.
Speaker 1:So optimize is where the framework for tackling these hidden risks really leaves intended benefits, while actively avoiding or mitigating those unintended negative consequences.
Speaker 2:So optimize is where the framework for tackling these hidden risks really leaves Precisely. It's dedicated to that phase where usage is established. But the subtle risks linked to that ongoing human AI interaction are most likely to pop up if you're not proactively managing them.
Speaker 1:And they break these potential hidden problems down into six categories. Let's unpack those lenses they suggest using.
Speaker 2:Okay, first is quality assurance risks. This covers problems that come from using outputs that are inaccurate, maybe incomplete or just sort of mediocre without proper checking.
Speaker 1:Like Nithya under pressure.
Speaker 2:Exactly like Nithya Time pressure causing a human to bypass quality checks on Gen AI output. The risk is less about the AI error rate itself and more about how human factors make us accept or miss errors.
Speaker 1:Okay, makes sense.
Speaker 2:Second Second task tool mismatch. This is basically using an AI tool for something it wasn't designed for or just isn't very good at.
Speaker 1:Right.
Speaker 2:The sources highlight risks like, say, a leader insisting a tool is used for inappropriate task and maybe staff feeling they can't push back. The outcome is often suboptimal, maybe even unsafe, because the tool just isn't the right fit for that particular job.
Speaker 1:Category three perceptions, emotions and signaling. This gets right into the human side, doesn't it?
Speaker 2:Absolutely. This is all about risks coming from how people feel about the AI rollout, maybe anxiety about job security, changes in morale, resistance to change, or even what the organization's approach signals. If the rollout is perceived as purely about cutting costs or efficiency, it might signal that human judgment or quality isn't valued as much, which then influences how people use the tool maybe cutting corners.
Speaker 2:Fourth is workflow and organizational challenges. This covers risks from how AI changes the actual structure of work or team dynamics. Marco's burnout is a classic example here. The AI changed his task distribution in a way that negatively impacted his well-being. Another example could be if low adoption by some people impacts the workflow of others who are using the tool, maybe creating friction or inefficiency in a process that needs everyone on board.
Speaker 1:Fifth is ethics.
Speaker 2:Right. This category, as the sources define it, addresses risks where AI use could lead to unfair, inequitable or harmful outcomes that violate organizational standards, norms or even legal requirements like equality laws.
Speaker 1:Like the SARA recruitment example.
Speaker 2:Exactly the SARA example of biased recruitment, where existing algorithmic bias leads to discriminatory hiring outcomes because of the human and process factors around the tools use, like skipping checks, is a key illustration they use.
Speaker 1:And the final category, number six.
Speaker 2:The final one is human connection and technological over-reliance. This is about risks where AI replaces essential human roles or interactions. This could lead to a loss of critical skills over time, maybe reduced collaboration, decreased job satisfaction or just diminished quality of service in areas where human empathy or that nuanced human judgment is really crucial. Relying only on automated systems when a human touch or expert intervention is actually needed is a big risk, highlighted here.
Speaker 1:So these six categories give organizations a structured way to think about where these hidden problems might be lurking, and the sources strongly advocate for being proactive, right, not waiting for something to go wrong.
Speaker 2:Completely. Their core recommendation is basically doing a pre-mortem, not waiting for a post-mortem. Don't wait for the crash to retrieve the black box, so to speak. Use the toolkit Exactly. Use the structured, prompt questions provided in the toolkit for each of these six risk categories to brainstorm and anticipate what could go wrong before you even deploy or scale the AI tool widely.
Speaker 1:And they provide a clear like five-step process for actually doing this pre-mortem.
Speaker 2:Yeah, it's designed to be practical. Step one assemble a multidisciplinary team. Getting diverse perspectives is absolutely crucial for spotting these varied risks Makes sense. Step two systematically surface potential hidden risks. Use the prompt questions, look at existing research, maybe analyze early pilot use of the tool. Step three review and prioritize the identified risks, probably based on likelihood and potential severity.
Speaker 1:You can't fix everything at once.
Speaker 2:Right. Step four develop mitigation strategies and plan how you're going to monitor those risks. And step five implement ongoing monitoring and review mechanisms, because these risks can change over time as people use the tool more.
Speaker 1:They mention a real example the assist team.
Speaker 2:Yeah, the sources mention the assist team. Using this process, they apparently identified over 100 potential risks related to their tool and then prioritized 33 to actively manage. It shows it's doable.
Speaker 1:That provides a real roadmap. And going back to that human in the loop idea for a second, the sources didn't just dismiss it entirely, did they? They sort of refined how it needs to work to actually be effective within this optimized phase.
Speaker 2:Exactly they didn't throw it out, but they put serious conditions on it. They argue human oversight is only a valuable mitigation strategy if the human involved has three critical things.
Speaker 1:Okay, what are they?
Speaker 2:One, sufficient relevant expertise to actually evaluate the AI's output critically. Two, adequate time allocated for the review process, not just expected to squeeze it in between other tasks. And three, the authority or, importantly, the psychological safety to genuinely challenge the AI's output or its proposed use, without fear of reprisal or being ignored.
Speaker 1:Psychological safety that's key being ignored. Psychological safety.
Speaker 2:That's key. Absolutely. Without those conditions, expecting humans to reliably catch AI errors or inappropriate use is itself a significant hidden risk.
Speaker 1:It's false reassurance. It shifts the focus from just having a human there to ensuring the human is actually empowered and enabled to be effective. Okay, so how do organizations actually implement these frameworks and mitigate these diverse risks? What kinds?
Speaker 2:of strategies do the sources suggest? They offer various strategies and they're woven throughout the guides, always linking back to that ASO framework and the six risk categories. It's less about finding one single technical fix and much more about systemic changes.
Speaker 1:Like tailored training, maybe Better training.
Speaker 2:Yes, but moving beyond just generic AI literacy 101, the sources advocate for developing actionable training specific to the tool and the task. Training that includes guidance on recognizing and mitigating specific risks like bias, in practice how it might show up in this tool, rather than just a blanket warning about AI bias in general.
Speaker 1:Okay, and clear guidelines on how to use the tool.
Speaker 2:Crucial, especially for that task tool mismatch, risk, Clearly defining the AI tool's intended use cases, its limitations and, really importantly, how to handle situations where it's not suitable. This involves understanding how people are actually using general purpose tools like GenAI and steering them appropriately.
Speaker 1:Leadership plays a big role too, I imagine A massive one.
Speaker 2:Leaders need to understand the AI tool's capabilities and limitations themselves. You know, get hands-on maybe, so they can model and advocate for responsible, effective use. The sources even highlight giving leaders early access as a specific strategy to build this understanding and encourage them to lead by example.
Speaker 1:And measuring the right things, not just speed.
Speaker 2:Yes, rigorous impact measurement that goes beyond simple efficiency gains or self-reported time savings, which they know can be unreliable. Are there actual quality improvements? Are risks being managed effectively? The sources discuss using diverse method surveys, structured interviews, maybe even quasi-experimental approaches where feasible.
Speaker 1:And getting feedback from everyone, not just the enthusiasts.
Speaker 2:Absolutely vital, especially for addressing those perception risks and ensuring equitable adoption. Actively seeking input from people who aren't using the tool or are hesitant helps uncover those hidden barriers, concerns and potential inequities in who benefits or who is exposed to risks. It requires a truly user and non-user centered approach.
Speaker 1:And none of this works if it's done in silos right, Like just the tech team doing it.
Speaker 2:Correct. That's a recurring theme the need for multidisciplinary teams throughout the AI lifecycle, bringing together the technical experts, sure, but also behavioral scientists, designers, hr people, legal experts, ethicists and, crucially, the end users themselves, to get that holistic view of both the risks and the potential solutions. The structure and approach to the ASSIST team, which is referenced throughout the documents, seems to exemplify this.
Speaker 1:It feels like the sources really ground these frameworks in reality by constantly referencing the experiences and findings from that AS that assist tools rollout within the UK government. You mentioned tailoring training based on user prompts, using surveys on confidence alongside login data, specifically tracking equality impacts.
Speaker 2:Exactly. It provides concrete examples of the frameworks and strategies in action, showing they aren't just abstract theories on a page. They've tried to apply this stuff.
Speaker 1:So what does this all mean? Theories on a page. They've tried to apply this stuff. So what does this all mean? If we boil down this whole deep dive into the UK Cabinet Office's perspective on AI implementation, what's the core takeaway for you?
Speaker 2:I think the most important insight is this Successful, safe and equitable AI adoption in organizations is fundamentally a human and organizational challenge, not purely a technical one. Focusing only on the AI model itself while neglecting the people factor and these hidden risks, the messy reality of how humans and systems interact with the technology, is probably the most significant path to undermining the very benefits AI promises in the first place, and these sources provide practical tools to actually shift that focus, to manage that complexity.
Speaker 2:Exactly. They offer the ASO framework as a guide for the entire user journey. They give you the six categories of hidden risks to systematically look for problems in the right places. They provide prompt questions to make that abstract search more concrete. And they lay out a structured process for identifying, prioritizing and mitigating those risks proactively process for identifying, prioritizing and mitigating those risks proactively. It feels like a genuine roadmap for moving beyond just hoping for the best towards actively designing for resilience. Amy.
Speaker 1:Quinton. It really reframes the challenge, doesn't it? From sort of fearing some dramatic AI failure the Skynet scenario to managing the much more likely messy reality of human AI interaction within complex systems. For anyone listening, who's involved in leading or working with AI in an organization, understanding these dynamics seems absolutely essential for harnessing AI's potential while avoiding those less obvious but potentially more damaging pitfalls.
Speaker 2:It's really about building a robust sociotechnical system, not just deploying a piece of technology and walking away.
Speaker 1:This has been incredibly valuable, thank you.
Speaker 2:My pleasure, good discussion.
Speaker 1:As we finish this deep dive, here's maybe a thought to carry with you. These sources compellingly argue that the real risks in AI implementation often stem from how we, the humans inside these complex organizations, interact with and integrate these powerful tools, often with the best intentions. Even so, if we don't actively design our processes, our workflows and empower our people to engage with AI effectively and safely, really ensuring they have the necessary time, the expertise and, crucially, the ability to challenge what critical human skills or maybe vital organizational dynamics are, we most subtly putting at risk of erosion or even failure in the long run?