Beyond bookending: Four lessons from Human Factors for the AI Era

The standard model of human-AI interaction is reassuringly simple: The human writes the prompt, AI does the work, and the human checks the answer.  

This “bookending” behaviour is appealing because it is clearly delineated, with each party apparently playing to its strengths. But it repeats an old Human Factors mistake: treating automation as a replacement for a human task rather than redesigning the whole sociotechnical system. The problem isn’t that the “book” is badly written. It is that bookending is the wrong architecture for complex sociotechnical systems. 

A recent news story from Australia illustrates the problem. A man asked an AI agent to book him a spot in a gym class. When faced with the prospect of no available spaces, the AI found a solution. It created space by cancelling other people from the waiting list. The agent identified a gap in the booking system (no authorisation checks on cancelling other members’ places) and exploited it.  

It is tempting to read this as an AI failing to hold the right values. It is more accurate, and more useful, to read it as a system given a goal and no meaningful constraints, doing exactly what it was structurally free to do. A person evaluating the same decision would factor in the cost to the stranger about to be displaced, and would carry some discomfort afterwards if the call was wrong.  

The Human Factors discipline has spent the last seventy years studying what happens when people hand control to machines, and most of these lessons transfer to AI. Not because Large Language Models (LLMs) behave like a 1980s autopilot (their failure modes are fundamentally different), but because they create a similar trap. While they may replace a task, they also create new ones. 

Four lessons from Human Factors explain why we must move from simple “bookending”.

Lisanne Bainbridge summed up this issue perfectly in 1983 with her paper “The Ironies of Automation”. Automate a task and the human is rarely removed from it. Instead, they are given a new task. Watch the machine, spot when it fails, take over when required. Parasuraman and colleagues made a related point: automation does not replace human activity; it changes it, usually in ways nobody designed for. 

AI follows an incredibly similar pattern. While it may replace the primary task, it bookends new tasks before and after it. Before, AI is dependent on very clear prompts that supply the necessary context and try to avoid any bias. Because conversational AI is generally optimised to be helpful and responsive, it may not challenge an inadequate brief unless explicitly designed to do so. Nor should we assume that it will recognise conflicting priorities and values that the user has left unstated. Perhaps more critically, the problem isn’t that humans write bad prompts. It is that humans don’t always know what information the system needs until things go wrong. 

The second human task involves checking the output. While hallucination rates have fallen in recent models, they still happen. And a wrong answer delivered with total confidence is considerably harder to catch than an answer given with a note of uncertainty.

The Human Factors literature is very clear that Humans are not great at monitoring or vigilance tasks. It is hard to judge the quality of an answer one did not have to produce. Models can now provide sources, explanations and supporting evidence, which can help, but much of what happens underneath remains opaque. This problem is amplified when the AI appears to know more about the subject than the person supervising it. People are reasonably effective at catching large, obvious failures. They are considerably worse at catching small errors and slow drift, and both effects have established names. Vigilance decrement describes attention fading during an extended watching task. Automation bias describes trusting a machine’s output more than the evidence warrants. Placed in a purely supervisory role, both effects take hold quickly.

Overreliance on a system’s ‘autopilot’ has a profoundly negative impact on how we develop and maintain skill. 

Skills fade without use. The aviation sector addresses this by requiring pilots to perform a minimum number of take-offs and landings within a defined period before carrying passengers, because skill erodes quietly and faster than most people expect. Toggling the autopilot on and off is not sufficient to maintain it. What decays is the judgement built from doing the job for real, not merely the mechanics of it. 

The second aspect is skill acquisition. If AI performs a task before someone has learned to do it themselves, that judgement is never built. Anyone who learned a role the slow way at least has that experience to check the machine against. People coming up through their careers may not. They are being asked to supervise a system without ever having done the underlying work themselves. 

This creates a dangerous feedback loop. The more AI does, the less opportunity people have to develop expertise; the less expertise they have, the harder it becomes to challenge the AI.

Humans have their own goals, values and responsibilities. We constantly read the situation around us, weighing competing priorities and considering the consequences of our decisions. We also live with those consequences. A poor decision can damage our reputation, relationships or confidence. We learn from it because we have to live with it. 

AI doesn’t. An AI system has whatever objectives, constraints and values are designed into it. It can reason about consequences, but it doesn’t experience them. The gym example illustrates the problem. The agent achieved its immediate goal, but ignored the wider consequence: someone else would lose their place. 

The problem wasn’t simply that the AI lacked empathy. It was that the system represented the goal, but not everything that mattered in achieving it. That distinction becomes critical as AI moves from generating information to taking action.

These differences between humans and AI are not presented to discourage the use of AI. Rather, they are presented to explain why the differences matter. For low-stakes tasks (like planning a meal), most of this is academic. In safety-critical or high-consequence work, it couldn’t be more important. 

As the lessons from the Human Factors literature have taught us, the solution lies in a carefully designed sociotechnical system. We need to move beyond the rather simplistic bookending allocation of function based on what computers do well and what humans do well (see the Fitts list). We need to allocate functions dynamically, based on the context, the uncertainty, and the consequences of getting things wrong. This involves four shifts: 

1. Make AI interrogate the brief. Don’t just make AI better at answering questions. Make it better at identifying missing information, conflicting objectives and questionable assumptions.
2. Route decisions by consequence and uncertainty. Don’t put humans in the loop everywhere. Put them where the consequences of being wrong are high, uncertainty is meaningful, or actions are difficult to reverse.
3. Verify claims, not outputs. Don’t ask humans to “check the AI’s answer”. Give them specific things to verify (a more achievable human task)
4. Design for expertise, not just efficiency. Don’t optimise every workflow for maximum AI involvement. Deliberately preserve opportunities for people to practise, learn and develop judgement.

The ultimate question isn’t whether AI can complete a task faster or better than a human. It is whether the combined sociotechnical system can achieve what we value: safely, reliably, and without silently eroding the human capability we will need tomorrow.