The Driver’s Guess: How We Make Sense of Self-Driving Cars

by

Author: Denis Avetisyan

As vehicles become more autonomous, drivers develop complex mental models to interpret the actions of opaque algorithms, but lack the tools to influence their behavior.

Research reveals the ‘folk theories’ drivers employ to navigate semi-autonomous systems and argues for increased participatory governance in algorithmic design.

Despite increasing reliance on artificial intelligence in everyday life, the opacity of algorithmic decision-making presents a fundamental challenge to user trust and agency. This is particularly acute in safety-critical systems like semi-autonomous vehicles, the focus of ‘Navigating Algorithmic Opacity: Folk Theories and User Agency in Semi-Autonomous Vehicles’. Our research reveals that drivers develop sophisticated “folk theories” to interpret often-unexpected algorithmic behavior, yet lack the informational resources to validate these understandings or participate in system governance. How can we design more transparent and participatory systems that empower drivers as epistemic agents, rather than passive data sources, and ultimately enhance the safety and accountability of autonomous technologies?

The Inevitable Obscurity: When Tools Become Ecosystems

Modern life is becoming inextricably linked to complex artificial intelligence systems, though often without conscious awareness. From the algorithms curating news feeds and recommending products, to the sophisticated software powering self-driving vehicles and even assisting in medical diagnoses, these systems are increasingly integrated into daily routines. However, the internal workings of many such AI remain largely obscured – functioning as ‘black boxes’ where the inputs and outputs are visible, but the precise logic connecting them is hidden. This isn’t necessarily due to malicious intent, but rather a consequence of the sheer scale and intricacy of the models themselves, often involving millions or even billions of parameters. The result is a growing dependence on technologies whose reasoning processes are difficult, if not impossible, for humans to fully comprehend, raising critical questions about trust, accountability, and the potential for unforeseen consequences.

The increasing prevalence of complex algorithms in critical decision-making processes introduces a significant challenge to both trust and accountability, a phenomenon often described as algorithmic opacity or the ‘black box problem’. These systems, while capable of remarkable feats, frequently operate in ways that are difficult, if not impossible, for humans to fully understand. This lack of transparency isn’t merely a technical hurdle; it creates a fundamental disconnect between action and rationale, making it challenging to identify biases, errors, or unintended consequences. Consequently, establishing responsibility when these systems fail, or produce unjust outcomes, becomes problematic, as the internal logic driving those outcomes remains obscured. This erosion of understanding undermines public confidence and hinders effective oversight, demanding new approaches to ensure these powerful tools are deployed responsibly and ethically.

The escalating complexity of modern artificial intelligence demands a shift in how technology is understood. Historically, dissection – meticulously examining individual components to deduce overall function – proved effective for simpler machines. However, contemporary AI systems, built upon millions of interconnected parameters and self-modifying algorithms, defy such linear analysis. The sheer scale and dynamic nature of these systems render traditional ‘reverse engineering’ impractical, if not impossible. Consequently, researchers are exploring novel methodologies – including information-theoretic approaches, network analysis, and emergent behavior modeling – to move beyond simply knowing what an AI does, to understanding how it arrives at its decisions. This isn’t merely a matter of technical curiosity; a deeper comprehension of these inner workings is crucial for ensuring responsible development, mitigating biases, and fostering genuine trust in increasingly pervasive automated systems.

Beyond Instruction: The Rise of Algorithmic Interaction

Traditional designed technologies operate based on explicitly programmed instructions, resulting in predictable and repeatable outputs for given inputs. Modern Artificial Intelligence, especially systems built on machine learning paradigms, differs fundamentally; these systems learn from data and adjust their internal parameters, meaning behavior is not fixed at the point of deployment. This learning process introduces a degree of unpredictability and necessitates a shift from simply operating a tool to interacting with a system whose functionality evolves over time. Consequently, user engagement must move beyond issuing commands and instead focus on providing data, evaluating outputs, and iteratively refining the AI’s performance through continued interaction and feedback.

Strategic interaction with modern AI systems differs fundamentally from the operation of designed technologies due to the non-deterministic nature of AI outputs. Unlike tools with fixed functionalities, AI – particularly machine learning models – generates responses based on learned patterns and probabilistic calculations. Consequently, achieving desired results often requires iterative refinement of inputs, careful framing of prompts, and ongoing assessment of outputs. This iterative process represents a negotiation between the user and the AI, where the user strategically adjusts their approach based on the AI’s responses to converge toward a satisfactory outcome. The user is not simply instructing a device, but rather engaging in a dynamic exchange to guide the AI’s behavior and shape its output.

Agentified human knowledge refers to the incorporation of human expertise, values, and biases into the algorithms and datasets used to train artificial intelligence systems. This isn’t simply programming explicit rules; rather, it involves representing knowledge in a format that AI can process, often through large-scale data labeling, reinforcement learning signals derived from human feedback, or the pre-training of models on human-generated text and code. Consequently, AI behavior isn’t purely a function of the algorithm itself, but is significantly influenced by the characteristics of the data and the implicit assumptions embedded within it. This process frequently occurs without explicit awareness, leading to outputs that reflect the perspectives and limitations of the humans involved in the AI’s development and training, rather than objective truth.

The Ghost in the Machine: Folk Theories and the Illusion of Control

When interacting with complex and opaque systems, such as autonomous vehicles, individuals routinely construct “folk theories” – readily accessible, intuitive explanations to rationalize observed behaviors. These theories function as cognitive shortcuts, allowing users to predict system actions and understand unexpected outcomes despite limited access to internal mechanisms. The development of these explanations is a natural response to the inherent unpredictability of complex systems and serves to reduce uncertainty and maintain a sense of control. This process is particularly prevalent when system behavior deviates from expected norms, prompting users to actively interpret and ascribe meaning to otherwise inexplicable actions.

The application of metaphors, particularly the ‘Cyborg Metaphor’, is a common cognitive strategy employed when individuals attempt to understand interactions with complex automated systems. This metaphor conceptualizes the human-machine relationship not as a strict division of labor, but as an integrated partnership where human intention and algorithmic action are blended. Drivers, for example, may attribute agency to the vehicle, interpreting its actions as collaborative rather than purely mechanical, and projecting human-like qualities onto the system to rationalize its behavior. This framing allows for a more intuitive, though potentially inaccurate, understanding of the vehicle’s decision-making processes, facilitating a sense of control and predictability when interacting with opaque automation.

Drivers encountering unexpected behavior from Advanced Driver-Assistance Systems (ADAS), such as phantom braking, frequently develop explanatory narratives – termed ‘folk theories’ – to account for the system’s actions. This study confirms that these explanations are actively constructed by drivers as a means of interpreting opaque algorithmic behavior. While providing a sense of understanding, these intuitive theories can be inaccurate due to the inherent complexity and non-human logic of the algorithms governing the ADAS. The research indicates that drivers prioritize creating a coherent narrative over accurately understanding the technical cause of the anomalous event, potentially leading to inappropriate trust or distrust in the system and influencing future interactions.

Beyond Governance: Cultivating Justice in the Age of Data Assemblages

Data Justice represents a necessary shift in data governance paradigms, moving beyond purely technical or economic considerations to explicitly address systemic inequities perpetuated by data-driven systems. This approach recognizes that data collection, analysis, and deployment are not neutral processes, but are shaped by and reinforce existing power structures. Prioritizing fairness and equity in Data Justice necessitates critical examination of data sourcing methods, algorithmic design choices, and the potential for discriminatory outcomes. Implementation requires proactive measures to mitigate bias, ensure data privacy, and promote accountability, ultimately aiming to redistribute the benefits of data-driven innovation more equitably across all affected communities.

Data Colonialism describes the capture of value from data generated by individuals and communities, particularly those in the Global South, without equitable compensation or agency. This process often mirrors historical colonial practices, wherein data is extracted to benefit external entities – typically corporations or governments – while those who generate the data receive minimal to no benefit and have limited control over its use. Key characteristics include asymmetrical power dynamics, the imposition of data collection practices without informed consent, and the reinforcement of existing inequalities through data-driven decision-making. The resulting data asymmetries can lead to exploitation, discrimination, and the erosion of local knowledge systems, hindering self-determination and reinforcing dependence on external actors.

Participatory Algorithmic Governance (PAG) establishes processes for including individuals and communities impacted by data-driven systems in the development and monitoring of ‘Data Assemblages’. These assemblages consist of the interconnected network of people, algorithms, and the data they utilize, functioning as socio-technical systems. PAG moves beyond traditional stakeholder engagement by advocating for direct participation in the design phase, allowing affected groups to define requirements, contribute to data labeling and validation, and influence algorithmic choices. Oversight mechanisms within PAG frameworks include community-based auditing of algorithms for bias, transparent reporting of data usage, and the establishment of redress mechanisms for harms resulting from automated decision-making. The goal is to shift control over Data Assemblages away from solely technical experts and towards a more equitable distribution of power and accountability.

The study of driver ‘folk theories’ reveals a predictable outcome: systems, even those promising autonomy, remain tethered to human understanding, however imperfect. These mental models aren’t bugs to be engineered out, but emergent properties of complex interactions. The vehicle becomes less a machine executing code and more a locus where human intention and algorithmic action converge – and inevitably, diverge. As Barbara Liskov observed, “Programs must be right before they are wrong.” This research underscores that even sophisticated algorithms aren’t immune to being ‘wrong’ in the eyes of those who must interpret their behavior, and that a lack of participatory governance only exacerbates the widening gap between design intent and user experience. The vehicle’s opacity, therefore, isn’t merely a technical challenge, but a fundamental limitation of systems striving for control while simultaneously relying on human adaptation.

The Road Ahead

The proliferation of ‘folk theories’ within the context of semi-autonomous vehicles isn’t a bug; it’s the inevitable consequence of coupling human perception with systems exceeding its grasp. These mental models, born of necessity, demonstrate a profound truth: opacity isn’t simply a technical challenge, but a fundamental condition of complex systems. The vehicle doesn’t explain its actions; it demands interpretation. And yet, current designs treat the driver less as a co-evolutionary partner and more as a redundant sensor, a final failsafe against predictable failures.

The focus on ‘algorithmic accountability’ often mistakes the symptom for the disease. The system will always exceed the boundaries of formal verification. The crucial question isn’t how to make the algorithm transparent, but how to cultivate a more porous boundary between the internal logic and the external world – a world populated by drivers actively contributing to the refinement of the very models governing their experience. Data assemblages, if truly participatory, must become arenas for collective sensemaking, not just passive data collection.

Future work should abandon the pursuit of complete explanation and embrace the inherent ambiguity of these systems. The task isn’t to eliminate folk theories, but to design architectures that can absorb them, that can evolve with them. If the system is silent, it is not merely plotting – it is waiting for a story to be told.

Original article: https://arxiv.org/pdf/2602.07312.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2026-02-10 14:28