Artificial intelligence is moving from the airport terminal to the cockpit, with aviation companies exploring systems that can automate pilot tasks now, and support fully autonomous flight later. Apparently, AI could make flying safer, more efficient, and less dependent on overstretched human crews. But transport history on the ground and in the air offers unsettling reminders that safety-critical automation errors are devastatingly costly. What, then, could possibly go wrong?

At Quonset State Airport in Rhode Island, an experimental Cessna Caravan fitted with Merlin Lab’s “Merlin Pilot” system, was flown completely autonomously with passengers. During the flight, the test pilot sat at the controls but did not fly the aircraft, as the system handled all piloting tasks as well as acceleration, take-off, turning, descent, and landing.

The system goes beyond conventional pilot assistance because it’s designed to interact directly with air traffic instructions rather than simply hold altitude or speed. Merlin Pilot uses a natural-language processing model to listen to a mock air traffic controller and respond over the radio using a computer-generated female voice. The machine wasn’t following a pre-set route, but instead interpreting spoken instructions and converting them into real-life action.

Merlin says it has already completed hundreds of test flights while working towards certification from the Federal Aviation Administration (FAA) and the first major proving ground may be military rather than passenger aviation. The company has already secured a $100 million contract with the US Air Force to bring the technology to C-130 cargo planes, while the CEO insists the goal is not to “flip a switch” to pilotless aircraft, but to place artificial intelligence alongside human pilots and build trust gradually.

Of course, aviation already depends heavily on automation. Autopilot exists, along with auto-land, fly-by-wire control systems, traffic-collision assistance, and flight management computers. In fact, modern aviation is often seen as being safer these days largely because of the machines that already help humans avoid fatigue, error, and overload.

Artificial intelligence, however, is a different ball game. Traditional automation follows designed rules within strictly-regulated boundaries for specific situations. AI systems instead are trained on data, from which it interprets how to proceed in future situations, rather than being explicitly programmed for every scenario. The FAA’s own roadmap for AI safety assurance says aviation faces a new challenge with systems that “achieve performance and capability by learning rather than design”, because the field lacks the same engineering principles that guide traditional design assurance.

As such, the regulator is acknowledging that AI does not fit neatly into the classic safety model. Traditionally, aviation is the industry that checks, certifies, tests, duplicates, simulates, and signs off. Artificial intelligence systems guiding aircraft work on totally different operational foundations.

The European Union Aviation Safety Agency (EASA) is moving in the same direction, but appears more optimistic in its language. In its Artificial Intelligence and Aviation report, EASA says AI is “set to play a role in all aviation domains”, offering advanced assistance to aviation professionals and process optimisation that could make the sector safer and more sustainable. It does, however, also warn of the risks around the complexity of machine-learning systems, ethics, and cybersecurity.

On the roads, the first generation of autonomous vehicle testing demonstrated the true cost of artificial intelligence errors. On 18 March 2018, an Uber automated test vehicle struck and killed 49-year-old Elaine Herzberg in Tempe, Arizona. The National Transportation Safety Board said the vehicle’s developmental automated driving system was active at the time. In its report, it pointed to Uber’s “inadequate safety culture” and said the company “did not adequately manage the anticipated safety risk of its automated driving system’s functional limitations, including the system’s inability in this crash to correctly classify and predict the path of the pedestrian crossing the road.”

In October 2023, California regulators suspended Cruise’s driverless robotaxi service after concluding its vehicles posed “an unreasonable risk to public safety”. The suspension followed several incidents, including one in which a Cruise robotaxi ran over a pedestrian – who had already been hit by a human-driven vehicle – and stopped on top of her, leaving her trapped with traumatic injuries.

Cruise was later fined $1.5 million after federal regulators said the company had filed incomplete reports that failed to disclose that its vehicle dragged the pedestrian around 20 feet. NHTSA’s deputy administrator said companies developing automated driving systems must prioritise “safety and transparency from the start”.

The risk with artificial intelligence is not only about whether the public can learn to trust the machine itself, but also the companies building the systems, the regulators approving their use, and the internal reporting culture that decides what the public learns when something inevitably goes wrong.

Automation has already gone wrong in the air, too. The Boeing 737 MAX was grounded worldwide after two crashes – Lion Air Flight 610 in October 2018 and Ethiopian Airlines Flight 302 in March 2019 – killed 346 people in total. Investigators focused heavily on the Maneuvering Characteristics Augmentation System (MCAS), an automated flight-control system that could push the aircraft’s nose down after receiving faulty sensor data. MCAS itself was not artificial intelligence, but it demonstrates how dangerous automation can be when introduced into high-stakes transport systems.

The NTSB later criticised assumptions made around MCAS, saying Boeing’s safety assessment did not adequately consider how multiple flight deck alerts and indications could affect pilots’ recognition of, and response to, unintended MCAS activation caused by erroneous angle-of-attack data.

Meanwhile, the aviation industry looks to the benefits. the International Civil Aviation Organization (ICAO) summarised that “AI is revolutionizing the aviation industry, optimizing processes and improving efficiency in key areas such as air traffic management, predictive maintenance and safety”. It can help predict equipment failures, reduce fuel burn, optimise routes, manage traffic congestion, improve pilot training, and process weather or traffic data at speeds no human ever could. But the ICAO also warns that its introduction raises questions about human-machine interaction, operator situational awareness, decision-making, and over-reliance on systems.

But when it comes to automated flights, the sheer cost of a single error is difficult to overstate. What if 99,999 flights are delivered safely, but AI fails to properly assess one combination of weather, sensor error, air traffic conflict, and human confusion that was impossible to model in the first place?

What if pilots become the monitors of the systems that they rarely need to overrule, but are expected to take control in seconds in a specific scenario that the machine can’t solve itself? And what if the real warning here is not AI’s seemingly inevitable takeover, but a quiet degradation in human skill? Any errors in the air, as seen in the 737 MAX catastrophes, can cost hundreds of lives.

Regulators insist the transition will be cautious. The FAA says artificial intelligence should be introduced incrementally, starting with lower-risk applications and experience feeding into broader safety methods. But the commercial pressure will not be neutral. Airlines face pilot shortages, cost pressures, fuel constraints, and constant demands for efficiency. Tech companies need customers, and governments want innovation. If AI is framed as the future of aviation safety, then any resistance can be made to sound anti-progress.

Informed scepticism is needed here. Aviation became safe through a culture that distrusted failure and investigated accidents obsessively. Extreme caution is a requirement as AI enters the airspace, without the same assumptions we see in other industries that the machine will be smarter than the human. And more importantly, the public needs more than just glossy language about innovation. It deserves to know who is responsible when the algorithm is wrong, how pilots remain skilled enough to intervene, what regulators can truly certify, and whether the first generation of AI aviation will be tested with enough humility before passengers become part of the experiment.