The Irreplaceable Human Intuition

Navigating the Looming Crisis of Vanishing Expertise

In the modern economy, a precious resource powers progress across myriad fields: the accumulated knowledge, judgment, and intuition of human experts. This “intellectual dark matter,” as technologist and analyst Marko Jukic calls it, is the product of decades of lived experience that often can’t be easily codified or automated. And it is now at risk of vanishing as veteran specialists retire or pass away without transferring their expertise to the next generation.

The looming competence and succession crisis threatens to slowly cripple our economy and society in ways that may seem inexplicable. The aerospace giant Boeing, for instance, has seen its reputation shattered by the 737 Max disasters, which some attribute to the company’s loss of engineering prowess as it pushed out seasoned experts in favor of outsourcing and youth. If this pattern holds across other industries, we may face more frequent catastrophic failures as the people who deeply understand complex systems exit the workforce.

Some hope that artificial intelligence can fill this void by capturing the knowledge of subject matter experts and replicating human judgment at scale. But while AI has made remarkable strides, it still struggles to match human intuition for solving poorly defined problems. The oil industry relies on a dwindling cadre of veteran “well pickers” to find productive drilling sites, trusting their instincts over sophisticated machine learning models and simulations. Certain innately human capabilities – imagination, emotional perception, abductive reasoning – remain extraordinarily difficult to automate.

As such, there are categories of jobs that are likely to prove stubbornly resistant to AI displacement: leadership roles demanding vision and high-stakes decision-making under uncertainty; creative fields like art and design that depend on novelty and an understanding of the human condition; positions requiring exceptional interpersonal skills, like counseling and teaching; and specialized physical trades, like equipment repair, that reward resourcefulness in unpredictable situations. Even as AI augments many facets of this work, the core skills are fundamentally human.

The consequences of failing to replenish our reserves of embodied expertise are dire. A recent study found that the average large U.S. company loses $47 million in productivity each year due to inefficient knowledge sharing. For smaller businesses with just 10 employees, the annual cost is still a staggering $50,000. Beyond the financial toll, expertise loss can lead to more large-scale disasters like the Boeing case, as the most experienced practitioners take their know-how with them.

Irreplaceable experts in some domains may command skyrocketing wages, even as more routine white-collar roles are automated away. Labor shortages could intensify in critical fields, from aerospace engineering to nursing, as the baby boomer generation retires en masse. And if the destruction outpaces the creation of new knowledge, we could face a civilizational regression.

The core issue, as highlighted by Harold Robertson in his article “Complex Systems Won’t Survive the Competence Crisis,” is that changing political mores have established the systematic promotion of the unqualified and sidelining of the competent. This has continually weakened our society’s ability to manage modern systems. At its inception, it represented a break from the trend of the 1920s to the 1960s, when the direct meritocratic evaluation of competence became the norm across vast swaths of American society.

By the 1960s, the systematic selection for competence came into direct conflict with the political imperatives of the civil rights movement. Administrative law judges have accepted statistically observable disparities in outcomes between groups as prima facie evidence of illegal discrimination. The result has been clear: any time meritocracy and diversity come into direct conflict, diversity must take priority.

The resulting norms have steadily eroded institutional competency, causing America’s complex systems to fail with increasing regularity. In the language of a systems’ theorist, by decreasing the competency of the actors within the system, formerly stable systems have begun to experience normal accidents at a rate that is faster than the system can adapt. The prognosis is harsh but clear: either selection for competence will return, or America will experience devolution to more primitive forms of civilization and loss of geopolitical power.

This competence crisis is unfolding from the core of the American system outwards. Government agencies, which are in charge of overseeing all the other systems, have seen the quality of their human capital decline tremendously since the 1960s. The most immediate danger is at safety-critical agencies like the Federal Aviation Administration (FAA), where a terrifying uptick in near-miss incidents in 2023 has raised alarms.

The decline in the capacity of government contractors is likewise obvious, with Boeing’s 737 MAX crashes and KC-46A Pegasus tanker issues as prime examples. Nonprofits, including universities, charities, and foundations, are the next-most-affected class of institutions, entrapped by the government policies that are subject to and the opinions of their donor base. Publicly-traded corporations face immense pressure to prioritize diversity over competence to avoid lawsuits and scandals, even at the cost of performance.

To avert this fate, we must act now to preserve and transmit endangered human expertise. That means reviving apprenticeship models to pass down implicit knowledge before it is lost. In the railway industry, for example, experienced workers are mentoring new hires to ensure the continuity of vital maintenance skills. Exploring collaborative workflows that combine human judgment and machine capabilities, such as AI-assisted medical diagnosis, can yield the best of both worlds.

Redesigning education to prioritize the human skills that are hardest to automate, fostering the creative, critical, and emotional intelligence to tackle undefined challenges, is also key. Vocational programs like Germany’s “dual-training” system, which combines classroom learning with on-the-job experience, offer a model for cultivating applied expertise.

At the policy level, expanding high-skilled immigration can help alleviate talent shortages in the near term. But a sustainable solution will require major investments in upskilling and lifelong learning to continuously renews the workforce’s knowledge base.

For investors, the implications are profound. Companies that can effectively harness and retain specialized human capital, through knowledge management systems, apprenticeship programs, and AI-augmented workflows, will likely enjoy a significant competitive advantage. Those that allow critical know-how to walk out the door may see their market value erode. In an era of rapid technological change, the ability to preserve and build upon technological change, the ability to preserve and build upon intellectual dark matter will separate the winners from the losers.

But why exactly is human intuition so difficult to replicate with AI?

At its core, intuition is the ability to understand something instinctively, without the need for conscious reasoning. It’s a form of knowledge that feels effortless and instantaneous, often described as a “gut feeling” or “sixth sense”. While it may seem like magic, intuition is actually a product of vast subconscious pattern recognition, honed through years of experience and learning.

Consider a veteran surgeon who can immediately spot a rare complication, or a master mechanic who can diagnose an engine problem by sound alone. Their intuition is not a supernatural power, but rather a finely tuned instrument that can detect subtle signals and anomalies that others miss. This deep situational awareness and ability to connect disparate dots is the hallmark of human expertise.

Intuition also draws heavily on our uniquely human capacity for abstraction and analogical reasoning. We can intuitively grasp the essence of complex ideas and see parallels between seemingly unrelated domains. This allows us to make creative leaps and solve problems in novel ways, without being constrained by rigid rules or algorithms.

Moreover, intuition is closely intertwined with emotion and empathy. Our gut feeling is often guided by our values, experiences, and understanding of human nature. A skilled therapist or negotiator relies on intuition to read between the lines and navigate delicate interpersonal dynamics. Emotional intelligence is a key ingredient in building trust, influence, and leadership.

So why is intuition so hard to automate?

Today’s AI systems are incredibly adept at pattern recognition within narrow domains, but they still struggle to transfer that learning to new context. They can spot tumors or predict equipment failures with superhuman accuracy, but AI can’t intuit the broader implications or device creative solutions on their own. They lack the fluid intelligence and adaptability that comes from a lifetime of diverse experiences.

AI also has a hard time dealing with ambiguity and uncertainty. It thrives on clear rules and statistical regularities, but real-world problems are often ill-defined and open-ended. Human judgment is essential for navigating the gray areas and making sound decisions in the face of incomplete information.

Perhaps most importantly, AI systems lack the rich emotional and social intelligence that underpins human intuition. They can analyze sentiment and detect facial expressions, but they don’t truly understand the subtleties of human interaction. They can’t build rapport, inspire trust, or read between the lines like a skilled human communicator.

That said, AI is getting better at mimicking certain aspects of intuition. Advances in transfer learning and few-shot learning are enabling AI to be more flexible and adaptable. Progress in affective computing and social robotics is imbuing machines with greater emotional intelligence. And techniques like inverse reinforcement learning and imitation learning are allowing AI to infer complex goals and strategies from human behavior.

But even as AI becomes more intuitive, it is unlikely to fully replicate the depth and breadth of human intuition anytime soon. The tacit knowledge and wisdom of experienced professionals is the product of a lifetime of learning, not something that can be easily coded into an algorithm.