Rethinking How We Measure the Value of AI Research

This post explores and analyses the article ‘How to track the economic impact of public investments in AI’ by Julia Lane and colleagues. The article was published in 2024.

Introduction

Governments and corporations worldwide are investing unprecedented sums in artificial intelligence research. The United States federal government alone committed over $3 billion to AI in 2023, while private sector investments reach hundreds of billions annually.

These massive investments rest on a fundamental assumption: AI will transform economies and generate sustained job growth. Yet a critical question remains unanswered: How do we measure whether these investments are actually paying off?

As Lane, Owen-Smith, and Weinberg argue in their Nature article, our traditional statistical systems fail to capture AI’s true economic impact. When Texas applied the newly created NAICS classification for “AI research and development laboratories,” they identified only 298 firms employing just 1,021 workers — a dramatic undercount of AI’s actual footprint.

The fundamental problem is that AI doesn’t operate within conventional industrial boundaries. AI talent and applications permeate virtually every sector of the modern economy, from healthcare to finance, manufacturing to entertainment. This cross-cutting nature makes AI part of what economists now call the “hard to measure” economy — which may constitute up to four-fifths of advanced economies. The authors propose a paradigm-shifting alternative: instead of tracking AI through industry codes or counting publications, we should follow the people who embody AI knowledge as they move from research settings into the broader economy.

This people-centred approach recognises that in emerging knowledge-intensive fields like AI, human capital — not machines or infrastructure — is the primary vehicle through which value is created and diffused.

This shift from measuring outputs (publications, patents) to tracking knowledge flows through people represents a fundamental rethinking of how we evaluate scientific impact. It’s a perspective that, as we’ll explore, has implications far beyond just AI research.

Key Insights from the Article

Beyond Industry Codes: Why Traditional Metrics Fail AI

Traditional economic accounting methods fundamentally struggle with research-led fields like AI. The NAICS classification system’s single AI category captures only a tiny fraction of the actual AI workforce. This measurement challenge isn’t unique to AI — it’s an issue with all emerging technologies that diffuse across multiple sectors rather than creating their own distinct industry.

This measurement gap creates a critical dilemma for policymakers: they cannot effectively manage what they cannot measure. In rapidly evolving fields like AI, measurement becomes even more crucial for informed decision-making and accountability for public investments.

Following the Knowledge: A People-Centred Solution

The authors propose an alternative approach focused on the individuals driving AI innovation. Their insight is refreshingly straightforward: in emerging industries where ideas matter more than physical infrastructure, people are the primary value-creators.

The approach works by:

1. Identifying AI researchers: Start with academics who present at major AI conferences and also receive government grants to form a “seed set” of principal investigators.
2. Expanding the network: Use university administrative data (UMETRICS) to track everyone connected to these researchers — students, staff, and vendors.
3. Following career paths: Track these individuals as they transition into private-sector employment using state employment records.

What makes this approach powerful is its ability to map how knowledge diffuses through the economy via human capital. When a doctoral student trained in an AI lab joins a healthcare company, they bring cutting-edge “AI know-how” with them. This captures a knowledge transfer that traditional metrics would miss entirely.

Early Findings

Using this methodology, the authors found that AI science investments affect more than 36 million US workers employed across 18 different sectors. This is dramatically higher than conventional metrics suggest.

Perhaps most interestingly, they found that people employed in AI industries tend to earn more on average than those who are not. This wage premium represents a quantifiable return on the initial research investments, providing a concrete way to measure impact.

The Ohio pilot implementation is working to provide even more granular insights, such as whether firms hiring AI scientists pay higher wages to all employees and enjoy faster earnings growth compared to other companies.

Broader Applications

While the article focuses on AI, the authors emphasise that their framework can be applied to other fields of scientific research. The fundamental insight — that in many emerging fields, people are the main value-creating unit — applies broadly.

The approach is also scalable internationally, as it leverages standard administrative data used for human resources and tax purposes. Similar data infrastructures exist in many countries, making this potentially a global solution for measuring innovation impact.

Building the Infrastructure: From Concept to Implementation

What’s particularly promising about this approach is that much of the necessary data infrastructure already exists. The UMETRICS system captures information on more than 580,000 grants supporting nearly 1 million employees and 1.2 million vendors.

By linking this grant data with state employment records, researchers can track knowledge flows from academia to industry without creating entirely new data collection systems. The code to collect, integrate, and analyse this data could be standardised and reused across organisations.

The article notes that this vision is gaining traction. The National Artificial Intelligence Research Resource (NAIRR) task force has recommended this people-centred evaluation approach in their final report to President Biden and Congress.

Rethinking Scientific Value: People as Knowledge Carriers

The Limits of Bibliometrics in the Knowledge Economy

Traditional bibliometric measures face serious limitations in capturing how scientific knowledge actually creates economic value. Citation counts tell us about academic influence but reveal little about practical applications. Patent counts may indicate potential for commercialisation but miss the broader diffusion of ideas.

The wage premium identified for workers with AI experience offers something bibliometrics cannot: a market-based valuation of knowledge. When employers are willing to pay more for individuals with specific research experience, they’re signalling the economic value of that knowledge in a way that publication metrics simply cannot capture.

Rethinking Value in Scientific Research

What struck me most about this article is how it challenges traditional notions of measuring scientific value. By focusing on individuals as knowledge carriers, we acknowledge that the most valuable output of research isn’t always papers or patents, but trained experts who apply specialised knowledge in diverse settings. This aligns with how innovation actually spreads in practice — less through reading academic papers and more through hiring people with relevant expertise.

The method proposed by Lane and colleagues has several compelling strengths:

First, it captures cross-sectoral impact that traditional metrics miss. AI isn’t just changing computer science — it’s transforming healthcare, finance, manufacturing, and practically every other industry. Following people as they move between sectors provides a more accurate picture of this diffusion.

Second, it offers quantifiable economic metrics through wage data. The wage premium for workers with AI experience provides a concrete measure of the market value of that knowledge. This gives policymakers a more tangible return-on-investment metric than citation counts ever could.

Third, it acknowledges the importance of training and human capital development in research funding. This approach recognises that educating the next generation of scientists and practitioners is a central purpose of research funding, not just an incidental benefit.

Balancing Multiple Dimensions of Scientific Impact

While the economic lens is valuable, we must be careful not to reduce scientific value solely to economic metrics. Scientific research serves multiple purposes — advancing fundamental knowledge, addressing societal challenges, and creating economic value. The ideal evaluation system would balance these dimensions rather than privileging any single aspect.

The authors’ approach excels at capturing cross-sectoral impact and providing quantifiable economic metrics. However, it may undervalue research contributions that don’t immediately translate to workforce outcomes — such as theoretical advances that take decades to find practical applications.

Practical and Ethical Challenges of Implementation

Implementing this people-tracking approach raises important practical and ethical questions. Privacy concerns are significant — tracking individuals’ career paths requires careful attention to data protection and consent. The authors mention privacy-preserving features, but the details of implementation matter greatly.

There are also equity considerations. Will this approach inherit existing biases in research credit attribution? Could it disadvantage fields with less direct economic applications? These questions require thoughtful engagement as we develop new ways to measure scientific impact.

The Debate Horizon: Key Tensions in Measuring Scientific Impact

The article’s approach to measuring AI’s economic impact raises several contentious issues worth exploring:

Prioritising Economic Returns vs. Scientific Discovery

Perhaps the most fundamental tension is whether science should be measured primarily by its economic returns. The article’s approach might inadvertently shift funding priorities toward research with clear commercial applications, potentially at the expense of basic science. While economic impact is important, should it be the dominant metric? Or does this approach risk undermining the curiosity-driven research that often leads to breakthrough discoveries?

While the value of understanding economic returns, especially when public funds are involved, is widely recognised, there’s something uncomfortable about reducing scientific research to workforce outcomes and wage premiums. Some of history’s most transformative discoveries came from research with no obvious economic applications at the time.

Privacy and Surveillance Concerns

The data linkage required for this approach raises significant privacy questions. While the authors briefly mention privacy-preserving features, they don’t address the fundamental tension between tracking individuals’ careers and respecting their autonomy. Do researchers and students implicitly consent to having their careers tracked when they accept grant funding? Should there be limits on how granular this tracking can be?

We should be particularly concerned about how this data could be misused. Could information about which companies hire AI researchers become a competitive intelligence tool? Might this tracking create a chilling effect on career mobility if researchers know they’re being monitored?

Equity and Inclusion Implications

Another controversial aspect is whether this measurement approach might exacerbate existing inequities in research funding. Fields with clear commercial applications already tend to receive more funding than those without. If economic impact becomes even more central to funding decisions, will that further disadvantage historically underfunded areas?

This concern extends to individuals as well. Research shows that women and minorities are less likely to be credited for their academic contributions. Would a people-tracking approach inherit these biases, or might it actually help recognise contributions from those who don’t end up as lead authors on publications?

Academic Freedom and Research Direction

The article’s approach could fundamentally change how research priorities are determined. If funding increasingly follows economic impact potential, will that reduce academic freedom to pursue diverse research directions? Could we end up with a more homogenised research landscape focused primarily on commercially promising areas?

This raises questions about who should decide research priorities. Should it be market forces, as reflected in employment and wage data? Government strategic priorities? Or should individual researchers retain significant autonomy in determining promising directions?

International Competition and Cooperation

Finally, I’m curious about how this measurement approach might affect international research dynamics. Countries with more sophisticated data infrastructure might gain advantages in directing their AI investments. Could this create new forms of competition, where nations race to build better research impact tracking systems?

Conversely, could standardised approaches to measuring research impact facilitate more international cooperation by creating common metrics? Would shared measurement systems help coordinate global research efforts on pressing challenges like climate change or pandemic preparedness?

These contentious issues don’t have simple resolutions, but discussing them openly will help us develop more nuanced perspectives on how we should measure and value scientific research in AI and beyond.

From Measurement to Management: The Path Forward

The authors have offered a compelling new paradigm for measuring how scientific investments create value. By tracking knowledge flows through people rather than publications, we gain a more accurate understanding of how research investments shape our economy.

This approach has immediate implications for multiple stakeholders:

For funding agencies, it suggests the need to evaluate grants not just on publication outputs but on workforce development outcomes. How many people were trained? Where did they go? What economic value did they create?

For universities, it highlights the importance of tracking graduate outcomes beyond academia and developing more integrative data systems that connect research funding to career trajectories.

For policymakers, it offers more concrete metrics to justify continued investment in scientific research, while providing a clearer picture of how knowledge diffuses across economic sectors.

Important questions remain about implementation. How can we balance the need for data with privacy protections? How should we account for scientific contributions that don’t translate directly to economic value? Can this approach be implemented internationally while respecting different data privacy regulations?

As AI continues to transform our economy, having better ways to measure its impact becomes increasingly crucial. The people-centred approach represents a promising step toward more evidence-based science policy — one that recognises that in our knowledge economy, human capital is often the most valuable output of scientific research.

The next critical step will be moving from measurement to management — using these insights to shape more effective science policies that maximise both economic returns and broader societal benefits from our substantial investments in AI and other emerging technologies.

The Medium version of this post can be found here.

References

• Lane, J., Owen-Smith, J., & Weinberg, B. A. (2024). How to track the economic impact of public investments in AI. Nature, 630(8016), 302–304. https://doi.org/10.1038/d41586-024-01721-1