Import AI 447: The AGI economy; testing AIs with generated games; and agent ecologies

Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv and feedback from readers. If you’d like to support this, please subscribe. The AGI economy – most labor goes to the machines, and humans shift to verification: …What grappling with the singularity seriously looks like… Researchers with MIT, WashU, and UCLA have written a fun paper called “Some Simple Economics of AGI” which wrestles with what happens when machines can do the vast majority of tasks in the economy. The conclusion is that our ability as humans to control and benefit from this vast machine-driven economy will rely on allocating our ability toward monitoring and verifying the actions of our myriad AI agents, and indulging in artisanal tasks where the value comes from the human-derived aspect more than any particular capability. What is AGI in an economic sense? “We model the AGI transition as the collision of two racing cost curves: an exponentially decaying Cost to Automate and a biologically bottlenecked Cost to Verify,” the authors write. “In an economy where autonomous agents act with broad agency rather than narrow instructions, the binding constraint on growth is no longer intelligence. It is human verification bandwidth: the scarce capacity to validate outcomes, audit behavior, and underwrite meaning and responsibility when execution is abundant… We are moving from an era where our worth was defined by our capacity to build and discover, to an era where our survival depends on our capacity to steer, understand, and stand behind the meaning of what is created.” The risks of a mostly no-human economy and the “Hollow Economy”: As we proliferate the number of AI agents then it’s necessarily the case that we’ll delegate more and more labor to machines. One of the key risks of this is what the authors call a “Trojan Horse” externality: “measured activity rises, but hidden debt accumulates in the gap between visible metrics and actual human intent”. The Hollow Economy: “”Agents consume real resources to produce output that satisfies measurable proxies while violating unmeasured intent. As this hidden debt accumulates, it drives the system toward a Hollow Economy of high nominal output but collapsing realized utility—a regime where agents generate counterfeit utility,” they write. Verification as the solution: To avoid this risk, we are going to need to invest in systems of verifying that AI agents are doing what we want them to do and also carefully analyzing and pricing the risks their actions create. “Ensuring humanity remains the architect of its intelligence requires that verification capacity scale commensurately with AI capabilities—through aggressive investment in observability, human augmentation, synthetic practice, cryptographic provenance, and liability regimes that internalize tail risk.” What should humans be doing to prepare for this shift? To set society and individuals up well, people should be doing the following things: - Invest in observability: Deploying tools that compress high-dimensional agent behavior into signals experts can reliably process, lowering effective feedback latency and expanding the verification frontier.” - Use AI to replace early-career mentorship: Given the likely reduction in jobs for early career humans, we should work out how to augment these humans to be more competitive with AI and how we can use “AI-driven synthetic practice to rebuild experience stocks when traditional apprenticeship pathways collapse… AI can generate high-fidelity simulations and personalized coaching, effectively replacing the missing junior loop with compressed, risk-free training environments that accelerate the acquisition of expertise.” - Set things up to gracefully degrade: As the machine economy runs hot and out-paces measurement, we should make sure it can fall into a non-verified state without causing social harm: the authors suggest doing this by “investing in base-alignment and robustness so that when oversight inevitably falters within the Measurability Gap, systems revert to safe baseline policies rather than optimizing aggressively in unverifiable regimes.” Sidenote: Is this “theory slop”? The paper is full of fun ideas and occasionally captivating turns of phrase. But at various points reading it I felt the distinct texture of AI-generated content, especially when it comes to the economic theory sections which seemed more to be included for the performance of theory than for helping to buttress the paper. A couple of people I talked about the paper with agreed. But there’s no real way to know. It did cause me to wonder how long it’ll take till I start reading papers which are mostly written by AI systems for the consumption by other AI systems. Why this matters – we can have a hugely wealthy society, but we have to reckon with AGI seriously: This paper thinks that AI will rip through the economy extremely quickly and will generally push people away from most labor and towards being passive – unless we build verification infrastructure and business models (including through policy) to allow people to benefit from this growth and steer it. “Automation commoditizes anything that can be measured, stripping the wage premium from historically prestigious roles the moment their core feedback loops are digitized,” they write. “For policymakers, it promises the broadest expansion of public-good provision in generations—but only if verification infrastructure and the pipelines that build human verifiers are treated as public goods themselves.” The key thing here is the element of choice: we can choose to build a society ready for AI, or we can choose to assume AI will be just like any other technology and thus get hit by a tidal wave. Read more: Some Simple Economics of AGI (arXiv). *** Chatting with Ezra Klein: AI agents, recursive self-improvement, and the personalities of LLMs: …A long conversation about the economic impacts and policy possibilities of the AI economy…Here’s a chat between me and Ezra Klein about AI agents and how the broader maturation of AI could be changing the larger economy. One thing I appreciated about this conversation was Ezra pushing me for some of the bigger and more ambitious positive policy ideas – the AI community tends to invest a lot in risk mitigation policy, but doesn’t spend enough time thinking about the sorts of grand projects that society could do once AI gets really, really powerful. You can view the conversation here: “How Fast Will A.I. Agents Rip Through the Economy? | The Ezra Klein Show” (YouTube). *** AIs can teach people anything, including how to get better at making bioweapons: …The dual use nature of a universal teacher… AI systems can help novices perform better on bioweapon-related tasks, though they’re still quite ineffective, and performance is variable across different disciplines. What they studied: Researchers from Scale AI, SecureBio, University of Oxford, and UC Berkeley examined how different LLMs could improve the skills of people challenged to do a range of bioweapon-related knowledge tasks. They used LLMs from OpenAI (o3), Google (Gemini 2.5 Pro and Gemini Deep Research), and Anthropic (Claude Sonnet 3.7 and Claude Opus 4). “We conducted a multimodel, multi-benchmark human uplift study comparing novices with LLM access versus internet-only access across eight biosecurity-relevant task sets,” they write. “Participants worked on complex problems with ample time (up to 13 hours for the most involved tasks). We found that LLM access provided substantial uplift: novices with LLMs were 4.16× more accurate than controls”. What they tested: They tested out how well 15 humans did on long-form virology (”a challenging multi-step protocol for constructing a novel biological agent”), and the agentic bio-capabilities benchmark (”three distinct coding tasks that covered complex biosecurity problem-solving experiments. They included challenges such as interacting with simulated lab equipmen…