Why AGI is not like climate change
July 24, 2025
The development of Artificial General Intelligence (AGI) presents a governance challenge of unprecedented difficulty. While many global problems are hard to solve, most share a common strategic structure: with enough time and trust, cooperation becomes the most rational path.
Using game theory, we can see this pattern clearly. We will use ordinal payoffs, where outcomes are ranked from best to worst (4 = Best, 1 = Worst), to assess the incentives for two competing players (A and B) and for Humanity as a whole. Crucially, we will also track the payoff for Humanity as a whole, revealing where individual incentives diverge from the collective good. Payoff = (Player A, Player B, Humanity).
Some well known global problems are a Prisoner’s Dilemma in the short term but transform into a Stag Hunt in the long term, where the best prize for everyone is mutual cooperation. The AGI problem is uniquely dangerous because it breaks this pattern.
The Standard Pattern: From Short-Term Dilemma to Long-Term Cooperation
The core of most global coordination challenges lies in overcoming short-term incentives to secure a better long-term future. The goal of governance is to build the trust needed to make this shift.
Example 1: Overfishing
In the short term, the game is a Prisoner’s Dilemma. The immediate incentive is to overfish for a quick profit, especially if you fear others will do the same. This logic pushes everyone toward a (2, 2, 1) tragedy of the commons.
| Others: Fish Sustainably | Others: Overfish | |
|---|---|---|
| You: Fish Sustainably | (3, 3, 3) | (1, 4, 2) |
| You: Overfish | (4, 1, 2) | (2, 2, 1) - Mutual Ruin |
In the long term, the game becomes a Stag Hunt. A healthy, sustainable fishery is vastly more valuable than any short-term gain. The (4, 4, 4) outcome of mutual cooperation becomes the best prize for the players and for Humanity.
| Others: Fish Sustainably | Others: Overfish | |
|---|---|---|
| You: Fish Sustainably | (4, 4, 4) - Sustainable Bounty | (1, 3, 2) |
| You: Overfish | (3, 1, 2) | (2, 2, 1) |
The entire challenge of fisheries governance is creating mechanisms (treaties, quotas, enforcement) that give players the confidence to aim for the long-term (4, 4, 4) prize.
Example 2: Climate Change
Climate change follows the same pattern on a global scale.
In the short term, it is a Prisoner’s Dilemma. The incentive is to pollute to avoid costly abatement (the “free-rider” problem), pushing nations toward a (2, 2, 1) outcome of a worsening climate.
| Others: Abate Pollution | Others: Pollute | |
|---|---|---|
| Your Nation: Abate Pollution | (3, 3, 3) | (1, 4, 2) |
| Your Nation: Pollute | (4, 1, 2) | (2, 2, 1) - Status Quo Deterioration |
In the long term, it becomes a Stag Hunt. The catastrophic costs of a runaway climate make a stable planet the ultimate prize, making the (4, 4, 4) outcome the most rational goal for all.
| Others: Abate Pollution | Others: Pollute | |
|---|---|---|
| Your Nation: Abate Pollution | (4, 4, 4) - Stable Climate | (1, 3, 2) |
| Your Nation: Pollute | (3, 1, 2) | (2, 2, 1) |
Again, the diplomatic effort is focused on convincing nations to abandon the short-term game and coordinate for the superior long-term prize.
Why AGI is Different: A Permanent Dilemma
The AGI problem is the hardest game of all because it does not transform with the time horizon. The incentive to defect is permanent.
The AGI Payoff Matrix (Player A, Player B, Humanity)
| Player B: Cooperate | Player B: Defect | |
|---|---|---|
| Player A: Cooperate | (3, 3, 4) - Mutual Safety | (1, 4, 2) - Sucker & Temptation |
| Player A: Defect | (4, 1, 2) - Temptation & Sucker | (2, 2, 1) - Mutual Ruin |
Analysis of Preferences:
- The logic is unavoidable: no matter what Player B does, Player A’s best move is to Defect (4 > 3; 2 > 1). This pushes both rational actors toward
(2, 2, 1)—a high-risk race that is the worst possible outcome for Humanity. - Crucially, a long-term view does not change this. The prize for unilateral victory is seen as so absolute that it remains the
(4)temptation forever. There is no “Stag Hunt” to evolve into.
The Nature of the Prize: Why Unilateral Victory is Paramount
This permanent incentive to defect stems from the perceived nature of a true AGI. The “Temptation” is not a temporary advantage; it is a terminal victory.
- Existential Security: The first actor to create a single, controllable superintelligence could enforce its will globally, ending all strategic competition permanently.
- Economic Singularity: The entity controlling the first AGI could achieve absolute and permanent economic dominance.
- The Fear of Irrelevance: The “Sucker” payoff
(1)is not just a strategic setback; it is the risk of your nation, culture, or company being permanently erased or rendered irrelevant. This existential fear makes securing the(4)seem like the only rational path.
The connection to AI alignment
The payoff matrix makes misalignment mathematically clear:
- Humanity’s Best Outcome: The highest payoff for Humanity is (4), which occurs only with mutual cooperation (3, 3, 4). This is the ideal state we want the system to achieve.
- Player’s Best Outcome: However, the incentive structure for each individual player pushes them away from this collective good. For Player A, the highest possible payoff is (4), which occurs if they defect while Player B cooperates (4, 1, 2).
This is why it’s essential for us to align not just AI but also the players ( corporations / nations ) developing AGI! See here for more on this argument.
A Governance Challenge Unlike Any Other
Most global challenges are about convincing actors to sacrifice short-term temptations for a greater long-term mutual reward—to shift from playing a Prisoner’s Dilemma to playing a Stag Hunt.
The AGI problem is terrifyingly different.
It has the Prisoner’s Dilemma structure but with a prize for defecting that is infinitely greater. And it lacks the Stag Hunt transformation that provides a pathway to cooperation in other domains. The rational move for an individual player is to pursue the unilateral prize, even though this collectively leads to a high-risk arms race that is the worst outcome for humanity.
The player’s ultimate goal is fundamentally misaligned with humanity’s, creating a governance challenge unlike any we have ever faced.