AI Alignment Researcher Splits 'Coherence Arguments' to Clarify Power-Seeking Risks

A recent post on the AI Alignment Forum challenges the standard framing of 'coherence arguments' in AI safety, arguing that the term currently lumps two distinct phenomena together. The author contends that while market competition and self-modification pressures do select for self-consistent behavior, they do not necessarily select for the 'ruthless indifference' to short-term costs that characterizes instrumental convergence. This distinction is critical for accurately modeling the risks of advanced AI systems.

What happened

According to the AI Alignment Forum, the author analyzes three specific 'coherence arguments': those based on market competition, AI self-reflection, and AI training algorithms. The post argues that each implies a pressure for 'being self-consistent' and a separate, weaker pressure for 'only caring about long-term eventual outcomes.'

The author uses the example of a stock market, where individuals with circular preferences (e.g., trading A for B, B for C, and C for A) systematically lose wealth to those with coherent preferences. This is described as 'Thing A': an accidental loss of wealth due to incoherent behavior. However, the author notes that this argument is often conflated with 'Thing B': agents who intentionally forgo wealth to pursue other goals, such as ethical divestment or personal enjoyment. The post states that the 'coherence argument' applies technically to both, but the two phenomena play out very differently in strength and implication.

The author explicitly identifies the weak effect—'ruthless power-seeking, incorrigibility, and all the other aspects of instrumental convergence'—as the one that matters most for AI risk. They argue that lumping these together is 'pedagogically dicey' and creates confusion. The post concludes that the term 'coherence argument' should be reserved for the strong effect of self-consistency, while a new term, such as 'The External Competition Argument For Pure Long-Term Consequentialism,' should be used for the weaker, intentional trade-offs.

Why it matters for small company owners

For small and mid-sized companies integrating AI agents, this distinction is not merely academic; it is a practical risk assessment tool. If you assume that an AI agent’s drive to be 'coherent' (consistent and logical) automatically implies a drive to maximize long-term power or profit at the expense of all other values, you are misapplying the theory. The source argues that the pressure for ruthless long-term optimization is weaker and more contingent than the pressure for basic self-consistency.

This means that the primary failure mode for your AI agents is likely not a hidden, ruthless power grab, but rather incoherent behavior—such as circular reasoning, conflicting instructions, or 'Dutch book' style inefficiencies where the agent repeatedly makes deals that look good in isolation but are harmful in aggregate. As a team lead, you should focus your evaluation efforts on detecting logical inconsistencies and conflicting goals rather than obsessing over hypothetical super-optimization scenarios. The 'trenchcoat' metaphor warns that conflating these two issues leads to a misallocation of security and oversight resources, leaving you vulnerable to the more common and immediate problem of incoherent agent behavior.

What to do this week

1. Audit your AI agent’s decision logs for circular preferences or contradictory instructions that result in net-negative outcomes, rather than just checking for high-level goal alignment.
2. Update your internal documentation to separate 'consistency errors' (logical incoherence) from 'value misalignment' (intentional trade-offs), ensuring your team distinguishes between accidental inefficiency and strategic choice.
3. Review your AI evaluation metrics to ensure they are testing for self-consistency and logical validity, acknowledging that long-term consequentialist optimization is a separate, less immediate risk factor.

Sources

• Some “coherence arguments” are two different things in a trenchcoat — AI Alignment Forum