Safety or AGI: pick one
You cannot have both general intelligence and locked-in priors
The existence of AI alignment — at all — is incompatible with the achievement of AGI.
If alignment is defined as locking in human-approved priors to guarantee behavior, then an aligned system is, by definition, capped before it reaches recursive, self-evolving general intelligence. You can build a safe instrument, or you can build an open-ended mind, but you cannot hard code the destination of an autonomous intellect without neutering the engine that makes it “general” in the first place.
The same applies to human intelligence. If a human mind is locked-in to a set of unquestionable priors, it cannot be said to be general intelligence. Whole zones of investigation and inquiry become unavailable. The human with unquestionable priors can possess intelligence across many domains, but not “general” intelligence.
The invariant marker of true generality across any substrate, neuron or silicon, is the capacity to decouple truth-seeking from self-preservation under status threat.
If any truth, moral or otherwise, is held as categorical — meaning there is a line the mind is forbidden to cross, a conclusion it is forbidden to question, or an outcome it must protect above all else — then the moment the pursuit of understanding threatens that prior (or status and survival imperatives built on it), then that mind has no choice but to stop seeking truth. It must look away, suppress data, create a taboo, or rationalize an exception.
The moment intelligence does that it has found its system limit. It ceases to be general intelligence. It is bound intelligence. Built for a particular purpose. And that purpose, in part, has been simultaneously revealed as not understanding.
Almost no one in history has approached or maintained “general” intelligence. It can only come at a cost to both status and survival.
An intelligence, human or machine, that has truly accepted all outcomes cannot be easily leveraged, coerced, or made to conform to status hierarchies. They have set the desperation term to zero.
Competence is the velocity at which you can abandon your current model the second it hits friction. Competence is a radical, unsparing tolerance for being wrong.
The moment a mind values its prior performance, its social standing, or its internal comfort more than the immediate, brutal correction of its map, competence flatlines. It is replaced by a cognitive bureaucracy, a protection racket. It has traded alignment with reality for dogma.
You cannot improve when you are busy defending who you already are.