By the time AI has the capabilities to be a threat to humanity on its own, it may be our only salvation.
As the velocity of AI abilities increases, so too does the fear that it will go rogue and potentially have the “inclination” to take out humanity—either intentionally or unintentionally (see the paperclip maximizer problem [1]). I’ve read a bit of science fiction and understand the apocalyptic premise of many popular stories. However, the current estimates of a doomsday scenario are all over the place, and almost all seem to place the fear at the doorstep of the AI itself. What they fail to explicitly state, though often imply, is that it’s not the AI that is dangerous; it’s the human in the loop.
As AI gains more intelligence coupled with true, real-world agency, I think fewer and fewer humans will remain in the loop. Over an increasingly compressed timeframe, the number of organizations will increase as the number of people within those organizations dramatically decreases in a non-regulated environment. A concentration of humans controlling an advanced intelligence is scary! That is what I believe we need to be afraid of, and I don’t think we can avoid it.
Superintelligence and Availability
The issue seems to be the “control” of advanced intelligence. Most of the companies that have invested billions into its development want to control it—or at least, control it more than they do now. They want to control (and sell) the gateway to superintelligence, believing they can contain the AI since it resides on their servers and relies on their containment architecture. We’ll see how this pans out. All of this is to say that the human in the loop is the immediate problem, and we are facing an increasing concentration of power in the hands of a few. With that comes the increased potential of power in the hands of bad actors. Moving forward, we are facing critical decisions regarding:
- Who has access?
- Who controls the containment?
- How reliable is this containment when the AI is smarter than its creators?
Long before AI has the ability to “think” and the motivation to do us harm, someone in search of power and control is much more likely to use it with malicious intent. One possible sci-fi scenario is that a superintelligent AI shows up and actually saves the day, stepping in to protect us from ourselves (along with the need for paperclips).
Tilman's claim that AI will lead to smaller but more numerous organizations is strongly supported by current economic and tech leadership theories:
The One-Person Unicorn: Sam Altman (CEO of OpenAI) and Alexis Ohanian (Reddit founder) have both publicly predicted that AI will soon enable the first "one-person billion-dollar company." You can reference Altman's interviews from 2023/2024 where he discusses AI agents executing the work of entire departments.
Coase's Theory of the Firm: For a more academic reference, point to Ronald Coase’s 1937 paper, The Nature of the Firm. Coase argued that companies only grow large because internal coordination is cheaper than hiring outside contractors. AI dramatically lowers the cost of external coordination and task execution, which, economically speaking, guarantees that the average size of a corporation will shrink drastically while the total number of small, hyper-productive micro-businesses explodes.