Sometimes, the best way to explain something is by telling a simple story!
The Sorcerer’s Apprentice
The Sorcerer’s Apprentice (By Disney 1940) Works as a perfect and eerie analogy for the kind of threat AGI actually poses, the only difference being that there will be no happy ending for humanity as we won’t have a master wizard that wakes up in the morning to clean up our mess.
Life imitates art: Don't Look Up
Don’t Look Up … but AGI instead of comet
— Elon Musk (@elonmusk) April 1, 2023
The Titanic
The unsinkable ship
that sank during its maiden voyage.
GPT-5
— Ori Nagel ⏸️ (@ygrowthco) August 13, 2024
Coming soon... pic.twitter.com/nPWmFteUvU
King Midas and the Golden Touch
The Myth of King Midas is an ancient example of Specification Gaming, the problem of translating a wish that exists in the realm of the mind into the realm of an optimising force operating in the physical world (In the myths that can be the Greek Gods or some other supernatural entity like a Genie and in the future that can be the AGI) In some variations of the legend, King Midas managed to lift his curse in the river of Pactolus. However, according to Aristotle, Midas eventually died of starvation as a result of his “vain prayer” for the gold touch, the curse never being lifted.
The bad ending works as a much better analogy to AGI, as the Corrigibility Open Problem tells us that a property of Artificial Generally Intelligent Optimisers is to resist modifications to the specification, therefore, unless alignment is solved first, there will be no Pactolus river for humanity.
