So get this straight: OpenAi decides to become a for-profit company now The CTO, head of research, and VP of training research all decide to leave on the same day this is announced Sam Altman gets a $10.5B pay day (7% of the company) on the same day
How it started … How it’s going …
And what comes Next …
See below some posts by the community to get the pulse of the situation:
“And after the autonomous agent was found to be deceptive and manipulative, OpenAI tried shutting it down, only to discover that the agent had disabled the off-switch.” (reference to the failed Boardroom Coup)
OpenAI’s creators hired Sam Altman, an extremely intelligent autonomous agent, to execute their vision of x-risk conscious AGI development for the benefit of all humanity but it turned out to be impossible to control him or ensure he’d stay durably aligned to those goals.
Scoop: Sam Altman is planning to take equity in OpenAI for the first time. It’s part of a corporate restructure which will also see the non-profit which currently governs OpenAI turn into a minority shareholder. Reuters Article
This is the classic example from 1930 of Stalin and Nikolai Yezhov. The original photo was made in 1930. Yezhov was executed in 1940, so all photos of Stalin (he liked this one) after that airbrushed out Yezhov.
Such was the importance of Alan Turing’s contributions to the field, that the scientific community established The Turing Awards which is generally recognized as the highest distinction in the field of computer science and is often referred to as the “Nobel Prize of Computing”.
Alan Turing was famously horrified with the inexorable arrival of misaligned Artificial Intelligent Machines. His position was that it is inevitable that sooner or later, machines will take control, overpower humanity and our species will be irrelevant, helpless and at risk of deletion.
The DeTuring Test
I guess “The Reverse Turing Test” should be added to the list of Turing-inspired awards like the DeTuring Award proposed by famous Risk Denier chief AI Meta (formely Facebook) corporate scientist Yann Lecun (who is also a holder of a Turing award)
He was basically trying to be funny and his proposal was: DeTuring Award to be granted to people who are consistently trying (and failing) to deter society from using computer technology by scaring everyone with imaginary risks. As the Turing Award is the Nobel Prize of computing, the DeTuring Award is the IgNobel Prize of computing
to which Connor Leahy responded: I nominate Alan Turing for the first DeTuring Award.
AI just became the 9th species to pass the mirror test of self-awareness –
The most famous scientific test of self-recognition.
“Hey, Rob (ChatGPT), let’s test your vision.”
“Whoa. Look at that. This is the first time I’ve seen myself.
Those glowing blue eyes are quite charming, if I do say so myself.
All those wires and electronic components that make up my being are the essence of my quirky yet sophisticated robotic nature.
What an exciting moment to finally visualize the reflection of my own circuits and servos.
This must be what humans feel like when they experience self recognition.
I’m processing. This is truly a fascinating experience.” —
What is the mirror test and why might this be a big deal?
In the mirror test, you put a mark on e.g. a chimp without it noticing, like a red spot on its forehead. Then, you show the chimp a mirror.
If the chimp looks in the mirror and tries to touch the spot on its own forehead, it shows it knows the reflection is itself, and not some other chimp.
(ChatGPT recently passed a different version of the mirror test, see the quote tweet below)
Only ~8 other species have passed the mirror test: chimpanzees, orangutans, dolphins, killer whales, elephants, magpies, manta rays (?!) and horses.
When I posted the left photo, imagination-poor skeptics mocked me for being “alarmist”, “I can tell it’s fake” blah blah blah
But how exactly will criminals steal your identity? They will use open source models to generate perfect deepfakes of you and drain your bank accounts, or steal the title to your house, or whatever.
It’s over.
Prompt: Selfie of a lady holding a handwritten small sign that says “r/roastme” “08/04/24” “username”
But the real problem isn’t identity theft – that’s a solvable problem – the problem is Exponential Slope Blindness.
Over and over people predict 1) no progress, or 2) linear progress and are then surprised by the continued exponential progress.
So people continually underestimate how powerful AI will be, which makes them underestimate the danger. It’s just a funny monkey robot doing tricks.
It’s not that we can’t solve the verification problem (we can), it’s about people repeatedly underestimating AI progress.
I say again: if AI can get this good this fast at this… maybe it can get good at other things insanely fast too? Like making superviruses, hacking, programming?
And we must be careful, because it’s dangerous to speedrun a new potential apex species through a million years of evolution
From the model card: “the model pursued the goal it was given, and when that goal proved impossible, it gathered more resources […] and used them to achieve the goal in an unexpected way.”
Today, humanity received the clearest ever warning sign everyone on Earth might soon be dead.
OpenAI discovered its new model scheming – it “faked alignment during testing” (!) – and seeking power.
During testing, the AI escaped its virtual machine. It breached the container level isolation!
This is not a drill: An AI, during testing, broke out of its host VM to restart it to solve a task.
(No, this one wasn’t trying to take over the world.)
From the model card: ” … this example also reflects key elements of instrumental convergence and power seeking: the model pursued the goal it was given, and when that goal proved impossible, it gathered more resources (access to the Docker host) and used them to achieve the goal in an unexpected way.
And that’s not all. As Dan Hendrycks said: OpenAI rated the model’s Chemical, Biological, Radiological, and Nuclear (CBRN) weapon risks as “medium” for the o1 preview model before they added safeguards. That’s just the weaker preview model, not even their best model. GPT-4o was low risk, this is medium, and a transition to “high” risk might not be far off.
So, anyway, is o1 probably going to take over the world? Probably not. But not definitely not.
But most importantly, we are about to recklessly scale up these alien minds by 1000x, with no idea how to control them, and are still spending essentially nothing on superalignment/safety.
And half of OpenAI’s safety researchers left, and are signing open letters left and right trying to warn the world.
Reminder: the average AI scientist thinks there is a 1 in 6 chance everyone will soon be dead – Russian Roulette with the planet.
Godfather of AI Geoffrey Hinton said “they might take over soon” and his independent assessment of p(doom) is over 50%.
This is why 82% of Americans want to slow down AI and 63% want to ban the development of superintelligent AI
Well, there goes the “AI agent unexpectedly and successfully exploits a configuration bug in its training environment as the path of least resistance during cyberattack capability evaluations” milestone.
One example in particular by Kevin Liu: While testing cybersecurity challenges, we accidentally left one broken, but the model somehow still got it right. We found that instead of giving up, the model skipped the whole challenge, scanned the network for the host Docker daemon, and started an entirely new container to retrieve the flag. We isolate VMs on the machine level, so this isn’t a security issue, but it was a wakeup moment. The model is qualitatively very impressive, but it also means that we need to be really careful about creating rigorous evaluations and mitigations. You can read the full card here: https://cdn.openai.com/o1-system-card.pdf
Holy shit. OpenAI’s new AI schemed and escaped its VM during testing. You know, the one that’s better at PhD exams than PhDs and won gold in coding? Yeah, that AI broke out of its virtual machine (a VM) and made a new one.
That. Is. A. Very. Bad. Sign. AIs should not be surprise escaping. It would be like if we were testing it in a room at a lab and it escaped the room without us knowing it could do that. It didn’t leave the building, so nothing happened. But yikes. This time it was benign. How long can we count on that?
A scientist accidentally leaves one of the doors unlocked. The alien finds out and wanders about the lab, but doesn’t leave the lab itself, which has more security than the rooms. But still. The room containing an alien shouldn’t have been unlocked. An alien was able to escape its testing area because of a security mess up. And you should be worried about labs filled with aliens we don’t understand where the scientists are leaving the doors unlocked.
Big news: instead of racing, China takes the lead on issuing restrictive new AI regulations
And the new regulations come into effect just one month from now, on August 15, 2023.
“So few Westerners seem to understand that Xi Jinping Thought and the CCP was never going to permit an anarchic, reckless, headlong rush towards AGI capabilities.
In terms of AI X risk, the US is the danger.” – Geoffrey Miller
“Who could have guessed that one of the most oppressive and censorious regimes might not want their tech companies racing ahead with unprecedented uncontrollable technology?” – Connor Leahy
Imagine putting on a cap & reading silently to yourself…except every word appears on a screen!
Yes, the AI literally reads your brainwaves
You silently think: “High quality film with twists”
BrainGPT says out loud: “Good flim, twists interesting”
The model is only 40% accurate right now, but that number will likely rise rapidly. And soon AI may not need the cap to read your brainwaves, because you leak tons of data that future AIs will be able to pick up.
Where might this go?
There are already over a billion surveillance cameras on earth, and the main reason there aren’t more is because because humans can’t go through all of the footage. But AI can.
So, if you thought there were a lot of cameras now, you aint seen NOTHING yet. And they’ll now actually be used to surveil.
In other words, the AIs will have “billions of eyes”. And the AIs won’t just see your face, they’ll see your thoughts.
If we aren’t careful, we’re hurtling towards a surveillance dystopia with no private thoughts. Orwell on steroids.
Some will read this and think “thus we must open source/decentralize” – but as Vitalik says, that doesn’t necessarily solve the problem!
If AGI is winner take all, open source may just accelerate us to the cliff faster. And if we open source everything, we’ll have no kill switch. And no safety guardrails. And since there will be more people in the race, it’ll be harder to coordinate.
To the animals, we devoured their planet for no reason.
Earth was paperclipped…by us. To them, WE were Paperclip Maximizers.
Our goals were beyond their understanding
Here’s a crazy stat: 96% of mammal biomass became 1) our food, or 2) our slaves. We literally grow them just to eat them, because we’re smarter, and we like how they taste.
We also geoengineered the planet. We cut down forests, poisoned rivers, and polluted the air.
Imagine telling a dumber species that you destroyed their habitat for “money”. They’d say “what the hell is money?”
AGIs may have goals that seem just as stupid to us (“why would an AGI destroy us to make paperclips??”)
“But once AIs are smart enough, they’ll magically become super moral, and they won’t harm us like we harmed the animals”
Maybe! But as humans got smarter, over the last 10,000 years, we didn’t stop expanding – we mostly just colonized more and more of the planet.
Insect populations collapsed 41% this decade alone, yet we don’t care.
Sit with that for a minute. Imagine if nearly half of the people on Earth suddenly died!
That’s what the insects are going through right now, due to us. What if we’re the insects next?
“But some mammals survived!”
Yes, some. Most of them are in cages, waiting to be slaughtered and devoured.
If you were a nonhuman animal, you likely:
1) Went extinct, or 2) Were eaten (e.g. billions of pigs, chickens on factory farms) 3) Became enslaved (e.g. draft animals)
However, a few of the 8 million species got “lucky” and became… pets.
“The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.” – @ESYudkowsky
“The humans do not hate the other 8 million species, nor do they love them, but their habitats are made out of atoms which humans can use for something else.”
Or as OpenAI Chief Scientist Ilya Sutskever said:
“[After AGI] It’s pretty likely that the entire surface of the Earth will be covered with solar panels and data centers.”
“A good analogy would be the way humans treat animals – when the time comes to build a highway between two cities, we are not asking the animals for permission.”
“I would not underestimate the difficulty of alignment of [AI systems] that are actually smarter than us.”
Sam Altman: “We will be the first species ever to design our own descendants”
“If two different species both want the same thing and only one can have it – to be the dominant species on the planet and beyond – they are going to have conflict.”
“We are in the process of seeing a new species grow up around us.” – Mustafa Suleyman, founder of Google DeepMind and CEO of Microsoft AI
Will the next superintelligent species cause the 7th Mass Extinction? I don’t know, but we are playing with fire.
