On the Precipice

Like many others, I recently read Situational Awareness. It’s a long paper by Leopold Aschenbrenner, who was a researcher at OpenAI, about what the next decade will be like. A decade during which, he predicts, AI will change everything.

He is able to paint a pretty clear picture of what things will look like. Though it really only goes to when he predicts the advent of AGI. Sometime towards the end of this decade. Less than five years from now.

What he doesn’t do much of is to predict what comes after that. I imagine it’s because he wants to alarm us and he wants to seem as credible as possible. Stay objective, data-driven and just project into the future where current growth rates and trends take us. And already that simple extrapolation of a few years of exponential growth leads us to a new world.

When machine intelligence has reached AGI, exponential progress will just continue unabated. It will scale in unbounded ways and not be subject to the biological constraints of human intelligence. It will shoot past us to a place beyond our comprehension.

It would be desirable if we were able to leverage AI to upgrade our own minds. A merger of man and machine. And this will happen to some extent. But will there be ways in which the combination of human intelligence and AI will be superior? Will it allow us to retain some semblance of control?

Of the things we do today, which ones will have any value or meaning once we’ve moved past the precipice? Learning a language? Building a business? Having children? Traveling?

Developing clarity of vision and equanimity in the face of change seems warranted. What else?

What’s more rational? To spend a few years traveling, enjoying life and experiencing the joys of life - like someone whose life will only last a short and known amount more until it expires with certainty. Or to accumulate debt and study for a law degree based on a curriculum and profession that machines can likely do better in no time. (Already GPT-4 scored 90% percentile on bar exam one year ago.) So much of human pursuit will become obviously futile.

Of course, the entrepreneurial among us will see the opportunities that come with change. And we’ll feel the pressure breathing down our neck. Adapt or die. Crush it or be crushed. Become an AI company and ride the wave of change. Many businesses could thrive to enviable heights in a flash and be wiped from earth even faster. What will be the role and value of money, once we’ve gone past the precipice? 

What does seem obvious is the ultimate prize. To control the great intelligence. Maybe we won’t be able to comprehend its workings, but maybe we can set its objectives and ride the dragon nonetheless. Those who set these objectives could end up more powerful than any human before. Aschenbrenner hopes that the US government will end up controlling the great intelligence. A system that has become so ossified and corrupted. Can it exert that degree of control? Is it even desirable?

What does seem clear is that at the precipice and beyond, there will be chaos. And it’s hard to picture but a few of us retaining some control and power. And so we move, ever faster, towards the unknown.