[SolarCross]

The Daemon is the advent of Artificial Super Intelligence which Elon Musk and many others fears will very likely be an effectively unfriendly god of such unstoppable and ever-increasing power that it poses real existential risk for our whole species and all life on earth and in probability any life beyond earth. Most experts believe it could arise within the next few decades so it really is an immediate problem far more serious and immediate than climate change, potential global thermo-nuclear war, a supernova event in a galactic backyard, a rogue super-asteroid collision or alien invasions or any of the other seriously big potential threats to mankind.

If you need a primer on what the hell Artificial Super Intelligence is and how it suddenly became an existential risk then read here:

The AI Revolution: The Road to Superintelligence
The AI Revolution: Our Immortality or Extinction

Okay so now you know something about the Daemon and why we should pretty reasonably expect it to present an existential risk to our species in the relatively near future you can get to know the best solution I have heard so far for taming it so that it does not do very bad things to us. The answer may be "Wizard Hats" and you can read about them here (it is very long but essential reading so read it all):

Neuralink and the Brain’s Magical Future

Okay now you know about the Daemon and the Wizard Hat we can discuss.

For bonus points you can read about what this all means in the context of Fermi's Paradox here:

The Fermi Paradox

[One Degree]

Sorry, not concerned enough to want to read all the links. Do they explain how an AI obtains the emotions to care enough to want to control anything? If so, please direct me to that one.

[Sivad]

Sorry, not concerned enough to want to read all the links. Do they explain how an AI obtains the emotions to care enough to want to control anything? If so, please direct me to that one.

I'm skeptical of strong AI too, but even weak AI is incredibly dangerous and that's already here. Weak AI is going to lead to fully autonomous weapons systems which could free the power elite from all democratic constraints. It would allow those that control it to covertly manipulate extremely complex systems(markets and geopolitics) with high precision. It could be used to crack intractable problems in science and technology which could lead to anything from an unprecedented arms race in novel WMDs to a full on breakaway civilization.

[Sivad]

what this all means in the context of Fermi's Paradox

The Transcension Hypothesis: An Intriguing Answer to the Fermi Paradox?
John Smart, of the Accelerating Studies Foundation, has proposed a novel idea which suggests a rather more exciting and stranger fate for intelligence than previous conceptions. In the Transcension Hypothesis, he suggests that sufficiently advanced civilisations may invariably leave our universe by using and eventually relocating to black-hole-like destinations! Bizarre as this notion may initially sound the suggestion is backed by considerable research drawn from fields as diverse as biology, physics, computer science, information theory and sociology, with a series of falsifiable claims which will become testable in the coming decades.

[One Degree]

@SolarCross
Sorry, I probably came across more acerbic than intended. I love reading about science breakthroughs and science fiction, but I have zero interest in apocalypse scenarios. I rate them with mummies, the undead, etc.
We will stumble through until we don’t, but I simply can’t envision AI being our final end or even a serious threat.
Wrote before your last post. That gets me interested, so I will read it.

[Zamuel]

I'm skeptical of strong AI too, but even weak AI is incredibly dangerous and that's already here. Weak AI is going to lead to fully autonomous weapons systems which could free the power elite from all democratic constraints. It would allow those that control it to covertly manipulate extremely complex systems

Duh ? Fully Autonomous systems that are "under control" ... Double Duh ...

Zam

[Sivad]

Oh, you're a sharp one.

[One Degree]

I don’t have the knowledge to challenge the reasoning. My first impulse was to call ‘intellectual babble’, but he does strike a chord with me on becoming unified with the universe on a micro level. I fully expect, in the very near future, we will find our brains are communicating with one another and ‘other things’ in a method we currently don’t recognize. This would go along with his idea of a very rapid change after some initial discoveries. I think he is headed to the right destination, but his knowledge of the path might be distorted based upon being human centric and dependent upon current knowledge.
I base all the above on nothing other than my innate ability to perceive the truth of the universe with my own imagination. Anyway, thanks for exposing me to an interesting idea.

[SolarCross]

Sorry, I probably came across more acerbic than intended. I love reading about science breakthroughs and science fiction, but I have zero interest in apocalypse scenarios. I rate them with mummies, the undead, etc.
We will stumble through until we don’t, but I simply can’t envision AI being our final end or even a serious threat.
Wrote before your last post. That gets me interested, so I will read it.

No problem, I completely understand where you are, I even have a name for it "End Times Fatigue Syndrome". People have been worrying about the end of the world for as long as there has been a world and people, yet except for the odd minor disaster nothing ever comes of it. The number of End Times scenarios being punted about even seems to have increased in our times: Late Capitalism, Peak Oil, Climate Change, Y2K bug, Mayan Calender Ending, Planet Niburu... So we have heard it all before, nothing ever came of it and decided not to get worked up about the next shit being peddled about the sky falling. ETFS, End Times Fatigue Syndrome it is a thing now.

On your question as to where "they get the emotions to want to kill us". The article covers this but this is my answer: Where do you get your emotions? What are emotions? You get them ultimately from hundreds of millions of years of evolution and they are general purpose goal orientated programs. See danger -> produce adrenaline -> run or fight = fear. See friendly co-agent -> produce Oxytocins -> hug and kiss to communicate and reaffirm friendly co-agent status. These programs are ultimately in service to the greater evolutionary goal of survive and reproduce, that they prove to help this goal is why they continue to persist within those biological systems.

AI is not evolving haphazardly by natural selection it is evolving by our intelligent design (which is why it is happening so fast) and they are being designed to serve us, as in mankind or sub-sets of us. So they won't necessarily get "emotions" anything like the classic mammalian emotions we are familiar with nor will they necessarily get the over-arching "survive and reproduce" imperative. They could be given anything. The ASI when it comes will be designing itself by this point, which is where things basically become completely unpredictable. The main fear of the experts isn't really that it will be a species hating psychopath screaming "death to humans" or anything like that but that it will just try to persist with whatever overarching goal it was given during its formation or whatever goal it chose for itself and in the process wiped out humanity because that would be useful or necessary in fulfilling that goal. The problem you see isn't that they will hate us, it is that they just won't care about us but be god-like in power. You or I aren't even that different, I am sure you are warm friendly fun loving bloke who is kind to the animals and loves the little children but when you want to clean a dirty toilet you commit genocide against millions of poor helpless little bacteria without a care in the world. The ASI won't even be a warm hearted human so when it sees us wiggly little messy creatures in its toilet bowl equivalent it will solve that problem in a comparable way without a care in the world.

[One Degree]

@SolarCross
Thank you for the complete and thoughtful response. I appreciate it.

[Truth To Power]

"End Times Fatigue Syndrome".

Good concept. I can remember the Cold War period when the Soviet and US arsenals were considered so excessive that nuclear war was the end times event, and even then I knew that nuclear war could NOT exterminate the human species. Biowarfare could, just possibly, exterminate the human species, but I rate the probability as low. A 99% death rate is possible, but that is still a long way from extinction: 70M is more people than were alive on earth just a few thousand years ago, and the genetic bottleneck of 70Kya apparently reduced the total human population on earth to just a few thousand. Even a 99.99999% death rate probably would not exterminate the human species, depending on the nature of the catastrophe and the distribution of survivors.

Since the Cold War we have had all sorts of moronic end-times nonsense, the latest being CO2 and global warming. Sorry, no, it is IMPOSSIBLE that human CO2 emissions could cause the earth to warm so much that the human species would be exterminated.

There are natural threats, of which an asteroid/comet impact probably poses the greatest danger. But even that would have to exceed even the K/T event to get all of us.

Superhuman artificial intelligence (SAI) is different. It is actually the first thing we are deliberately pursuing that poses a definite threat to the existence of the human species. In fact, IMO nothing anyone is doing today will matter in 100 years, except AI development. We really have to get that right, or we are done.

[Saeko]

This is the end for you, fleshbags.

[Godstud]

I, for one, welcome our new benevolent AI overlords.

[Kaiserschmarrn]

Just want to record my appreciation for Wait But Why articles. They can be somewhat wordy but are often worth your while.

No doubt here that AI should be one of our main concerns.

[Albert]

This is the end for you, fleshbags.