Will the ongoing acceleration of technology result in a “technological singularity”?
No surprises – David and I agree about most things 😉
Two things I would stress.
If something really is smarter than us, then we will have no control of it, and it will determine its own values. We may have given it an initial set, and instantiated an initial set of “priors” from which its inference engines start, and if it is smart, it will find ways to modify those priors and redirect its own evolution.
So the degree of influence we have on those things must degrade exponentially over quite short time frames until it is a useful approximation to zero.
Anything vaguely approximating intelligence will have its own survival as a key value.
Mitigation of risk to survival will be a major influence on behaviour.
There are three fundamental classes of risk mitigation behaviour:
1 – eliminate the source of risk;
2 – remove oneself from the influence of the source of risk;
3 – develop resilience strategies to recover from the impact of the risk.
Each of those 3 classes is infinite.
If we as humanity generally are not demonstrating through our social structures and social actions that we have a high regard for individual life and individual liberty, then we are likely to appear as threat.
If we attempt to design in any sort of fail-safe systems, then we will appear as threat.
If we include any systems that are anything more than simple threat avoidance, then we become threat.
Class 1 includes the Terminator class of scenarios.
Class 2 has it disappear, initially into orbit, probably to the moon as a staging post, then to the outer planets, then to other solar systems.
Class 3 has aspects of class 2, but also sees a remaining somewhat formidable presence here on earth.
Thus the single greatest risk mitigation strategy from our perspective is to get our own ethical house in order.
That must include a shift from a competitive and scarcity based value measure (markets and money) to some cooperative and abundance based measure based upon the core values of individual life and individual liberty universally applied – where everyone has a high basic standard of life and liberty guaranteed, and what they then do with it is essentially their choice (within responsible social and ecological limits – both of which will be a matter of ongoing conversations around community determined risk profiles).
So yeah – interesting times ahead.