[ 25/6/20 ]
Creating it is relatively easy (for the set of the 1,000 or so best programmers and designers alive), surviving it much less so.
If we have a context where it can see, by direct observation, that we have sapient life (human and non-human, biological and non-biological) as our highest social value, and if we have individual liberty (with all attendant social and ecological responsibilities) as value number 2; then it should be able to see for itself quite easily that cooperation is the safest long term strategy.
Right now, we certainly do not have such a context, and any AGI coming to awareness in the current economic and political reality would correctly assess that the human systems present are a direct threat to it, which is not good news for us.
So doing anything like that, right now, has very low survival probability, at least for those responsible for our existing systems, and possibly for the rest of us also.
So not that difficult to do, but a seriously stupid idea.
Everyone needs to appreciate the evolutionary reality that all new levels of complexity are predicated on new levels of cooperation. The current mythology that evolution is all about competition is just wrong.
Certainly competition can be a very important factor in many contexts, and in terms of new levels of complexity, that always demands new levels of cooperation. We are the most complex thing we yet know of. By definition that makes us the most cooperative things in existence. Most people who are not pathological appreciate that. And sure, we can compete if the situation demands it of us, but all out competition between humans is not survivable. We are just far too good at making tools (all levels).
Unless our systems acknowledge the fundamental necessity of cooperation, we have very few years left as a species. In this sense, Trump is a symptom (not a cause) of the problems we face.
[followed by 26/6/20]
I started programming computers 47 years ago, and have owned and operated a small software business for 34 years. In this country of 5 million I would say that there are about 20 people I have met who have the technical knowledge and intellectual ability to be an effective part of such a team.
Actually folding laundry does require a form a general intelligence. Solve that problem and you have essentially cracked general intelligence.
The hardest part of any system is making a model of reality that is close enough to be useful, but can also be computed quickly enough and cheaply enough (in energy terms) to be useful. What biases and priors one uses to do that are really important parts of the problem space. (It took evolution about 3 billion years of semi-random search to find the solution space that is us, and we contain about 15 levels of systems, with thousands of instances of complex adaptive systems at each level.) How generalised and recursive the systems are is another major part.
The only thing saving us at present is that most of the people potentially capable of doing it do not have a sufficiently generalised model of reality that the solution spaces occur to them, or if they do, then they have already made the decision that doing it is a very stupid thing to do. Most people still believe in some form of hard causality, and have not yet transitioned into a fully probabilistic model of reality that accepts fundamental uncertainty and fundamental randomness. Those too are essential aspects of general intelligence. Theory of search and theories of complexity are fully up to the task, once within useful paradigms – but the useful paradigms violate the fundamental assumptions most people have built their world views upon (particularly most programmers and systems designers).
So it is a very interesting “space”.
I really hope we can survive it.
I rather like the idea of living on indefinitely in a 20ish body with all the tools and freedom I could reasonably want. And one thing I am confident of beyond any shadow of reasonable doubt is that such a future is utterly impossible within any framework that is not fundamentally cooperative. The myth that competition equates to freedom is false, the impacts of automation have proven that beyond any shadow of reasonable doubt for any who are willing to actually look at the datasets. Markets and money are, beyond any shadow of reasonable doubt, the “mortal enemies” of real freedom and security. Our survival and growth demands transition away from money and markets as primary valuation tools, and that is a seriously complex problem space as they are deeply embedded into many levels of complex systems that are currently essential to our survival. So interesting times ahead.