Agree with several that it is a very impressive achievement.
It seems that at recursive levels – all understanding comes as the result of experience, so the technique is recursively applicable.
Doesn’t directly address many of the survival level problems that evolution has developed useful heuristics for. Like guarding against the many classes of the halting problem, and running multiple simultaneous explorations of many different classes of threat assessment and threat mitigation strategies while simultaneously searching the space of useful opportunities and exploring useful numbers of them. Maintaining a balance between those two classes of strategy is one of the more difficult problems evolution has seemed to negotiate successfully. And we still see many instances of failures in practice where individuals and groups are excessively precautionary or excessively proactionary – and I acknowledge a very wide spectrum of context sensitive utility between those extremes (shadows of that old Greek virtue of the mean showing again).
And in the wider context, there are many levels of threat present.
Can we come to agreements about how much solar output will be available to humanity? Can we reasonably ask for and expect 30%?
How about solar mass – can we agree to leave that alone?
Can we reserve effectively 99.99999% of earth mass and energy for organic life forms, and 99% of lunar mass/energy, and 50% of the mass of the outer planets?
How about 95% of the energy available in region described by the cone centred in the sun earth axis that is 4 LD (Lunar Distance) radius at earth?
Will AGI think such things reasonable?
Will it consider us sufficiently interesting, and safe, to keep around?
I hope so. I am reasonably confident that our particular forms of computation and engagement will have some interest. And that destroying us would pose a significant risk to AGI’s own survival, in an encounter with any sufficiently further advanced life form.
And we could do a lot to improve our chances by looking past the scarcity based market systems we now value so highly, and beyond them to abundance and cooperative based systems that empower individuals within a context of respect for individual life and individual liberty, and the diversity that must increase exponentially in such a system.
The mathematical and logical existence of such forms is now proven beyond any reasonable doubt (thank you Robert Axelrod and Elinor Ostrom).
It is a very complex suite of problems, and there are no definite and computable answers, just an infinite nest of unknowns with a large set of unknowables.
It seems that we can only work with probabilities and our best heuristics, and a willingness to keep questioning assumptions (at ever recursive levels).