While I align with the vast bulk of what you have written Eliezer, there are two significant exceptions.
The first relates to your example of the “be paid five dollars” above.
Why do you continue to promote a values system (dollars) based in scarcity?
Is not the fundamental disjunct between the use of a scarcity based value system (markets and their denominational value money) and our exponentially expanding capacity to deliver universal abundance clear to you?
Oxygen in the air is arguably the single most important thing to any human being, yet of no monetary value, due to universal abundance.
In a world where most things were in fact necessarily scarce, that wasn’t a particularly important observation.
In a world where automation and robotics is enabling the fully automated production (and thence universal abundance) of a large and exponentially expanding set of goods and services, it becomes a fundamentally limiting and damaging paradigm.
Sure economic systems are complex.
Sure the signals of price and profit carry information in an informal way (and many people make much money by interfering with those signals at many different levels).
And we are capable of generating much more information, with far greater reliability and with multiple independent methods of estimating reliability, using modern communication systems.
In a world where value is based upon scarcity, it makes sense to reduce oxygen to the point that people need to buy it (if someone could work out a reliable and cheap way of doing that).
The perversity of this single outcome should point clearly to all the problems that lie in our near future if we do not transcend the use of money as a measure of value, and deliver universal abundance of all necessities including reasonable levels of freedom.
And of course there is a test of reasonableness (which is highly dimensional).
At a more practical and mundane level, the needs of capital and the needs of ordinary people are now significantly at variance (with an exponentially widening gap).
The other problem is the many dimensions of the computational complexity of reality.
How does one establish priors, particularly when dealing with small samples in stochastic systems?
It isn’t even logically an option!
Our brains seem to involve a large number of heuristic hacks that have worked in general over evolutionary time (genetic and cultural), but do not necessarily prepare us for our exponentially changing present (in any of the dimensions: technical, social, intellectual, behavioural).
Identifying the contextual limits of those “hacks” and developing useful alternatives seems to be a large part of what you do so very well, but you don’t yet seem to be clearly signaling the limits imposed by complexity and recursion on the approaches you are currently championing.
Working in real time requires useful heuristics (at every level).
Certainly we need people to be testing the utility of heuristic sets in novel situations (recurse to infinity), and we all need to be clear that no one can operate real time from first principles, we all need our many levels of heuristics to deal with the complexity around us. For some people the idea of God performs such a useful heuristic role (though few of them would describe it in such a fashion), and you and I are not in that set of people.
And I am certain that neither of us do first principle calculations on everything that impinges on our lives (there are just too many of them). We need many levels of heuristics, and we need to be able to modify them when required. And making that judgment seems to be the great art of life.