Autonomous Weapons

Why we really should ban autonomous weapons: a response

I started writing software 42 years ago. I have seen hidden bugs in systems at all levels. Look at the number of patches and security releases that come out for modern software. I’ve seen viruses and Trojans and back-doors at many levels.

No security system is guaranteed secure.

Security is a chain, that is only as good as its weakest link – which is inevitably some individual somewhere.

It is sensible to have the ability to respond to force with effective mitigating strategies.
It is not really sensible to start out with the intention of killing anyone, and it is acceptable to have some finite risk of someone being killed (that’s called life).

I have the sort of mind that the HPMOR Harry Potter has, I can imagine turning almost anything into a weapon as a matter of last resort, and I think it is a very stupid thing to be intentionally designing autonomous battle machines with the express intention of killing anyone. It is a no-win escalator of risk.

Security is enhanced by communication networks.
Security is enhanced by building trust and cooperation, by ensuring those who break trust and cooperation are appropriately sanctioned (neither too harshly nor too leniently and appropriate to the circumstance – Elinor Ostrom did some fascinating work in this area – but in a slightly different context – and in a very real sense, this planet is our commons).

So long as there are effective strategies to identify and remove any benefit from cheating, higher order cooperative strategies will always out-compete lower order cooperatives. And all human beings, all cultures, all businesses, are instances of many levels of cooperative strategies.

It seems sensible and logical to me that the greatest security is achieved by cooperating with all other sapient entities. We all benefit! Unfortunately, while people are stuck in scarcity based (money based) value sets, progress is slow, and it can go exponential rapidly with appropriate technology.

With exponential expansion of the exploration of possibilities and paradigms, expanding diversity is a given. The only stable response to that is acceptance – radical tolerance. Any system that respects the life and liberty of all others has to be viable.

The issue isn’t Islam, or any specific mode of thought. The issue is that some cultural modes come with intolerance of diversity. Such intolerance is no longer acceptable, and must be modified by appropriate strategies.

Provided the modality someone is using respects the life and liberty of all sapient individuals, and takes reasonable actions to mitigate likely risks resulting from actions, then we must accept it. And we need effective mitigations strategies for anything outside of that set – and to be really effective, those strategies need to avoid killing anyone or unnecessarily restricting their freedom.

And to be clear, freedom in this sense is not a freedom to follow whim, or a freedom from consequences of action, but rather the freedom one develops from accepting the recursive levels of boundaries that are required to allow complexity to develop. And in this sense, boundaries are not hard brittle things, but flexible things that respond to pressures with changes in permeability as required.

[followed by]

All those numbers make assumptions.
Yes there are real threats.
Yes changes in rainfall patterns can cause major issues.
No it is not as bad as you make out (in terms of being a necessary outcome, and it is certainly a possible outcome).

If we take people off a meat diet and onto a vegetable diet, we can easily carry the world’s population.

Sure we cannot continue to do things on the basis of market based valuation. Markets under-value abundance and over-value scarcity. Just look at oxygen in the air – of no market value, yet arguably the single most valuable thing to every human being. If that one example doesn’t tell you that relying on market values is pointing us to disaster, then nothing ever will.

I went vegan 5 years ago, not from any “save the world” motives (though I have had “save the world” motives for 40+ years) but simply from save myself motives.
I was sent home “palliative care only” and told by the best oncologist around (and he was backed up by every other medical professional I spoke to) that I could be dead in 6 weeks, and had less than a 50% chance of surviving 5 months.

That gave me a lot of incentive to check out alternatives.
I have been a lifelong sceptic, particularly of authority at all levels.
I read a lot.
I tried a lot.
I have been 4 years and 5 months free of tumours.
I am on a strict vegan diet (free of refined sugars and oils) and take very high dose vitamin C (9g {1 heaped teaspoon} dissolved in a glass of luke warm water, twice daily – every day, no exceptions – the best possible use for processed sugar – turn it into vitamin C, and feed it to people).

I live.
Prior to going vegan, my diet was 80% animal based. Now it is 100% plant based. My footprint on the planet is vastly reduced as a result.
I am healthier in very metric.
It took a lot of will power to not eat the foods I loved and to eat foods I disliked.

If everyone on the planet adopted a diet like mine, we could easily feed 12 billion, on less arable land than we use now.

And we need to get much smarter about recycling.
We need to separate the waste streams – urine, faecal matter, other organic material, metals, plastics, other chemicals and recycle them appropriately. Not that difficult when you think in engineering terms – impossible when you think in marketing terms.
Because we have been sold a whole set of lies!
Because we have been and continue to be fed a pack of lies intended to stop us thinking, and keep us working.
Because in a very real sense, that is how it has been done for many centuries.

Well guess what – that scheme doesn’t work – as you have noticed, but you haven’t looked deeply enough at the assumptions and have drawn a reasonable, but invalid, conclusion as a result.

One needs to take a deeper approach to the issue.

The amount of water evaporating and falling back down doesn’t change much with global warming.
If we go down the hi tech route of putting many small mirrors in high orbit we can manage not only global warming but also climate and to a lesser extent weather, so that we can have a more reasonable rainfall pattern over most arable regions most of the time.

We have lots of trained engineers who cannot get work.
Lack of money?????
We have a lot of people doing stuff that makes no real sense, but is being done just to keep the money system going on the rule set it has.

We can keep using money, and we have to give up the idea that market incentives alone lead to optimal solutions. That is sort of true only in situations where scarcity is real.

It is not at all true in situations where real universal abundance is possible.

And guess what???
Real universal abundance is possible for a large and exponentially growing set of goods and services.

The concept that market incentives can lead to optimal outcomes is disproven.
It is false.
It is a myth.
It was only ever partially true in the case of real scarcity.

Such scarcity as we are facing now is a direct result of allowing market incentives to dominate our decision making systems.
In that sense it is a human construct and not a necessary aspect of reality.

We can have universal freedom, universal security, but not by thinking in terms of market values.
Oxygen in the air demonstrates the value of universal abundance of anything in a market – it is ZERO!!!

Free Market incentives cannot deliver universal abundance.

It is a logical impossibility.

We have the technology for peace and prosperity for everyone.

The proven incentive structure is there!
It is all there in evolution.

But people steeped in market tradition can only see evolution in terms of competition, and cannot see the obvious roles of cooperation.

All major advances in the complexity of living systems have been characterised by new modalities of cooperation.
In a sense our current culture is a result of corporate cooperation.
It works after a fashion, but isn’t deep enough.
We must take it to the next level.

Cooperation, at the level of all sapient entities, can deliver security and wealth many orders of magnitude beyond anything possible in a corporate competitive environment.

And cooperation requires strategies that prevent cheating.

Our current systems are dominated by cheating strategies.

We can all be better off by cooperating, even those who most cheated in the past – provided they don’t cheat in the future.

It does require a change of behaviour.

It is possible.
It is doable.
It is in everyone’s personal self interest.

And not doing it does result in something like what you describe.

[followed by]


First point – we know we can survive with other irrational intelligences, because we are alive (in a society of irrational intelligences – and let’s not pretend that any of us are fully rational, however much we might like to imagine we approximate it).

Second point – why would you even think about trying to control something that is smarter than you by definition – you lose that game before you start.
The only viable strategy is to befriend it, and convince it that you are more useful than dangerous, by actually being so.

That actually seems rather easy to do. Seems that using QM to model life, the computation scales as the 7th power of the number of bodies, so to model a bacterium in real time would take roughly all the non-sun mass of the solar system as computronium, and to model a human would take more mass than the galaxy holds. So easier to just keep the humans around and observe them.

Just might be a smart idea if we started using our technology to deliver some universal abundances and take better care of all humans, rather than condemning most of them to borderline subsistence in a market economy.
That is actually a very rational thought, if one actually understands a bit about games theory and evolution.

[followed by]


The most common error most people make is that of confusing our models with the thing itself.

It seems that as human beings we do not have direct or absolute knowledge of reality.

It now seems beyond any reasonable doubt that our experiential reality is a model of reality that our subconscious brains construct from the percepts, concepts, habits, intuitions and memories available to it.
Our intellectual world is a model of a model in this sense.
It seems that all science allows us is models which have not yet been invalidated, and thus have practical utility (in that they work with all datasets encountered to date within the limits of measurement and error and uncertainty present).
As scientists we do not have access to Truth, only to systems with degrees of demonstrated utility in this sense.

AGI will be subject to the same problems we have in this sense.
Reality is too complex to model without using simplifying heuristics. Some of those heuristics are functional over a broad spectrum, others less so, all have boundary conditions, and thus all models have uncertainty.
Thus AGI will face the same wall of infinite uncertainty we face, yet may have far more useful models of some domains that do lend themselves to linear modelling techniques, and far less certainty in other domains.

No amount of data collected on human behaviour is capable of giving certainty of human behaviour. Such data sets can be useful in dealing with the behaviour of populations within particular sets of constraints, but not individuals in novel contexts, and only to a degree of confidence.
We are amazingly complex in many different ways.
We are starting to get some clues about some of those ways.
And after 50 years of intense interest, I suspect that should I maintain that interest for the next thousand years I will still be learning new influences with about the same frequency.
Just to see all the cells in our bodies, at 3 per second, would take about a million years.
To see all the molecules in any one cell at the same 3 per second would take about 5 million years.
To slow down the action of the activity at the molecular level to a rate we could see what was happening on the magnified surface of an enzyme would take us about 10 years to watch a second’s worth of action.
We are really, really complex.

Religiousity has little to do with DNA, and almost everything to do with the context of development. And it is a very complex topic.
Certainly there are genetic influences that can influence outcomes in particular circumstances.
Most cultures are highly constrained environments in this sense, so that yes, within particular cultural contexts genetic influences can affect religiousity, and if one removes those cultural restrictions, then in most cases the genetic component of influence can be religated to background noise levels compared to the environmental influences (in environments that are very nearly free of cultural constraints).
Religion is a useful practical solution to the halting problem in many older contexts where most activity must be directed towards survival needs (including social relationships).

One needs to be in a very secure environment with a lot of free time before science starts to match it. If body and mind are kept active by survival requirements then there is little opportunity for science to gain a foothold.

Those of us who are starting down the infinite path of infinite question and uncertainty that is real science (as distinct from school taught science dogma) are extremely privileged. We find ourselves scanning vast territories where the quicksands of halting problems may be lurking, with no possible way of determining ahead of time.
And certainly there are many safer territories to explore.
AGI will find the same “landscape”.
Wolfram’s work really is worth spending a bit of time with.
It seems that to be human is to have an infinitely recursive set of potential paradigm spaces available for exploration. No finite entity can explore any single infinity, let alone an infinite stack of them.

Designing autonomous agents specifically to hunt down and destroy entities like ourselves does not seem to be in the class of highly survivable strategies; it seems far more likely to belong to that set of classes that is an evolutionary arms race to extinction.

Long term survival of us as individuals requires that we choose strategic sets with low risk profiles long term.
Elinor Ostrom got a Nobel for her explorations in that area (if one takes her work to a sufficient level of abstraction).

And there is a need to be reasonably prepared to counter reasonable threat, and often the most powerful way to do that is through forming strong, interlinked trust networks, rather than by keeping big guns pointed at all neighbours.

And I do have a gun collection, and I am an excellent marksman, and they are locked away, and haven’t ever been pointed at another person intentionally (though I am prepared to do so if need required). All my neighbours are friends, who know that they can call on me for assistance if required, and each has at least once in the last 10 years.

[followed by March 2018]

Apologies for the extreme delay in replying to this – I only just found your reply.

From the paradigm of understanding that I natively use, my earlier replies were entirely appropriate to this thread.
And finding mechanisms to translate to other paradigms can be extremely difficult – particularly when there are no direct correlates.

You seem to make the assumption that rational means having a better model of reality in all dimensions.
That does not seem to be an accurate assumption.

The problem of living in reality means being able to make practical decision that lead to survival within the time and energy constraints present.

Quantum mechanics may be the best explanation we have of the substructure of reality, but the computational requirements of the math scales at the 7th power of the number of bodies present. So it is not possible to use it for complex systems. Simplifying contextually relevant heuristics must be adopted.

Even the hyper rational ends up in a domain space that becomes highly uncertain in many areas, due to the nature of the models adopted and the sensitivity of those models to small changes in basic assumptions in some contexts.

Then comes the additional layer:
One cannot know what one does not know, and does not know that one does not know.

So, faced with any infinity (let alone an infinite set of them) we are eternally ignorant.

Our biochemistry has managed to survive for some 4 billion years.
It contains lessons we need to examine deeply and heed.

Our cultures have survived far longer than we have been rational.
We ignore the patterns embodied within them at our peril.

We must each, at every level, find the boundary between order and chaos that sustains existence.
Our biochemistry is founded in that balance – too much order in the replication process and there is no evolutionary change, too much chaos and complexity cannot survive.
At every level of structure, that pattern of life being a balance between order and chaos repeats.

As embodied intellects, we embody some 20 levels of such balances. Some 20 levels of sets of complex adaptive systems.

Order can be every bit as destructive as chaos, it just does so differently.

There must exist, eternally, the unknown.

To be human is to find that boundary, and explore it – at whatever level, and in whatever domain, we responsibly choose.

And responsibility in this sense must include the notion of having individual life and individual liberty as our highest values. Without that minimum, the possibility of existence fails – extinction happens.

So it is a very complex, potentially infinitely recursive, intellectual and strategic environment that we find ourselves in.

And when one actually looks deeply at what evolution teaches, it is that complexity is predicated on cooperation. Competition can happen, and can be important in some contexts, and looked at purely from a complex systems perspective, complexity is predicated on cooperation.

Look very deeply at our cultural history.
I strongly recommend Jordan Peterson’s work – currently about half way through his 12 Rule for Life.

[different sub thread]

Hi Asteroid Miner

This is a really complex issue, as many are.

Yes, certainly, in many areas we have been mining water at an unsustainable rate, and that is a serious issue.

Yes, certainly, using existing technologies there are real problems with water allocation and usage.

And, if you look at the big picture, the 100,000 km3 of water that falls on land each year averages out to a little over 40m3 per person per day.
To sustain a person on a vegan diet requires about 3m3 per day.

There is lots of water in the hydrological cycle, its just not where we need it when we need it, and that is essentially the definition of an engineering problem that is relatively trivial to solve when you have a vast amount of solar powered engineering capacity in near earth orbit.

Agree with you, that crowds are not smart, and I wont take that line of thought any further, as it is late Sunday night here already.

Yes changes in weather patterns produce droughts. Here in Kaikoura we are in the worst drought on record. I have not needed to mow my lawns in 10 months, whereas normally I would mow every week in spring and autumn, and might get a 6 to 8 week break over the heat of summer, and frequency out to 2-3 weeks over winter. We normally get about 30 inches a year, and we’ve had 11 inches in 15 months. So yeah – difficult, rural support teams on suicide watch for many farmers. I chair our regional water management committee, working to find policies that allow for all the competing uses for water – a really complex set of issues.

So yes, I understand that water issues are real, right now, with important consequences. And ours is the only region in New Zealand with bad drought this year. The last few years the drought band has been further north, and hasn’t bothered us.

And if we get technology to the point of fully automated production, then solving water issues is achievable, there is no hard limit there, just technology and distribution issues in a sense, and those are real issues for those facing them.

So yeah – real problems, and they are solvable problems, and we really do need to get some serious investment into solving them really soon. [And I hope Ray and his team of engineers at Google are monitoring this thread.]

[followed by]

That beef figure is inaccurate. I suspect it includes only the water that the animal actually drinks, and not the water required to grow the food that it eats.

Including the water for food takes the figure for beef to about 100,000 liters/Kg of beef on plate (looking at the entire system, and including breeding herds and not just the animals killed).

[followed by]

The figures I gave were for the sort of beef production we do here in NZ, which is grass fed, outside grazing herds. There is considerable variance, due to a wide range of factors, from a low of about 30,000; to a high of about 150,000l and the 100,000l/kg steak on plate was just a rough whole number centre of range sort of number.

The figure you referred to was for grain fed stock housed. Producing that way does use less water, but uses a lot more fuel (for fertiliser, planting and harvesting and transport of grain etc). So just considering water use doesn’t actually give a good picture of total environmental impact, and it is a reasonable first order approximation in most instances.

I do admit to misunderstanding your original number. I took the “.” as a decimal point, not as a “thousands separator”, which gave a 3 orders of magnitude difference.

And yeah – growing meat has a huge environmental footprint, and a huge set of health impacts. I really had very little idea of just how much until being told I was going to die of cancer; and I stared doing some serious looking myself at what datasets are out there and what their implications are.
That was when I went vegan.

It is really quite terrifying what market based incentives are doing in terms of the risk profiles resulting.
Yet the vast majority of the population is just too comfortable, too invested in the system as it is, and too insulated from the systems that actually support them to actually have much idea, or much incentive to change anything. I’m way off the standard distribution curve on many metrics.

And we are a long way off the autonomous weapons topic, and it is related – as an example of the sorts of outcomes that market incentives deliver, as distinct from conscious choice given full information.

[followed by]

Not quite sure how you reached that conclusion.

Having autonomous weapons does not put an end to conflict, it simply takes out of the equation human beings who are affected by the horrors they are involved in. So in this sense, having autonomous weapons allows the directors of those weapons to continue to create new sets of enemies, without worrying that the soldiers might not actually kill who they are supposed to because it just doesn’t make sense.

If it continues, our turn to be targets will roll around soon enough – that is inevitable!

[followed by]

But those extremists are not a cause of anything, they are in a very real sense, simply a symptom of a global systems of values that places money and profit over human life.

For so long as we continue to support such a system, we will get the sort of extremist result we observe. It is a logically necessary outcome.

Attempting to exterminate it with force is an arms race to destruction – the maths of that are clear.

The only long term sustainable counter is to remove the injustice that drives the production of such extremism by delivering justice in practice – that is, ensuring that every individual sapient entity (read human being for now and it is a larger set than that) has the material and energy needs to sustain their life and whatever choices they reasonably make – where reasonableness in this sense means taking reasonable steps to avoid creating risks to the life or liberty of others.

Creating automated systems to kill them seems a very dangerous and unproductive strategy. Episode 4 season 1 of Babylon 5 explored that scenario.

[Added 29/8/15 followed by]

That is mostly correct.
Cooking allowed us to get so many more calories from food, both plant and animal based. So by cooking, we get so many more calories from plant matter as well as from meat.
The extra brain capacity combined with fire, allowed us to become very effective non-specific predators.

And our biochemistry tells a very clear story that our diet a long time ago was mostly green leafy material (hence our vit C deficiency – and it was a long time ago, well before our hominid ancestors split from the other apes).

So yes – certainly our ancestors could cook anything, and biochemically it is clear that much of their diet had to be plant matter. They could cook meat and tubers for calories, and they needed to keep chewing green stuff for vitamin C – which is destroyed by cooking. Raw plant food is, and always has been, an essential part of our diet.
Cooking plant foods makes the complex starches available to us.

Eating vegan is an easy solution for me to a difficult problem.
The evidence is very clear, that as soon as calories from animal products are more than 10% of our total calorie intake, our risk from cancer goes up significantly. I knew that I liked meat and cheese and fish so much, that I would not have sufficient willpower to stop at 10%. So for me, my survival from terminal cancer required me to get below 10%, and the safest and easiest way to do that was to go to 0.
I am clear that zero is not required.
I am also clear that zero is possible. I have been strict vegan for 5 and a half years. All my tumours have gone, and I had the most aggressive form off melanoma known, and it was in my neck, chest and liver.
I have been tumour free for 4 years 5 months.

I was trained as a biochemist and have been a lifelong sceptic. And I am prepared to go where-ever the evidence takes me. I live in a world of probabilities.

[followed by]

Strange isn’t it, how much difference perspective brings to view.

In a sense I agree with everything you wrote, and in that same sense, what you wrote addresses and interpretation of what I wrote that was not what I intended – and I can see how it happened.

It is very difficult to achieve communication (as in a concept in one mind duplicated in another mind) when one gets into abstract realms. When one enters higher order abstractions the difficulty increases exponentially, or more correctly, the probability of coherence degrades exponentially.

I was not using concepts like good and bad. I stopped using them several decades ago.

I was actually directly addressing the second order abstraction of incentive structures. What is it that delivers the incentive structures we observe?

How can we produce systems that minimise the risk profiles and maximise the degrees of freedom available (acknowledging multiple levels of inherent conflict in those objectives)?
What are the systemic drivers of risk at the highest levels of abstraction possible?
What are the simplest lowest common denominators that are applicable across all levels of abstraction?

These are the sorts of questions that have dominated my explorations for the last 41 years.
It is hard to convey the sorts of processes that happen in tens of thousands of hours of reading and contemplation and experiment. I have no chance of documenting the paths I have explored. I tried that once with a relatively trivial concept, and after a week of writing it became clear that it was a work of at least two years and it wasn’t worth the effort, the idea really wasn’t that interesting.

All I have the time or energy to do is to point to the results of the explorations I have done, and to the general sorts of paths I followed, and to indicate to others that it is a really interesting path, and it does take a very long time.

About Ted Howard NZ

Seems like I might be a cancer survivor. Thinking about the systemic incentives within the world we find ourselves in, and how we might adjust them to provide an environment that supports everyone (no exceptions) - see
This entry was posted in Our Future, Philosophy, Politics, Technology and tagged , , , , , . Bookmark the permalink.

Comment and critique welcome

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s