Re: poly: Dumbing down AIs (was: Modeling Economic Singularities

From: Nick Bostrom <>
Date: Fri Apr 24 1998 - 18:07:50 PDT

Robin Hanson wrote:

> Nick B. writes:
> >However, might it be possible to limit the AI in such a way that it
> >doesn have consciousness or count as a person (and hence doesn't have
> >rights) while retaining most of the benefits? If the AI were
> >domain-specific, and only did exactly what it was told, and did not
> >have any ability for self-reflection or long-term planning, would
> >that save us from having to give it person-status?
> It might save you under some rulesy rights ethics, but a more
> consequential ethics would call this a shame, perhaps even a crime.

Note that I was't proposing this, only asking about it's technical

> Imagine that the U.S. still had slavery, and instead of abolishing
> slavery, someone proposed that we genetically modify slave babies so
> that they met your criteria.

The relevant analogy seems rather to be if somebody proposed to
genetically modify zygotes, not babies. I don't think it is ethically
wrong to modify human zygotes in such a way that they grow up to be
brainless organ-banks.

> To me, this outcome is worse than slavery.
> The lives of slaves are not as fun as those of free folk, to be sure,
> but they are lives worth living to those slaves; they rarely commit
> suicide, for example. In contrast, the creatures you describe are just
> not there in some important sense. You've in effect killed the slaves,
> but kept their functioning bodies, and called this an improvement.

You haven't killed any slaves; you've prevented them from being born.
Whether this is an improvement depends on what you put in their

> To me it seems that if you switch from a world where a certain
> creature doesn't exist, to a world where that creature exists and finds
> their life worth living, that switch seems an improvment.

I agree. In fact, I'm happy that there are so many people on our
planet for the simple reason that I think human lives tend to be
valuable. Of course, there might be good practical reasons for
limiting population growth, but in general I would say: the more, the
better (even if it would somewhat lower the average quality of life).

Nick Bostrom
Department of Philosophy, Logic and Scientific Method
London School of Economics
Received on Sat Apr 25 00:13:51 1998

This archive was generated by hypermail 2.1.8 : Tue Mar 07 2006 - 14:45:30 PST