Re: poly: Dumbing down AIs (was: Modeling Economic Singularitie

From: Nick Bostrom <>
Date: Sun Apr 26 1998 - 17:42:28 PDT

Robin Hanson wrote:

> I don't see an important difference between killing and preventing from
> living.

I think I tend to agree with that in principle, though it's a very
controversial question in ethics. In practise, of course, there is a
huge difference between killing someone and preventing someone from
being born. For example, killing someone sets a precedent that might
induce fear among living people.

>And you did say what would be in their place: conscious AIs.
> I'm saying I think it is unethical to replace conscious AIs with
> unconsious ones, just for the purpose of avoiding having to give them
> "rights".

By replacing some conscious AI:s with unconscious ones, and having
these save us lots of money by doing dirty or boring work for us, we
get more resources that we can spend building an exquisitely conscious
AI:s that can live interesting lives and enjoy themselves. Remember
that this discussion started when I asked whether it would be
technically feasible for an entrepreneur to get an unconscious AI to
do a task that a conscious AI could do, thereby saving salaries etc.

At this point you might object: If there is a certain job that can be
done equally cheaply by either a mindless robot or a conscious AI,
then, other things equal, it would be better to have the job done by
the AI.

-Well, that makes sense. But are other things equal? In the one case
we would have to give the worker rights, in the other we wouldn't. I
suppose that the overall utility will depend on what rights our
society requires us to give the AI. If we have a bad rights system,
say with nasty unions imposing high minimum wages etc., then the
total utility might be higher if we can circumvent the rights system
by building an unconscious robot.

> >... Of course, there might be good practical reasons for
> >limiting population growth, but in general I would say: the more, the
> >better (even if it would somewhat lower the average quality of life).
> Why not apply this logic to AIs?

I think we should.

Nick Bostrom
Department of Philosophy, Logic and Scientific Method
London School of Economics
Received on Sun Apr 26 23:48:20 1998

This archive was generated by hypermail 2.1.8 : Tue Mar 07 2006 - 14:45:30 PST