All Annotations of [Preview]

saved by10 people, first byJeff Schilling on 2006-03-02, last byChristopher Watson on 2008-07-09

• Ray saying
  that the rate of improvement of technology was going to accelerate and that
  we were going to become robots or fuse with robots

• Why the future doesn't need us.
  

  
  

  Our most powerful 21st-century technologies - robotics, genetic engineering, and nanotech - are threatening to make humans an endangered species.

  
  

  By Bill Joy

• Why the future doesn't need us

• Why the future doesn't need us.
  

  
  

  Our most powerful 21st-century technologies - robotics, genetic engineering, and nanotech - are threatening to make humans an endangered species.

• In
  >
  The Agony and the Ecstasy,
  >
  Irving Stone's biographical novel of
  >
  
  Michelangelo,
  >
  
  Stone described vividly how Michelangelo released the statues from the
  >
  
  stone, "breaking the marble spell," carving from the images in his mind.
  >
  4
  >
  
  In my most ecstatic moments,
  >
  
  the software in the computer emerged in the same way. Once I had imagined
  >
  
  it in my mind I felt that it was already there in the machine, waiting
  >
  
  to be released. Staying up all night seemed a small price to pay to free
  >
  
  it - to give the ideas concrete form.
  >
• I was taken aback, especially given Ray's proven ability to
  imagine and create the future. I already knew that new technologies like
  genetic engineering and nanotechnology were giving us the power to remake
  the world, but a realistic and imminent scenario for intelligent robots
  surprised me.
  • on 2007-06-28 Mwesch
    Robots are the future personified. Smart Dust and HyperLocal technologies will make the idea of robots seem ridiculous to our grandchildren.

• 
  

  
  I recently had the good fortune to meet the distinguished author and scholar
  >
  
  Jacques Attali, whose book
  >
  Lignes d'horizons
  >
  (
  >
  Millennium,
  >
  in the
  >
  
  English
  >
  
  translation) helped inspire the Java and Jini approach to the coming age
  >
  
  of pervasive computing, as previously described in this magazine. In his
  >
  
  new book
  >
  Fraternités,
  >
  Attali describes how our dreams of utopia
  >
  
  have changed over time:
  >

  
  

  
  "At the dawn of societies, men saw their passage on Earth as nothing more
  >
  
  than a labyrinth of pain, at the end of which stood a door leading, via
  >
  
  their death, to the company of gods and to
  >
  Eternity.
  >
  With the Hebrews and
  >
  
  then the Greeks, some men dared free themselves from theological demands
  >
  
  and dream of an ideal City where
  >
  Liberty
  >
  would flourish. Others, noting
  >
  
  the evolution of the market society, understood that the liberty of some
  >
  
  would entail the alienation of others, and they sought
  >
  Equality
  >
  ."
  >

  
  

  
  Jacques helped me understand how these three
  >
  
  different utopian goals exist in tension in our society today. He goes
  >
  
  on to describe a fourth utopia,
  >
  Fraternity,
  >
  whose foundation is altruism.
  >
  
  Fraternity alone associates individual happiness with the happiness of
  >
  
  others, affording the promise of self-sustainment.
  >

  
  

  
  This crystallized for me my problem with Kurzweil's dream. A technological
  >
  
  approach to Eternity - near immortality through robotics - may not be the
  >
  
  most desirable utopia, and its pursuit brings clear dangers. Maybe we should
  >
  
  rethink our utopian choices.
  >

• 
  

  
  I realize now that she had an awareness of the nature of the order of life,
  >
  
  and of the necessity of living with and respecting that order. With this
  >
  
  respect comes a necessary humility that we, with our early-21st-century
  >
  
  chutzpah, lack at our peril. The commonsense view, grounded in this respect,
  >
  
  is often right, in advance of the scientific evidence. The clear fragility
  >
  
  and inefficiencies of the human-made systems we have built should give
  >
  
  us all pause; the fragility of the systems I have worked on certainly humbles
  >
  
  me.
  >

  
  

  
  We should have learned a lesson from the making of the first atomic bomb
  >
  
  and the resulting arms race. We didn't do well then, and the parallels
  >
  
  to our current situation are troubling.
  >

• The cause of many such surprises seems clear: The systems involved are
  >
  
  complex, involving interaction among and feedback between many parts. Any
  >
  
  changes to such a system will cascade in ways that are difficult to predict;
  >
  
  this is especially true when human actions are involved.
  >
• At around
  >
  
  the same time, I found Hans Moravec's book
  >
  Robot: Mere Machine to
  >
  
  Transcendent Mind.
  >
• He goes on
  to discuss how our main job in the 21st century will be "ensuring continued
  cooperation from the robot industries" by passing laws decreeing that they
  be "nice,"3 and to describe how seriously dangerous a
  human can be "once transformed into
  an unbounded superintelligent robot." Moravec's view is that the robots
  will eventually succeed us - that humans clearly face extinction.

• 
  Danny's answer -
  >
  
  directed specifically at Kurzweil's scenario of humans merging with robots
  >
  
  - came swiftly, and quite surprised me. He said, simply, that the changes
  >
  
  would come gradually, and that we would get used to them.
  >

  
  

  
  But I guess I wasn't totally surprised. I had seen a quote from Danny in
  >
  
  Kurzweil's book in which he said, "I'm as fond of my body as anyone, but
  >
  
  if I can be 200 with a body of silicon, I'll take it." It seemed that he
  >
  
  was at peace with this process and its attendant risks, while I was not.
  >

• we have yet to
  come to terms with the fact that the most compelling 21st-century technologies
  - robotics, genetic engineering, and nanotechnology - pose a different
  threat than the technologies that have come before. Specifically, robots,
  engineered organisms, and nanobots share a dangerous amplifying factor:
  They can self-replicate.
• Specifically, robots,
  >
  
  engineered organisms, and nanobots share a dangerous amplifying factor:
  >
  
  They can self-replicate. A bomb is blown up only once - but one bot can
  >
  
  become many, and quickly get out of control.
  >
• But while replication in a computer or a computer
  >
  
  network can be a nuisance, at worst it disables a machine or takes down
  >
  
  a network or network service. Uncontrolled self-replication in these newer
  >
  
  technologies runs a much greater risk: a risk of substantial damage in
  >
  
  the physical world.
  >

• 
  The 21st-century technologies - genetics, nanotechnology, and robotics
  >
  
  (GNR) - are so powerful that they can spawn whole new classes of accidents
  >
  
  and abuses. Most dangerously, for the first time, these accidents and abuses
  >
  
  are widely within the reach of individuals or small groups. They will not
  >
  
  require large facilities or rare raw materials. Knowledge alone will enable
  >
  
  the use of them.
  >

  
  

  
  Thus we have the possibility not just of weapons of mass destruction but
  >
  
  of knowledge-enabled mass destruction (KMD), this destructiveness hugely
  >
  
  amplified by the power of self-replication.
  >

• Thus we have the possibility not just of weapons of mass destruction but
  of knowledge-enabled mass destruction (KMD), this destructiveness hugely
  amplified by the power of self-replication.
• My life has been driven by a deep need to ask questions and find answers.
  >
  
  When I was 3, I was already reading, so my father took me to the elementary
  >
  
  school, where I sat on the principal's lap and read him a
  >
  
  story. I started school early, later skipped a grade, and escaped into
  >
  
  books - I was incredibly motivated to learn. I asked lots of questions,
  >
  
  often driving adults to distraction.
  >
• at UC Berkeley in the mid-1970s, I started
  staying up late, often all night, inventing new worlds inside the machines.
  Solving problems. Writing the code that argued so strongly to be written.
• As this enormous computing power is combined with the manipulative advances
  of the physical sciences
  and the new, deep understandings in genetics, enormous transformative power
  is being unleashed. These combinations open up the opportunity to completely
  redesign the world, for better or worse:
• Given the incredible power of these new technologies, shouldn't we be asking
  how we can best coexist with them? And if our own extinction is a likely,
  or even possible, outcome of our technological development, shouldn't we
  proceed with great caution?
• But genetic engineering technology is already very far along. As the Lovins
  >
  
  note, the USDA has already approved about 50 genetically engineered crops
  >
  
  for unlimited release; more than half of the world's soybeans and a third
  >
  
  of its corn now contain genes spliced in from other forms of life
  >.
• The many wonders of nanotechnology were first imagined by the Nobel-laureate
  physicist Richard Feynman in a speech he gave in 1959, subsequently published
  under the title "There's Plenty of Room at the Bottom." The book that made
  a big impression on me, in the mid-'80s, was Eric Drexler'sEngines of
  Creation, in which he described beautifully how manipulation of matter
  at the atomic level could create a utopian future of abundance, where just
  about everything could be made cheaply, and almost any imaginable disease
  or physical problem could be solved using nanotechnology and artificial
  intelligences
• Among the cognoscenti of nanotechnology, this threat has become known as
  the "gray goo problem." Though masses of uncontrolled replicators need
  not be gray or gooey, the term "gray goo" emphasizes that replicators able
  to obliterate life might be less inspiring than a single species of crabgrass.
  They might be superior in an evolutionary sense, but this need not make
  them valuable.
• It is most of all the power of destructive self-replication in genetics,
  >
  
  nanotechnology, and robotics (GNR) that should give us pause. Self-replication
  >
  
  is the modus operandi of genetic engineering, which uses the machinery
  >
  
  of the cell to replicate its designs, and the prime danger underlying gray
  >
  
  goo in nanotechnology.
  >

• 
  

  
  The nuclear, biological, and chemical (NBC) technologies used in 20th-century
  >
  
  weapons of mass destruction were and are largely military, developed in
  >
  
  government laboratories. In sharp contrast, the 21st-century GNR technologies
  >
  
  have clear commercial uses and are being developed almost exclusively by
  >
  
  corporate enterprises. In this age of triumphant commercialism, technology
  >
  
  - with science as its handmaiden - is delivering a series of almost magical
  >
  
  inventions that are the most phenomenally lucrative ever seen. We are
  >
  
  aggressively pursuing the promises of these new technologies within the
  >
  
  now-unchallenged system of global capitalism and its manifold financial
  >
  
  incentives and competitive pressures.
  >

  
  

  
  This is the first moment in the history of our planet when any species,
  >
  
  by its own voluntary actions, has become a danger to itself - as well as
  >
  
  to vast numbers of others.
  >

• Now, as then, we are creators of new technologies and stars of the imagined
  >
  
  future, driven - this time by great financial rewards and global competition
  >
  
  - despite the clear dangers, hardly evaluating what it may be like to try
  >
  
  to live in a world that is the realistic outcome of what we are creating
  >
  
  and imagining.
  >

• 
  
  
  >
  >

  
  

  
  Where can we look for a new ethical basis to set
  >
  
  our course? I have found the ideas in the book
  >
  
  Ethics for the New Millennium,
  >
  
  by the Dalai Lama, to be very helpful. As is perhaps well known but little
  >
  
  heeded, the Dalai Lama argues that the most important thing
  >
  
  is for us to conduct our lives with love and compassion for others, and
  >
  
  that our societies need to develop a stronger notion of universal
  >
  
  responsibility
  >
  
  and of our interdependency; he proposes a standard of positive ethical
  >
  
  conduct for individuals and societies that seems consonant with Attali's
  >
  
  Fraternity utopia.
  >

  
  

  
  The Dalai Lama further argues that we must understand what it is that makes
  >
  
  people happy, and acknowledge the strong evidence that neither material
  >
  
  progress nor the pursuit of the power of knowledge is the key - that there
  >
  
  are limits to what science and the scientific pursuit alone can do.
  >

• Similar difficulties apply to the construction of shields against robotics
  >
  
  and genetic engineering. These technologies are too powerful to be shielded
  >
  
  against in the time frame of interest; even if it were possible
  >
  
  to implement defensive shields, the side effects of their development would
  >
  
  be at least as dangerous as the technologies we are trying to protect against.
  >
• The new Pandora's boxes of genetics, nanotechnology, and robotics are almost
  open, yet we seem hardly to have noticed.
• Clearly, we need to find meaningful challenges
  >
  
  and sufficient scope in our lives if we are to be happy in whatever is
  >
  
  to come. But I believe we must find alternative outlets for our creative
  >
  
  forces, beyond the culture of perpetual economic growth; this growth has
  >
  
  largely been a blessing for several hundred years, but
  >
  
  it has not brought us unalloyed happiness, and we must now choose between
  >
  
  the pursuit of unrestricted and undirected growth through science and
  >
  
  technology and the clear accompanying dangers.
  >

• 
  My continuing professional work is on improving
  >
  
  the reliability of software. Software is a tool, and as
  >
  
  a toolbuilder I must struggle with the uses to which
  >
  
  the tools I make are put. I have always believed that making software more
  >
  
  reliable, given its many uses, will make the world a safer and better place;
  >
  
  if I were to come to believe the opposite, then I would be morally obligated
  >
  
  to stop this work. I can now imagine such a day may come.
  >

  
  

  
  This all leaves me not angry but at least a bit melancholic. Henceforth,
  >
  
  for me, progress will be somewhat bittersweet.
  >

• So I'm still searching; there are many more things to learn. Whether we
  are to succeed or fail, to survive or fall victim to these technologies,
  is not yet decided. I'm up late again - it's almost 6 am. I'm trying to
  imagine some better answers, to break the spell and free them from the
  stone.