Writing An Hadoop MapReduce Program In Python - Michael G. Noll - The Diigo Meta page

• 27 Sep 11
  

  Jochen Fromm

  how to write a simple MapReduce program for Hadoop in Python

  python hadoop mapreduce programming distributed parallel cluster

• 18 Nov 10
  

  David narayan

  hadoop python

• 24 Oct 10
  

  Sokratis Galiatsis

  hadoop delicious

• 22 Oct 10
  

  Sven Duzont
• 15 Oct 10
  

  michaelmurdock_

  for:michaelmurdock

• 23 Sep 10
  

  Alfred Reich

  Describes how to write a simple MapReduce program for Hadoop in the Python programming language.

  python hadoop mapreduce tutorial distributed

• 27 Aug 10
  

  hadoop largedatasets

• 15 Jul 10
  

  enarasim

  hadoop python wordcount example

  • The "trick" behind the following Python code is that we will use HadoopStreaming (see also the wiki entry) for helping us
  • print '%s\t%s' % (word, 1)
  • An important note about mapred.map.tasks: Hadoop does not honor mapred.map.tasks beyond considering it a hint. But it accepts the user specified mapred.reduce.tasks and doesn't manipulate that. You cannot force mapred.map.tasks but can specify mapred.reduce.tasks.
  • http://localhost:50030/

  2 more annotations...

• 30 Jun 10
  

  Manuel Rivas

  python

• 29 May 10
  

  Nick Hortovanyi

  python hadoop mapreduce programming distributed tutorial

• 26 May 10
  

  dan

  native (!) CPython support in hadoop

  mapreduce hadoop python cli for:strangefeatures

• 17 May 10
  

  Jay Luker

  python parallel hadoop mapreduce

• 24 Apr 10
  

  sabeall

  hadoop python mapreduce

• 16 Apr 10
  

  billy mcdermott

  programming hadoop python

• 22 Feb 10
  

  Harley Bussell

  python hadoop mapreduce distributed

• 14 Dec 09
  

  xarlie

  software linux opensource programming java

• 02 Dec 09
  

  Travis B Hartwell

  python hadoop mapreduce

• 30 Nov 09
  

  wei deng

  hadoop mapreduce python distributed

• 27 Nov 09
  

  res0nat0r

  cloud:tutorial

• 06 Oct 09
  

  Cloud S

  mapreduce hadoop python

• 02 Oct 09
  

  pinco pallo

  imported python development algorithm distributed database tutorial hadoop grid blog opensource data mapreduce scalability concurrency

• 25 Sep 09
  

  Doug Y'barbo

  Python Code ***** DistributedSystem DistributedProcessing

• 19 Sep 09
  

  Israel Ag Nouh Yattara

  linux python java tutorial code computing database distributed data google parallel map cluster performance scalability algorithm concurrency grid hadoop mapreduce clustering reduce scaling hadoopdb cloud-computing Marketing

• 20 Aug 09
  

  drjohnsonjr

  hadoop python

• 31 Jul 09
  

  alex madon

  hadoop python mapreduce

• 22 Jul 09
  

  pierre arlais

  distributed computing on large data sets

  mapreduce hadoop python distributed computing

• 08 Jul 09
  

  Nico Coetzee

  Even though the Hadoop framework is written in Java, programs for Hadoop need not to be coded in Java but can also be developed in other languages like Python or C++

  apache java hadoop hive python programming cluster distributed parallel tutorial

• 24 Jun 09
  

  feng bo

  hadoop

  • Prerequisites
• 30 May 09
  

  Julian R

  Hadoop

• 24 Apr 09
  

  Jan Aerts

  hadoop mapreduce tutorial from_delicious

• 17 Apr 09
  

  Stefan Kueppers

  programming python distributed tutorial howto parallel cluster mapreduce hadoop

• 03 Apr 09
  

  The home page of Michael G. Noll - working for a Safer Internet

• 

  Ivan Morales

  map reduce hadoop python cloud

• 31 Mar 09
  

  knobas last

  hadoop python

• 30 Jan 09
  

  Alexandre Mclean

  distributed mapreduce hadoop tutorial python

• 18 Jan 09
  

  Jarek Jarek

  devel python tutorial mapreduce hadoop

• 15 Jan 09
  

  nealrichter

  mapreduce hadoop nlp tutorial python

• 10 Dec 08
  

  Alex MIkhalev

  programming python tutorial scalability performance scaling

• 30 Nov 08
  

  Michael O'Brien

  hadoop python mapreduce programming

• 21 Nov 08
  

  pierreaubert

  hadoop python

• 07 Oct 08
  

  Andrew Perry

  The "trick" behind the following Python code is that we will use HadoopStreaming (see also the wiki entry) for helping us passing data between our Map and Reduce code via STDIN (standard input) and STDOUT (standard output). We will simply use Python's sys

  python code hadoop mapreduce tutorial howto parallel

• 28 Sep 08
  

  lucidus
• 23 Sep 08
  

  benpjohnson

  mapreduce python

• 04 Sep 08
  

  s_bergmann

  This tutorial describes how to write a simple MapReduce program for Hadoop in Python without using Jython.

  hadoop python

• 03 Sep 08
  

  Hamada Kazunori

  tutorial software scaling scalability reduce python programming performance parallel opensource

• 29 Aug 08
  

  John M

  hadoop java howto

• 11 Jul 08
  

  Noel O'Boyle

  python mapreduce

• 10 Jul 08
  

  Paulo Nuin

  python hadoop mapreduce cluster parallel

• 06 Jul 08
  

  Ricardo Vidal

  python hadoop mapreduce programming distributed parallel linux apps

• 05 Jul 08
  

  JingleMan Sweep

  apache clustering cluster algorithm python hadoop mapreduce programming distributed parallel tutorial map reduce delicious

• 06 Jun 08
  

  Scott Moody

  Writing An Hadoop MapReduce Program In Python
  In this tutorial, I will describe how to write a simple MapReduce program for Hadoop in the Python programming language.
  Contents [hide]
  1 Motivation
  2 What we want to do
  3 Prerequisites
  4 Python MapReduce Code
  4.1 Map: mapper.py
  4.2 Reduce: reducer.py
  4.3 Test your code (cat data | map | reduce)
  5 Running the Python Code on Hadoop
  5.1 Download example input data
  5.2 Copy local example data to HDFS
  5.3 Run the MapReduce job
  6 Improved Mapper and Reducer code: using Python iterators and generators
  6.1 mapper.py
  6.2 reducer.py
  7 Feedback
  8 Related Links
  Motivation
  
  Even though the Hadoop framework is written in Java, programs for Hadoop need not to be coded in Java but can also be developed in other languages like Python or C++ (the latter since version 0.14.1). However, the documentation and the most prominent Python example on the Hadoop home page could make you think that you must translate your Python code using Jython into a Java jar file. Obviously, this is not very convenient and can even be problematic if you depend on Python features not provided by Jython. Another issue of the Jython approach is the overhead of writing your Python program in such a way that it can interact with Hadoop - just have a look at the example in <HADOOP_INSTALL>/src/examples/python/WordCount.py and you see what I mean. I still recommend to have at least a look at the Jython approach and maybe even at the new C++ MapReduce API called Pipes, it's really interesting.
  Having that said, the ground is prepared for the purpose of this tutorial: writing a Hadoop MapReduce program in a more Pythonic way, i.e. in a way you should be familiar with.

  python hadoop howto tutorial configuration install

• 02 Jun 08
  

  Yong Zhang

  blog data database programming python

• 31 May 08
  

  Nitin Hayaran

  In this tutorial, I will describe how to write a simple MapReduce program for Hadoop in the Python programming language.

  hadoop python

• 30 May 08
  

  Jeremy James

  programming mapreduce hadoop python tutorial

• 

  mat tat

  python hadoop mapreduce

• 15 May 08
  

  Krishnan Subramanian

  python mapreduce hadoop

• 01 May 08
  

  Ben Godfrey

  A walkthrough of getting into Hadoop MapReduce with Python.

  programming hadoop python for:jorjun

• 17 Apr 08
  

  Charles Nadeau

  tutorial Python Hadoop

• 16 Apr 08
  

  ilya b

  Hadoop Python MapReduce programming distributed parallel

• 21 Mar 08
  

  Felix R.

  hadoop distributed python parallel code example programming mapreduce cluster del.icio.us

• 01 Mar 08
  

  Zarko Bizaca

  java c++ hadoop

• 19 Feb 08
  

  Mark Leyden

  python hadoop mapreduce

• 05 Feb 08
  

  giladb

  hadoop python mapreduce for:ninjamode

• 15 Dec 07
  

  voidfiles

  hadoop looks like a key in an infrastucture

  python hadoop mapreduce programming distributed cluster parallel

• 12 Dec 07
  

  Chris Douglas

  hadoop

• 17 Nov 07
  

  andrew baldwin

  hadoop mapreduce python

• 01 Nov 07
  

  hideaki-t

  python hadoop cluster distributed mapreduce

• 14 Oct 07
  

  cluster distributed hadoop mapreduce parallel python

• 13 Oct 07
  

  boxofbox

  tech_python tech_cluster_parallel_distributed

• 12 Oct 07
  

  mikeyp

  hadoop mapreduce python programming parallel

• 10 Oct 07
  

  kamyar

  python hadoop mapreduce distributed cluster parallel

• 09 Oct 07
  

  Tom Liu

  python hadoop mapreduce programming parallel for:ivyz for:zhangjiayin

• 

  bandris

  scaleablitity DIST

• 

  mallipeddi

  Great tutorial!

  hadoop mapreduce python tutorial delpress java

• 

  Fabien Schwob

  hadoop python distributed

• 25 Sep 07
  

  Carlos Santos

  cluster development hadoop java distributed mapreduce parallel programming python

• 24 Sep 07
  

  Miguel Angel Rasero

  hadoop python map

Would you like to comment?

Join Diigo for a free account, or sign in if you are already a member.