knobas last's Profile

Comparison of data analysis packages: R, Matlab, SciPy, Excel, SAS, SPSS, Stata

Python “immature”: matplotlib, numpy, and scipy are all separate libraries that don’t always get along.  Why does matplotlib come with “pylab” which is supposed to be a unified namespace for everything?  Isn’t scipy supposed to do that?  Why is there duplication between numpy and scipy (e.g. numpy.linalg vs. scipy.linalg)?  And then there’s package compatibility version hell.  You can use SAGE or Enthought but neither is standard (yet).  In terms of functionality and approach, SciPy is closest to Matlab, but it feels much less mature.

If your dataset can’t fit on a single hard drive and you need a cluster, none of the above will work. There are a few multi-machine data processing frameworks that are somewhat standard (e.g. Hadoop, MPI) but It’s an open question what the standard distributed data analysis framework will be.  (Hive? Pig?  Or quite possibly something else.)

(This was an interesting point at the R meetup.  Porzak was talking about how going to MySQL gets around R’s in-memory limitations.  But Itamar Rosenn and Bo Cowgill (Facebook and Google respectively) were talking about multi-machine datasets that require cluster computation that R doesn’t come close to touching, at least right now.  It’s just a whole different ballgame with that large a dataset.)