This link has been bookmarked by 1 people . It was first bookmarked on 21 Nov 2009, by Alex Ko.
-
21 Nov 09
-
In this article, I want to focus on two specific cool technologies that frequently go hand-in-hand: Google's Map-reduce (e.g. Hadoop), and Cloud Computing (e.g. EC2). For whatever reasons possibly including those I've mentioned above, temptation is high among many engineers today to use these sledge hammers to crack nuts.
-
as it turns out, our problem is very different in scale from, say, computing ranks on every document on the web. In fact, our graph is small by comparison, and we can afford to do most of our calculations on this graph in memory and in real-time. We are often able to get away with efficiently processing, within each of many simultaneous threads, the edges of any particular node in sequence. To be sure, one can indeed Hadoop this process by clever definitions of functions that will do the kind of recursive aggregation for the quantities that interest us. But we want to see if it is necessary at all, and for the foreseeable future at least.
-
Map-reduce is an optimization technique that buys efficiency using a clever divide-and-conquer approach. But to know what to divide, and how, and whether at all, one needs to first measure, estimate and calculate using tried and trusted techniques. Most of the time, a bigger home is not the right answer to clutter in a small one. An engineer who cannot write efficient code on a single computer most likely won't be able to write efficient code in the cloud. Only, it will be much harder to tell. And so in summary, let me leave you with these parting thoughts - One must first learn to walk properly on the ground before trying to fly in the clouds.
-
Would you like to comment?
Join Diigo for a free account, or sign in if you are already a member.