Hi

• In fact, its 32,000-node cluster is still the largest in the world. Now the Web giant is souping up its massive investment in Hadoop to give it a deep learning environment that’s the envy of the valley. With more than 600 petabytes of data spread across 40,000 Hadoop nodes, Yahoo is a big believer in all things Hadoop.

Today (April 2, 2012)

Listen in to JSConf 2012 (or follow the tweetstream) http://2012.jsconf.us/#/schedule where you'll get an update about Mojito and the Open Source plans. We put the code on http://github.com/yahoo/mojito

Read it, fork it, play with it, test it out, enhance it, suggest enhancements, add to it, make great stuff with it, and let's take Node.JS to the next level, as a community.

If you just want to write and run MapReduce jobs locally without a Hadoop cluster, you could consider the HackReduce starter kit. At one of the HackReduce events, I was able to use the configuration the organizers provided to quickly get started coding a MapReduce program in Java/Eclipse, testing locally and eventually deploying the program to run on the EC2 clusters that were provided.

Their source code is available on GitHub at https://github.com/hopper/HackReduce.

To run a job locally, you could follow one of the commands beginning with "java -classpath" in the middle of their README file. Unfortunately, I don't think there's a way to run your jobs on an actual Hadoop cluster as they don't seem to provide an AMI for their EC2 setup, and these clusters are probably only made available at HackReduce events.

If you want to explore setting up your own Hadoop cluster, their resources page (http://www.hackreduce.org/hadoop-and-mapreduce/) has several links to MapReduce and Hadoop tutorials which you might find useful.

I can see many reasons:

1. Simplicity - The simplicity makes it incredibly easy to understand.
2. Applicability - The paradigm is so simple that a very large number of already defined algorithms can be implemented using one or more Map-Reduce steps. This makes it very easy to implement large scale distributed systems for things that required a lot more information regarding distributed systems than before.
3. Scalability - The abstraction is implicitly scalable which takes makes it very useful for cases where the amount of data processing is huge.
4. Redundancy - Error correction can be implemented pretty easily using redundancy. Many other paradigms find it hard to implement fault tolerance without significant work, Map-Reduce makes it easier. Most Map-Reduce deployments are on Distributed Filesystems (HDFS/BigTable), making the data retrieval/deposit steps fault tolerant as well.
5. Infrastructure Requirements - Are not high. The underlying infrastructure is generic; a network of simple processing units with disk access.

Distributed sorting is a great benchmark for mapreduce implementations because it exercises all the parts of the framework well.

The most time consuming part probably is sending the data over the network from the mappers to the right reducers.

Over time a number of results on Distributed Sort benchmarks have been reported by Yahoo, Google and Microsoft:

• yahoo08 TeraByte Sort on Apache Hadoop http://sortbenchmark.org/YahooHadoop.pdf
• google08 Sorting 1PB with MapReduce http://googleblog.blogspot.com/2008/11/sorting-1pb-with-mapreduce.html
• yahoo09 Winning a 60 Second Dash with a Yellow Elephant http://sortbenchmark.org/Yahoo2009.pdf
• google11 Sorting Petabytes with MapReduce - The Next Episode http://googleresearch.blogspot.com/2011/09/sorting-petabytes-with-mapreduce-next.html
• microsoft12 Data in the Fast Lane http://research.microsoft.com/en-us/news/features/minutesort-052112.aspx

Each benchmark was sorting through keys of 100 bytes in length.

Here's is a summary of the results (hope I didn't mess this up :) ):

cores was only reported in the Yahoo results and it was 8 cores per node.

I computed throughput as the average of how much data is sorted by a CPU per second.

The MinuteSort result is interesting because it has the highest throughput per machine. The purpose of the benchmark is sort as much data as possible in 60 seconds, so I wonder if using just 250 machines is a limitation of their system.

As for hadoop vs google map reduce in 2011, there's no new benchmark from Yahoo.

It's impressive how Google MR has improved between 2008 and 2011. Can use double the number of nodes, sort through 10 times the data and have a 5 times higher node throughput.

The google11 article mentions:

We are excited by these results. While internal improvements to the MapReduce framework contributed significantly, a large part of the credit goes to numerous advances in Google's hardware, cluster management system, and storage stack.

The list of people who worked on and/or directly influenced Hadoop as part of their official duties at Yahoo! pre-2009 is around 40 or so people. A lot of them have spread out to other companies/projects. Many of them are no longer directly involved with Hadoop. The best way to find out if someone was actually, directly involved is to ask which org they were in. If it wasn't specifically Grid, then they were likely (but not 100% guaranteed) just a user.

(or if they remember how many bugs we hit when we upgraded Mithril to support users and permissions in HDFS. haha.)

BTW, Mike was never an employee of Yahoo!. Like him, there are a handful of people at that time that were working on Hadoop outside of Y!. They are rare, but they exist. ;)

At the most recent HBase hackathon, one idea we discussed doing in 2011 is something that would allow HBase to very closely approximate Haystack in a lot of ways.

The general idea is the same as what databases have been doing for a long time as external blob storage. When any row is written with a larger amount of data, we would append it to a "haystack" and then just write the metadata into the table itself. On retrieval, the metadata would be small enough to fit in cache and we would perform a single seek into the haystack to read the data.

The advantages here are:
- the large data wouldn't ever have to hit the HLog, and hence would increase throughput
- the large data wouldn't have to be compacted (or could be compacted much more infrequently) while the metadata could continue to be compacted in order to provide efficient lookup
- the haystack files would be immutable over a long period, allowing HDFS improvements like HDFS RAID to reduce storage requirements.

• Ceph
• QFS by Quantcast: http://quantcast.github.com/qfs/
• Cloud-Crowd by DocumentCloud: https://github.com/documentcloud/cloud-crowd
  
• HPCC by LexisNexis: http://hpccsystems.com/
• Condor: http://www.cs.wisc.edu/condor/ (application notes: www.cse.nd.edu/~ccl/research/pubs/pbragahe-thesis.pdf )
• Storm by Nathan Marz: https://github.com/nathanmarz/storm
• HaLoop: An offshoot of Hadoop to support iterative data processing: http://code.google.com/p/haloop/
• MapRejuice: the distributed client-side computing implementation of Map Reduce built on top of Node.js: https://github.com/ryanmcgrath/maprejuice
• GoCircuit: Paradigm for developing and sustaining Big Data apps

Subsequent to this question and two answers being provided, Yahoo! has indeed "spun off" a significant part of the Hadoop engineering team, forming the "Hortonworks" company http://www.hortonworks.com/ -- dedicating to delivering success with Hadoop. Yahoo! continues its significant investment in Hadoop engineering internally as this technology is very important to the company.

This is viewed as a "win-win" situation which should greatly benefit the Apache Hadoop community. It will help address the huge demand (and short supply) for expertise and services in this very sophisticated area of technology. And it provides a nice influx into the economics of Hadoop for Yahoo! and other ecosystem participants.

Hadoop consists of many subprojects: HDFS, MapReduce, Hive, Pig, HBase, and Avro. I believe this question refers to the MapReduce implementation, which can operate over a variety of storage systems. I agree that Hadoop could use decent competition; having used the software daily for many years, I disagree that it's becoming more complex with each release. As an example, see the heroic efforts of Chris Douglas on https://issues.apache.org/jira/browse/MAPREDUCE-64 to remove some of the tuning parameters for a MapReduce job. I suppose that's for another question though.

Both MapReduce implementations mentioned above (CouchDB and MongoDB) require that data live in the data stores specified and present far different semantics than those described in the Google paper.

Here are some MapReduce-ish implementations, all of which are either coupled to a single storage system, a single programming language, or implement only a small subset of the features of a mature MapReduce implementation:

• Disco: http://discoproject.org
• Misco: http://www.cs.ucr.edu/~jdou/misco/
• Phoenix: http://mapreduce.stanford.edu
• Cloud MapReduce: http://code.google.com/p/cloudmapreduce
• bashreduce: http://blog.last.fm/2009/04/06/mapreduce-bash-script
• Qizmt: http://code.google.com/p/qizmt
• HTTPMR: http://code.google.com/p/httpmr
• Galago's TupleFlow: http://www.galagosearch.org/guide.html
• Skynet: http://skynet.rubyforge.org
• Sphere: http://sector.sourceforge.net
• Riak: http://riak.basho.com/mapreduce.html
• Starfish: http://rufy.com/starfish/doc/
• Octopy: http://code.google.com/p/octopy/
• MPI-MR: http://www.sandia.gov/~sjplimp/mapreduce.html
• Filemap: http://mfisk.github.com/filemap/
• Plasma MapReduce: http://projects.camlcity.org/projects/plasma.html
• Mapredus: http://rubygems.org/gems/mapredus
• Mincemeat: http://remembersaurus.com/mincemeatpy/
• MapReduceTitan: http://www.kitware.com/InfovisWiki/index.php/MapReduce
• GPMR: http://www.idav.ucdavis.edu/research/projects/mgpu_mapreduce
• Elastic Phoenix: https://github.com/adamwg/elastic-phoenix
• Preregrine: http://peregrine_mapreduce.bitbucket.org/
• R3: http://heynemann.github.com/r3/

Also, Microsoft's DryadLINQ is available under an academic license (not quite open source) at http://research.microsoft.com/en-us/projects/Dryad.

Disclosure: I founded Cloudera, a company that provides commercial support for a distribution of Hadoop, among other things.

I believe the answer is "No". Contrary to some answers, the superiority of MPP systems for analyzing relational data has little to do with "ACID compliance"--indeed, many of these systems are used entirely offline, where consistency concerns are irrelevant. They are faster than Hadoop because they are optimized for relational operations, and because they don't make some of the dubious design decisions (such as materializing all intermediate results for fault tolerance) that appeared in the original MapReduce paper. Even later Hadoop iterations (e.g., Hive on Tez) don't perform as well on relational queries as purpose-built MPP systems like Impala or Presto.

Shameless plug: the University of Washington database group is working on such an MPP system that is compatible with HDFS: http://myria.cs.washington.edu

See, databases and mapreduce are entirely two different things. Mapreduce systems are intended for offline processing where as the databases/parallel databases are meant for online transaction processing. If we go with the published results mapreduce obviously beats the other warehouse analytics at large scale computation ("big data" they call it now). However if you worry about the scaling online transaction processing systems like databases and parallel databases, they pose an entirely new set of problems related to consistency and stuff (you can look into CAP theorem). Though I agree with you that there has been no popular parallel database till date, most of it is because of the overhead they incur because they stick to ACID properties. But recent research and use-cases have proved that one need not stick to all of the ACID properties and some of them might be relaxed. This resulted in the new key-value stores where they address issues by trading some ACID properties (for eg: relaxing consistency rules) for performance and this has shown some improvement. You can look into systems like HBase, Cassandra, Google F1 database and read some literature on CAP theorem, eventual consistency etc. Overall I feel that there has been no big parallel database because they tried to stick to ACID properties thus giving a large overhead. However modern systems are trying to bridge that gap by relaxing something or the other that they feel is unnecessary. This is my opinion and comments are welcome.

If your main criteria is simplicity, I think that the two main alternatives are Disco (http://discoproject.org/) and Filemap (http://mfisk.github.com/filemap), both mentioned in a previous answer. I particularly like Filemap because you can't get mapreduce like processing simpler that this.

One interesting alternative is cloud mapreduce (http://code.google.com/p/cloudmapreduce/) which is implemented on top of Amazon Web Services and follows a completely different architecture without a job tracker, on which individual nodes self-schedulle the task of a job.

I don't agree with some (most) answers that mention nosql databases that include a form of map reduce, as those are not generic solutions: you must use that particular db in order to use their implementation of map reduce.

there's a notable exception: riak (http://riak.basho.com). Even when its implementation of map reduce is in principle tied to the db, in reality its architecture as evolved towards a more generic sustrate for distributed applications and the implementation of map reduce is much more generic. I don't believe it's still a fully independent mapreduce engine, but I would bet it's close (I've consider working on that direction myself)

With respect of some framework like phoenix and mars, they don't qualify as complete alternatives to hadoop as are targeted to multicore and/or graphic processor architectures and don't consider the distribution over large clusters.

Finally, I would consider storm (https://github.com/nathanmarz/storm). Even when it doesn't implement a map reduce model, its stream processing capabilities offer an alternative programming model and make relatively easy to implement map-reduce-like jobs.

Apache Spark is used. So is Apache Storm. So is Apache Hadoop. So are many related technology projects -- used in different parts of the company for different things. It's not about switching gears, it's about using the best tools for the job. There are many different products and platforms -- and they have differing use cases. So you'd expect Yahoo to rely upon many elements of the Big Data ecosystem, and not interpret this as a move from one to the other. Moreover, it's important for the large company to develop competence in many (seemingly competing) technologies so that they can expand options in case some workloads are better applied to one compute style vs another.

Come to the http://2015.hadoopsummit.org/ to learn more.

Yes.

Yahoo recently released an updated product report for 2015, highlighting the closure of several products, regional sites, and support for older Apple devices in some cases.

Maps is among the products that will be closed by the end of June. But the other Yahoo products will continue to support maps.

This is the snippet from the full report:

The Yahoo Maps site will close at the end of June. However, in the context of Yahoo search and on several other Yahoo properties including Flickr, we will continue to support maps. We made this decision to better align resources to Yahoo’s priorities as our business has evolved since we first launched Yahoo Maps eight years ago

Google (company)'s relationship with Hadoop has always been sort of interesting.

In the early days, Google (along with IBM (company)) built some educational training materials for universities to use for teaching MapReduce to its students [1]. These materials utilized Hadoop because Google's internal secret sauce was still pretty much hidden away.

Additionally, there was some work done to add users to Hadoop with funding by Google.

After Christophe Bisciglia left Google, their direct contributions have mostly dried up. But they still remain a 'force' in the ecosystem if only for Google Summer of Code (GSoC) and the occasional paper.

It's worth noting that Google has publically stated that they will not pursue Hadoop for any MapReduce patent violations. [2]

1 - http://googlepress.blogspot.com/2007/10/google-and-ibm-announce-university_08.html

2 - http://gigaom.com/2010/01/19/why-hadoop-users-shouldnt-fear-googles-new-mapreduce-patent/ amongst others.