{"id":140,"date":"2011-05-02T18:33:14","date_gmt":"2011-05-02T13:03:14","guid":{"rendered":"http:\/\/www.jkspeaks.com\/wordpress\/?p=140"},"modified":"2011-05-02T18:40:44","modified_gmt":"2011-05-02T13:10:44","slug":"hadoop-the-little-known-yellow-elephant","status":"publish","type":"post","link":"https:\/\/www.jkspeaks.com\/wordpress\/big-data\/hadoop-the-little-known-yellow-elephant\/","title":{"rendered":"Hadoop – The Little Known Yellow Elephant"},"content":{"rendered":"

What is Hadoop!<\/strong><\/p>\n

Apache Hadoop is an open source Java framework for processing and querying vast amounts of data (Multi Petabytes) on large clusters of commodity hardware. The original concept behind Hadoop comes from Google\u2019s BigTable. Hadoop is an initiative started and led by Yahoo! Today Apache Hadoop has become an enterprise-ready cloud computing technology and is becoming the industry de-facto framework for big data processing.<\/p>\n

\"The<\/a><\/p>\n

Yahoo! runs the world\u2019s largest Hadoop clusters. They work with academic institutions and other large corporations on advanced cloud computing research. Yahoo engineers are the leading participants in the Hadoop community.<\/p>\n

Why Hadoop?<\/strong><\/p>\n

Primary goal of Hadoop is to reduce the impact of a rack power outage or switch failure so that even if these events occur, the data may still be readable. This kind of reliability is achieved by replicating the data across multiple hosts, and removing the need for expensive RAID storage on hosts. To add to this, the replication and node failures are handled automatically.<\/p>\n

Another thing unique to Big Data is, unlike in traditional data warehouses where IO operations take a major chunk of the time i.e. bringing data to the server for processing, here the data locations are exposed, and processing is sent to the place where data resides. This provides a very high aggregate bandwidth. Think of sending a 2MB Jar file to the place where data resides against bringing 2GB of data to the server for processing.<\/p>\n

Hadoop support and tools are available from major enterprise players, such as Amazon, IBM and others. Almost every big internet company like facebook, NY Times, last.fm, Netflix, etc. are using Hadoop to some extent.<\/p>\n

When and where can Hadoop be used?<\/strong><\/p>\n

When processing can easily be made parallel (certain types of sort algorithms), running batch jobs is acceptable, access to lots of cheap hardware is easy and there are no real-time data \/ user facing requirements like Document Analysis & Indexing, Web Graphs and Crawling, Hadoop can be used.<\/p>\n

Applications that require a high degree of parallel data intensive distributed operation like very large production deployments (GRID), processing large amounts of unstructured data will also find Hadoop to be a best fit.<\/p>\n

Having said this, one should also know where it should not be used. Below are a few points on the same:<\/p>\n