I want to debug a mapreduce script, and without going into much trouble tried to put some print statements in my program. But I cant seem to find them in any of the logs. 

Where does hadoop mapreduce framework send my System.out.print() statements ? (stdout)

Is the aggregation framework introduced in mongodb 2.2, has any special performance improvements over map/reduce?

If yes, why and how and how much?

(Already I have done a test for myself, and the performance was nearly same)

Is Mongodb Aggregation framework faster than map/reduce?

What is the key difference between Fork/Join and Map/Reduce?

Do they differ in the kind of decomposition and distribution (data vs. computation)?


Difference between Fork/Join and Map/Reduce

The default data block size of HDFS/Hadoop is 64MB. The block size in the disk is generally 4KB.

What does 64MB block size mean? -&gt;Does it mean that the smallest unit of reading from disk is 64MB?

If yes, what is the advantage of doing that?-&gt; easy for continuous access of large files in HDFS?

Can we do the same by using the disk&#39;s original 4KB block size?

data block size in HDFS, why 64MB?

I know how to &quot;transform&quot; a simple Java `List` from `Y` -&gt; `Z`, i.e.:

    List&lt;String&gt; x;
	List&lt;Integer&gt; y = x.stream()
            .map(s -&gt; Integer.parseInt(s))
            .collect(Collectors.toList());

Now I&#39;d like to do basically the same with a Map, i.e.:

    INPUT:
    {
      &quot;key1&quot; -&gt; &quot;41&quot;,    // &quot;41&quot; and &quot;42&quot;
      &quot;key2&quot; -&gt; &quot;42&quot;      // are Strings
    }

    OUTPUT:
    {
      &quot;key1&quot; -&gt; 41,      // 41 and 42
      &quot;key2&quot; -&gt; 42       // are Integers
    }

The solution should not be limited to `String` -&gt; `Integer`. Just like in the `List` example above, I&#39;d like to call any method (or constructor).

Java8: HashMap&lt;X, Y&gt; to HashMap&lt;X, Z&gt; using Stream / Map-Reduce / Collector

What scenarios would warrant the use of the &quot;[Map and Reduce][1]&quot; algorithm?

[1]:http://en.wikipedia.org/wiki/MapReduce

&lt;br /&gt; Is there a .NET implementation of this algorithm? 

Map and Reduce in .NET

I couldn&#39;t think of any good examples other than the &quot;how to count words in a long text with MapReduce&quot; task. I found this wasn&#39;t the best example to give others an impression of how powerful this tool can be.

I&#39;m not looking for code-snippets, really just &quot;textual&quot; examples.

Good MapReduce examples

Related to my [CouchDB](https://stackoverflow.com/questions/28975/anyone-using-couchdb) question.

Can anyone explain MapReduce in terms a numbnuts could understand?

Simple explanation of MapReduce?

In Map Reduce programming the reduce phase has shuffling, sorting and reduce as its sub-parts. Sorting is a costly affair.

*What is the purpose of shuffling and sorting phase in the reducer in Map Reduce Programming?*
  

What is the purpose of shuffling and sorting phase in the reducer in Map Reduce Programming?

In many real-life situations where you apply MapReduce, the final algorithms end up being several MapReduce steps.

i.e. Map1 , Reduce1 , Map2 , Reduce2 , and so on.

So you have the output from the last reduce that is needed as the input for the next map. 

The intermediate data is something you (in general) do not want to keep once the pipeline has been successfully completed. Also because this intermediate data is in general some data structure (like a &#39;map&#39; or a &#39;set&#39;) you don&#39;t want to put too much effort in writing and reading these key-value pairs.

What is the recommended way of doing that in Hadoop?

Is there a (simple) example that shows how to handle this intermediate data in the correct way, including the cleanup afterward?

Chaining multiple MapReduce jobs in Hadoop

According to the `Hadoop - The Definitive Guide`

&gt; The logical records that FileInputFormats define do not usually fit neatly into HDFS blocks. For example, a TextInputFormat’s logical records are lines, which will cross HDFS boundaries more often than not. This has no bearing on the functioning of your program—lines are not missed or broken, for example—but it’s worth knowing about, as it does mean that data-local maps (that is, maps that are running on the same host as their input data) will perform some remote reads. The slight overhead this causes is not normally significant.

Suppose a record line is split across two blocks (b1 and b2). The mapper processing the first block (b1) will notice that the last line doesn&#39;t have a EOL separator and fetches the remaining of the line from the next block of data (b2).

How does the mapper processing the second block (b2) determine that the first record is incomplete and should process starting from the second record in the block (b2)?

How does Hadoop process records split across block boundaries?

One of the main examples that is used in demonstrating the power of MapReduce is the [Terasort benchmark][1]. I&#39;m having trouble understanding the basics of the sorting algorithm used in the MapReduce environment. 

To me sorting simply involves determining the relative position of an element in relationship to all other elements. So sorting involves comparing &quot;everything&quot; with &quot;everything&quot;. Your average sorting algorithm (quick, bubble, ...) simply does this in a smart way.

In my mind splitting the dataset into many pieces means you can sort a single piece and then you still have to integrate these pieces into the &#39;complete&#39; fully sorted dataset. Given the terabyte dataset distributed over thousands of systems I expect this to be a huge task.

So how is this really done? How does this MapReduce sorting algorithm work?

Thanks for helping me understand.

  [1]: http://developer.yahoo.net/blogs/hadoop/2008/07/apache_hadoop_wins_terabyte_sort_benchmark.html

How does the MapReduce sort algorithm work?

Are there any dependencies between ***Spark*** and ***Hadoop***?

If not, are there any features I&#39;ll miss when I run ***Spark*** without ***Hadoop***?


Can apache spark run without hadoop?

When using MongoDB&#39;s [`$in`][1] clause, does the order of the returned documents always correspond to the order of the array argument?


  [1]: http://docs.mongodb.org/manual/reference/operator/query/in/

Does MongoDB&#39;s $in clause guarantee order

So, I&#39;ve been looking at [Hadoop][1] with keen interest, and to be honest I&#39;m fascinated, things don&#39;t get much cooler.

My only minor issue is I&#39;m a C# developer and it&#39;s in Java.

It&#39;s not that I don&#39;t understand the Java as much as I&#39;m looking for the Hadoop.net or NHadoop or the .NET project that embraces the [Google MapReduce][2] approach. Does anyone know of one?


  [1]: http://hadoop.apache.org/
  [2]: http://en.wikipedia.org/wiki/MapReduce

Is there a .NET equivalent to Apache Hadoop?

In Hadoop v1, I have assigned each 7 mapper and reducer slot with size of 1GB, my mappers &amp; reducers runs fine. My machine has 8G memory, 8 processor. 
Now with YARN, when run the same application on the same machine, I got container error. 
By default, I have this settings:
  

      &lt;property&gt;
        &lt;name&gt;yarn.scheduler.minimum-allocation-mb&lt;/name&gt;
        &lt;value&gt;1024&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
        &lt;name&gt;yarn.scheduler.maximum-allocation-mb&lt;/name&gt;
        &lt;value&gt;8192&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
        &lt;name&gt;yarn.nodemanager.resource.memory-mb&lt;/name&gt;
        &lt;value&gt;8192&lt;/value&gt;
      &lt;/property&gt;
 
It gave me error:

    Container [pid=28920,containerID=container_1389136889967_0001_01_000121] is running beyond virtual memory limits. Current usage: 1.2 GB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

I then tried to set memory limit in mapred-site.xml:

      &lt;property&gt;
        &lt;name&gt;mapreduce.map.memory.mb&lt;/name&gt;
        &lt;value&gt;4096&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
        &lt;name&gt;mapreduce.reduce.memory.mb&lt;/name&gt;
        &lt;value&gt;4096&lt;/value&gt;
      &lt;/property&gt;
But still getting error:

    Container [pid=26783,containerID=container_1389136889967_0009_01_000002] is running beyond physical memory limits. Current usage: 4.2 GB of 4 GB physical memory used; 5.2 GB of 8.4 GB virtual memory used. Killing container.
I&#39;m confused why the the map task need this much memory. In my understanding, 1GB of memory is enough for my map/reduce task. Why as I assign more memory to container, the task use more? Is it because each task gets more splits? I feel it&#39;s more efficient to decrease the size of container a little bit and create more containers, so that more tasks are running in parallel. The problem is how can I make sure each container won&#39;t be assigned more splits than it can handle?   

Container is running beyond memory limits

I hear a lot about map/reduce, especially in the context of Google&#39;s massively parallel compute system. What exactly is it?

What is Map/Reduce?

In Hadoop when do reduce tasks start? Do they start after a certain percentage (threshold) of mappers complete? If so, is this threshold fixed? What kind of threshold is typically used?

When do reduce tasks start in Hadoop?

I have a large CSV file containing a list of stores, in which one of the field is ZipCode.
I have a separate MongoDB database called ZipCodes, which stores the latitude and longitude for any given zip code.


In SQL Server, I would execute a stored procedure called InsertStore which would do a look up on the ZipCodes table to get corresponding latitude and longitude and insert the data into the Stores table.

Is there something similar to the concept of stored procedures in MongoDB for this?
Basically, for each insert I need to look up the latitude and longitude for that store and save that as well.

I am not too familiar with the concept of Map/Reduce, but would that be relevant here?
Thank you!

MongoDB Stored Procedure Equivalent

For now, I have a Hadoop job which creates counters with a pretty big name. 

For example, the following one: `stats.counters.server-name.job.job-name.mapper.site.site-name.qualifier.qualifier-name.super-long-string-which-is-not-within-standard-limits`. This counter is truncated on web interface and on `getName()` method call. I&#39;ve found out that Hadoop has limitations on the counter max name and this settings id `mapreduce.job.counters.counter.name.max` is for configuring this limit. So I incremented this to `500` and web interface now shows full counter name. But `getName()` of the counter still returns truncated name.  
  
Could somebody, please, explain this or point me on my mistakes? Thank you.

**EDIT 1**

My Hadoop server configuration consists of the single server with HDFS, YARN, and map-reduce itself on it. During map-reduce, there are some counter increments and after the job is completed, in `ToolRunner` I fetch counters with the use of `org.apache.hadoop.mapreduce.Job#getCounters`.  
  
  
**EDIT 2**  
  
Hadoop version is the following:

    Hadoop 2.6.0-cdh5.8.0
    Subversion http://github.com/cloudera/hadoop -r 042da8b868a212c843bcbf3594519dd26e816e79 
    Compiled by jenkins on 2016-07-12T22:55Z
    Compiled with protoc 2.5.0
    From source with checksum 2b6c319ecc19f118d6e1c823175717b5
    This command was run using /usr/lib/hadoop/hadoop-common-2.6.0-cdh5.8.0.jar

I made some additional investigation and it seems that [this issue][1] describes a situation similar to mine. But it&#39;s pretty confusing cause I&#39;m able to increase the number of counters but not the length of counter&#39;s name...
  
**EDIT 3**  
  
Today, I spent pretty much time debugging internals of the Hadoop. Some interesting stuff:  

 1. `org.apache.hadoop.mapred.ClientServiceDelegate#getJobCounters` method returns a bunch of counters from yarn with *TRUNCATED* names and *FULL* display names.
 2. Was unable to debug maps and reducers itself but with help of logging it seems that `org.apache.hadoop.mapreduce.Counter#getName` method works correctly during reducer execution.


  [1]: https://issues.apache.org/jira/browse/MAPREDUCE-5875

Hadoop truncated/inconsistent counter name

I commonly work with text files of ~20 Gb size and I find myself counting the number of lines in a given file very often.

The way I do it now it&#39;s just `cat fname | wc -l`, and it takes very long.  Is there any solution that&#39;d be much faster?

I work in a high performance cluster with Hadoop installed. I was wondering if a map reduce approach could help.

I&#39;d like the solution to be as simple as one line run, like the `wc -l` solution, but not sure how feasible it is.

Any ideas?

Count lines in large files

In mapreduce each reduce task write its output to a file named *part-r-nnnnn* where *nnnnn* is a partition ID associated with the reduce task. Does *map/reduce* merge these files? If yes, how?

merge output files after reduce phase

I am getting:

    FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask

While trying to make a copy of a partitioned table using the commands in the hive console:

    CREATE TABLE copy_table_name LIKE table_name;
    INSERT OVERWRITE TABLE copy_table_name PARTITION(day) SELECT * FROM table_name;

I initially got some semantic analysis errors and had to set:

    set hive.exec.dynamic.partition=true
    set hive.exec.dynamic.partition.mode=nonstrict

Although I&#39;m not sure what the above properties do? 

Full ouput from hive console:


    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks determined at compile time: 1
    In order to change the average load for a reducer (in bytes):
      set hive.exec.reducers.bytes.per.reducer=&lt;number&gt;
    In order to limit the maximum number of reducers:
      set hive.exec.reducers.max=&lt;number&gt;
    In order to set a constant number of reducers:
      set mapred.reduce.tasks=&lt;number&gt;
    Starting Job = job_201206191101_4557, Tracking URL = http://jobtracker:50030/jobdetails.jsp?jobid=job_201206191101_4557
    Kill Command = /usr/lib/hadoop/bin/hadoop job  -Dmapred.job.tracker=master:8021 -kill job_201206191101_4557
    2012-06-25 09:53:05,826 Stage-1 map = 0%,  reduce = 0%
    2012-06-25 09:53:53,044 Stage-1 map = 100%,  reduce = 100%
    Ended Job = job_201206191101_4557 with errors
    FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask



What is Hive: Return Code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask

I&#39;m trying to write a non-trivial Hive job using the Hive Thrift and JDBC interfaces, and I&#39;m having trouble setting up a decent JUnit test. By non-trivial, I mean that the job results in at least one MapReduce stage, as opposed to only dealing with the metastore.

The test should fire up a Hive server, load some data into a table, run some non-trivial query on that table, and check the results.

I&#39;ve wired up a Spring context according to the [Spring reference][1]. However, the job fails on the MapReduce phase, complaining that no Hadoop binary exists:


&gt; java.io.IOException: Cannot run program &quot;/usr/bin/hadoop&quot; (in
&gt; directory &quot;/Users/yoni/opower/workspace/intellij_project_root&quot;):
&gt; error=2, No such file or directory



The problem is that the Hive Server is running in-memory, but relies upon local installation of Hive in order to run. For my project to be self-contained, I need the Hive services to be embedded, including the HDFS and MapReduce clusters. I&#39;ve tried starting up a Hive server using the same Spring method and pointing it at [MiniDFSCluster][2] and [MiniMRCluster][3], similar to the pattern used in the Hive [QTestUtil][4] source and in [HBaseTestUtility][5]. However, I&#39;ve not been able to get that to work.

After three days of trying to wrangle Hive integration testing, I thought I&#39;d ask the community: 

 1. How do you recommend I integration test Hive jobs?
 2. Do you have a working JUnit example for integration testing Hive jobs using in-memory HDFS, MR, and Hive instances?

Additional resources I&#39;ve looked at:

* [Hive Unit Testing tutorial][6]
* [the Spring Hive example][7]


  [1]: http://static.springsource.org/spring-hadoop/docs/1.0.0.RELEASE/reference/html/hive.html &quot;Spring Hive context&quot;
  [2]: http://javasourcecode.org/html/open-source/hadoop/hadoop-0.20.203.0/org/apache/hadoop/hdfs/MiniDFSCluster.html
  [3]: http://javasourcecode.org/html/open-source/hadoop/hadoop-0.20.2/org/apache/hadoop/mapred/MiniMRCluster.html
  [4]: http://hive.apache.org/docs/r0.9.0/api/org/apache/hadoop/hive/ql/QTestUtil.html
  [5]: http://people.apache.org/~psmith/hbase/sandbox/hbase/hbase-core/testapidocs/org/apache/hadoop/hbase/HBaseTestingUtility.html
  [6]: http://dev.bizo.com/2011/04/hive-unit-testing.html &quot;Unit testing Hive&quot;
  [7]: https://github.com/SpringSource/spring-hadoop-samples/tree/master/samples/hive &quot;Spring Hive example&quot;


Edit:
I am fully aware that working against a Hadoop cluster - whether local or remote - makes it possible to run integration tests against a full-stack Hive instance. The problem, as stated, is that this is not a viable solution for effectively testing Hive workflows.

Integration testing Hive jobs

Suppose I have a collection with some set of documents. something like this.

    { &quot;_id&quot; : ObjectId(&quot;4f127fa55e7242718200002d&quot;), &quot;id&quot;:1, &quot;name&quot; : &quot;foo&quot;}
    { &quot;_id&quot; : ObjectId(&quot;4f127fa55e7242718200002d&quot;), &quot;id&quot;:2, &quot;name&quot; : &quot;bar&quot;}
    { &quot;_id&quot; : ObjectId(&quot;4f127fa55e7242718200002d&quot;), &quot;id&quot;:3, &quot;name&quot; : &quot;baz&quot;}
    { &quot;_id&quot; : ObjectId(&quot;4f127fa55e7242718200002d&quot;), &quot;id&quot;:4, &quot;name&quot; : &quot;foo&quot;}
    { &quot;_id&quot; : ObjectId(&quot;4f127fa55e7242718200002d&quot;), &quot;id&quot;:5, &quot;name&quot; : &quot;bar&quot;}
    { &quot;_id&quot; : ObjectId(&quot;4f127fa55e7242718200002d&quot;), &quot;id&quot;:6, &quot;name&quot; : &quot;bar&quot;}

I want to find all the duplicated entries in this collection by the &quot;name&quot; field. E.g. &quot;foo&quot; appears twice and &quot;bar&quot; appears 3 times.

All Mapreduce Solutions on Gang of Coders

Total of 25 Mapreduce Solutions