One key difference is that [F-J][1] seems to be designed to work on a single Java VM, while [M-R][2] is explicitly designed to work on a large cluster of machines. These are very different scenarios.

F-J offers facilities to partition a task into several subtasks, in a recursive-looking fashion; more tiers, possibility of &#39;inter-fork&#39; communication at this stage, much more traditional programming. Does not extend (at least in the paper) beyond a single machine. Great for taking advantage of your eight-core.

M-R only does one big split, with the mapped splits not talking between each other at all, and then reduces everything together. A single tier, no inter-split communication until reduce, and massively scalable. Great for taking advantage of your share of the cloud.

  [1]: http://gee.cs.oswego.edu/dl/papers/fj.pdf
  [2]: https://static.googleusercontent.com/media/research.google.com/en//archive/mapreduce-osdi04.pdf

There is a whole scientific paper on the subject, [Comparing Fork/Join and MapReduce][1]. 

The paper compares the performance, scalability and programmability of three parallel paradigms: fork/join, MapReduce, and a hybrid approach. 

What they find is basically that Java fork/join has low startup latency and scales well for small inputs (&lt;5MB), but it cannot process larger inputs due to the size restrictions of shared-memory,
single node architectures. On the other hand, MapReduce has significant startup latency (tens of seconds), but scales well for much larger inputs (&gt;100MB) on a compute cluster.

But there is a lot more to read there if you&#39;re up for it.

  [1]: http://www.macs.hw.ac.uk/cs/techreps/docs/files/HW-MACS-TR-0096.pdf

Would this be a proper way to dispose of a BackGroundWorker?  I&#39;m not sure if it is necesary to remove the events before calling .Dispose().  Also is calling .Dispose() inside the RunWorkerCompleted delegate ok to do?

    public void RunProcessAsync(DateTime dumpDate)
    {
        BackgroundWorker worker = new BackgroundWorker();
        worker.RunWorkerCompleted += new RunWorkerCompletedEventHandler(worker_RunWorkerCompleted);
        worker.DoWork += new DoWorkEventHandler(worker_DoWork);
        worker.RunWorkerAsync(dumpDate);
    }

    void worker_DoWork(object sender, DoWorkEventArgs e)
    {
        // Do Work here
    }

    void worker_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
    {
        BackgroundWorker worker = sender as BackgroundWorker;
        worker.RunWorkerCompleted -= new RunWorkerCompletedEventHandler(worker_RunWorkerCompleted);
        worker.DoWork -= new DoWorkEventHandler(worker_DoWork);
        worker.Dispose();
    }



Proper way to Dispose of a BackGroundWorker

How can I test all of the following methods code? I want to play scenarios when all of them are happening to see if my code works for save/restore process of an activity. So what should I do in the Emulator to get all methods tested?

    public class Activity extends ApplicationContext {
         protected void onCreate(Bundle savedInstanceState);
    
         protected void onStart();

         protected void onRestoreInstanceState(Bundle savedInstanceState);

         protected void onSaveInstanceState(Bundle savedInstanceState);
         
         protected void onRestart();
    
         protected void onResume();
    
         protected void onPause();
    
         protected void onStop();
    
         protected void onDestroy();
     }

How to test code built to save/restore Lifecycle of an Activity?

What is the key difference between Fork/Join and Map/Reduce?

Do they differ in the kind of decomposition and distribution (data vs. computation)?


Difference between Fork/Join and Map/Reduce

<p>What is the key difference between Fork/Join and Map/Reduce?</p>
<p>Do they differ in the kind of decomposition and distribution (data vs. computation)?</p>


I want to debug a mapreduce script, and without going into much trouble tried to put some print statements in my program. But I cant seem to find them in any of the logs. 

Where does hadoop mapreduce framework send my System.out.print() statements ? (stdout)

I have a large CSV file containing a list of stores, in which one of the field is ZipCode.
I have a separate MongoDB database called ZipCodes, which stores the latitude and longitude for any given zip code.


In SQL Server, I would execute a stored procedure called InsertStore which would do a look up on the ZipCodes table to get corresponding latitude and longitude and insert the data into the Stores table.

Is there something similar to the concept of stored procedures in MongoDB for this?
Basically, for each insert I need to look up the latitude and longitude for that store and save that as well.

I am not too familiar with the concept of Map/Reduce, but would that be relevant here?
Thank you!

MongoDB Stored Procedure Equivalent

In mapreduce each reduce task write its output to a file named *part-r-nnnnn* where *nnnnn* is a partition ID associated with the reduce task. Does *map/reduce* merge these files? If yes, how?

merge output files after reduce phase

Suppose I have a collection with some set of documents. something like this.

    { &quot;_id&quot; : ObjectId(&quot;4f127fa55e7242718200002d&quot;), &quot;id&quot;:1, &quot;name&quot; : &quot;foo&quot;}
    { &quot;_id&quot; : ObjectId(&quot;4f127fa55e7242718200002d&quot;), &quot;id&quot;:2, &quot;name&quot; : &quot;bar&quot;}
    { &quot;_id&quot; : ObjectId(&quot;4f127fa55e7242718200002d&quot;), &quot;id&quot;:3, &quot;name&quot; : &quot;baz&quot;}
    { &quot;_id&quot; : ObjectId(&quot;4f127fa55e7242718200002d&quot;), &quot;id&quot;:4, &quot;name&quot; : &quot;foo&quot;}
    { &quot;_id&quot; : ObjectId(&quot;4f127fa55e7242718200002d&quot;), &quot;id&quot;:5, &quot;name&quot; : &quot;bar&quot;}
    { &quot;_id&quot; : ObjectId(&quot;4f127fa55e7242718200002d&quot;), &quot;id&quot;:6, &quot;name&quot; : &quot;bar&quot;}

I want to find all the duplicated entries in this collection by the &quot;name&quot; field. E.g. &quot;foo&quot; appears twice and &quot;bar&quot; appears 3 times.

Find all duplicate documents in a MongoDB collection by a key field

I am getting:

    FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask

While trying to make a copy of a partitioned table using the commands in the hive console:

    CREATE TABLE copy_table_name LIKE table_name;
    INSERT OVERWRITE TABLE copy_table_name PARTITION(day) SELECT * FROM table_name;

I initially got some semantic analysis errors and had to set:

    set hive.exec.dynamic.partition=true
    set hive.exec.dynamic.partition.mode=nonstrict

Although I&#39;m not sure what the above properties do? 

Full ouput from hive console:


    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks determined at compile time: 1
    In order to change the average load for a reducer (in bytes):
      set hive.exec.reducers.bytes.per.reducer=&lt;number&gt;
    In order to limit the maximum number of reducers:
      set hive.exec.reducers.max=&lt;number&gt;
    In order to set a constant number of reducers:
      set mapred.reduce.tasks=&lt;number&gt;
    Starting Job = job_201206191101_4557, Tracking URL = http://jobtracker:50030/jobdetails.jsp?jobid=job_201206191101_4557
    Kill Command = /usr/lib/hadoop/bin/hadoop job  -Dmapred.job.tracker=master:8021 -kill job_201206191101_4557
    2012-06-25 09:53:05,826 Stage-1 map = 0%,  reduce = 0%
    2012-06-25 09:53:53,044 Stage-1 map = 100%,  reduce = 100%
    Ended Job = job_201206191101_4557 with errors
    FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask



What is Hive: Return Code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask

The event-driven programming model of node.js makes it somewhat tricky to coordinate the program flow.

Simple sequential execution gets turned into nested callbacks, which is easy enough (though a bit convoluted to write down). 

But how about parallel execution? Say you have three tasks A,B,C that can run in parallel and when they are done, you want to send their results to task D.

With a fork/join model this would be

  * fork A
  * fork B
  * fork C
  * join A,B,C, run D

How do I write that in node.js ? Are there any best practices or cookbooks? Do I have to [hand-roll a solution][1] every time, or is there some library with helpers for this?


  [1]: https://stackoverflow.com/questions/4496403/extjs-two-parallel-ajax-call

Coordinating parallel execution in node.js

What are the benefits of using the new [fork/join framework](http://download.oracle.com/javase/tutorial/essential/concurrency/forkjoin.html) over just simply splitting the big task into N subtasks in the beginning, sending them to a  cached thread pool (from [Executors](http://download.oracle.com/javase/7/docs/api/java/util/concurrent/Executors.html)) and waiting for each task to complete? I fail to see how using the fork/join abstraction simplifies the problem or makes the solution more efficient from what we&#39;ve had for years now.

For example, the parallelized blurring algorithm in the [tutorial example](http://download.oracle.com/javase/tutorial/essential/concurrency/forkjoin.html) could be implemented like this:


    public class Blur implements Runnable {
        private int[] mSource;
        private int mStart;
        private int mLength;
        private int[] mDestination;
  
        private int mBlurWidth = 15; // Processing window size, should be odd.
  
        public ForkBlur(int[] src, int start, int length, int[] dst) {
            mSource = src;
            mStart = start;
            mLength = length;
            mDestination = dst;
        }

        public void run() {
            computeDirectly();
        }

        protected void computeDirectly() {
            // As in the example, omitted for brevity
        }
    }

Split in the beginning and send tasks to a thread pool:

    // source image pixels are in src
    // destination image pixels are in dst
    // threadPool is a (cached) thread pool

    int maxSize = 100000; // analogous to F-J&#39;s &quot;sThreshold&quot;
    List&lt;Future&gt; futures = new ArrayList&lt;Future&gt;();

    // Send stuff to thread pool:
    for (int i = 0; i &lt; src.length; i+= maxSize) {
        int size = Math.min(maxSize, src.length - i);
        ForkBlur task = new ForkBlur(src, i, size, dst);
        Future f = threadPool.submit(task);
        futures.add(f);
    }
 
    // Wait for all sent tasks to complete:
    for (Future future : futures) {
        future.get();
    }

    // Done!

The tasks go to the thread pool&#39;s queue, from which they&#39;re executed as worker threads become available. As long as the splitting is granular enough (to avoid having to particularly wait for the last task) and the thread pool has enough (at least N of processors) threads, all processors are working at full speed until the whole computation is done.

Am I missing something? What&#39;s the added value of using the fork/join framework?

How is the fork/join framework better than a thread pool?

As far as I had understood `ForkJoinPool`, that pool creates a fixed number of threads (default: number of cores) and will never create more threads (unless the application indicates a need for those by using `managedBlock`). 

However, using `ForkJoinPool.getPoolSize()` I discovered that in a program that creates 30,000 tasks (`RecursiveAction`), the `ForkJoinPool` executing those tasks uses 700 threads on average (threads counted each time a task is created). The tasks don&#39;t do I/O, but pure computation; the only inter-task synchronization is calling `ForkJoinTask.join()` and accessing `AtomicBoolean`s, i.e. there are no thread-blocking operations. 

Since `join()` does not block the calling thread as I understand it, there is no reason why any thread in the pool should ever block, and so (I had assumed) there should be no reason to create any further threads (which is obviously happening nevertheless).

So, why does `ForkJoinPool` create so many threads? What factors determine the number of threads created?

I had hoped that this question could be answered without posting code, but here it comes upon request. This code is an excerpt from a program of four times the size, reduced to the essential parts; it does not compile as it is. If desired, I can of course post the full program, too.

The program searches a maze for a path from a given start point to a given end point using depth-first search. A solution is guaranteed to exist. The main logic is in the `compute()` method of `SolverTask`: A `RecursiveAction` that starts at some given point and continues with all neighbor points reachable from the current point. Rather than creating a new `SolverTask` at each branching point (which would create far too many tasks), it pushes all neighbors except one onto a backtracking stack to be processed later and continues with only the one neighbor not pushed to the stack. Once it reaches a dead end that way, the point most recently pushed to the backtracking stack is popped, and the search continues from there (cutting back the path built from the taks&#39;s starting point accordingly). A new task is created once a task finds its backtracking stack larger than a certain threshold; from that time, the task, while continuing to pop from its backtracking stack until that is exhausted, will not push any further points to its stack when reaching a branching point, but create a new task for each such point. Thus, the size of the tasks can be adjusted using the stack limit threshold. 

The numbers I quoted above (&quot;30,000 tasks, 700 threads on average&quot;) are from searching a maze of 5000x5000 cells. So, here is the essential code:

    class SolverTask extends RecursiveTask&lt;ArrayDeque&lt;Point&gt;&gt; {
	// Once the backtrack stack has reached this size, the current task
	// will never add another cell to it, but create a new task for each
	// newly discovered branch:
	private static final int MAX_BACKTRACK_CELLS = 100*1000;

	/**
	 * @return Tries to compute a path through the maze from local start to end
	 * and returns that (or null if no such path found)
	 */
	@Override
	public ArrayDeque&lt;Point&gt;  compute() {
		// Is this task still accepting new branches for processing on its own,
		// or will it create new tasks to handle those?
		boolean stillAcceptingNewBranches = true;
		Point current = localStart;
		ArrayDeque&lt;Point&gt; pathFromLocalStart = new ArrayDeque&lt;Point&gt;();  // Path from localStart to (including) current
		ArrayDeque&lt;PointAndDirection&gt; backtrackStack = new ArrayDeque&lt;PointAndDirection&gt;();
		// Used as a stack: Branches not yet taken; solver will backtrack to these branching points later

		Direction[] allDirections = Direction.values();
		
		while (!current.equals(end)) {
			pathFromLocalStart.addLast(current);
			// Collect current&#39;s unvisited neighbors in random order: 
			ArrayDeque&lt;PointAndDirection&gt; neighborsToVisit = new ArrayDeque&lt;PointAndDirection&gt;(allDirections.length);  
			for (Direction directionToNeighbor: allDirections) {
				Point neighbor = current.getNeighbor(directionToNeighbor);
				
				// contains() and hasPassage() are read-only methods and thus need no synchronization
				if (maze.contains(neighbor) &amp;&amp; maze.hasPassage(current, neighbor) &amp;&amp; maze.visit(neighbor))
					neighborsToVisit.add(new PointAndDirection(neighbor, directionToNeighbor.opposite));
			}
			// Process unvisited neighbors
			if (neighborsToVisit.size() == 1) {
				// Current node is no branch: Continue with that neighbor
				current = neighborsToVisit.getFirst().getPoint();
				continue;
			}
			if (neighborsToVisit.size() &gt;= 2) {
				// Current node is a branch
				if (stillAcceptingNewBranches) {
					current = neighborsToVisit.removeLast().getPoint();
					// Push all neighbors except one on the backtrack stack for later processing
					for(PointAndDirection neighborAndDirection: neighborsToVisit) 
						backtrackStack.push(neighborAndDirection);
					if (backtrackStack.size() &gt; MAX_BACKTRACK_CELLS)
						stillAcceptingNewBranches = false;
					// Continue with the one neighbor that was not pushed onto the backtrack stack
					continue;
				} else {
					// Current node is a branch point, but this task does not accept new branches any more: 
					// Create new task for each neighbor to visit and wait for the end of those tasks
					SolverTask[] subTasks = new SolverTask[neighborsToVisit.size()];
					int t = 0;
					for(PointAndDirection neighborAndDirection: neighborsToVisit)  {
						SolverTask task = new SolverTask(neighborAndDirection.getPoint(), end, maze);
						task.fork();
						subTasks[t++] = task;
					}
					for (SolverTask task: subTasks) {
						ArrayDeque&lt;Point&gt; subTaskResult = null;
						try {
							subTaskResult = task.join();
						} catch (CancellationException e) {
							// Nothing to do here: Another task has found the solution and cancelled all other tasks
						}
						catch (Exception e) {
							e.printStackTrace();
						}
						if (subTaskResult != null) { // subtask found solution
							pathFromLocalStart.addAll(subTaskResult);
							// No need to wait for the other subtasks once a solution has been found
							return pathFromLocalStart;
						}
					} // for subTasks
				} // else (not accepting any more branches) 
			} // if (current node is a branch)
		    // Current node is dead end or all its neighbors lead to dead ends:
		    // Continue with a node from the backtracking stack, if any is left:
			if (backtrackStack.isEmpty()) {
				return null; // No more backtracking avaible: No solution exists =&gt; end of this task
			}
			// Backtrack: Continue with cell saved at latest branching point:
			PointAndDirection pd = backtrackStack.pop();
			current = pd.getPoint();
			Point branchingPoint = current.getNeighbor(pd.getDirectionToBranchingPoint());
			// DEBUG System.out.println(&quot;Backtracking to &quot; +  branchingPoint);
			// Remove the dead end from the top of pathSoFar, i.e. all cells after branchingPoint:
			while (!pathFromLocalStart.peekLast().equals(branchingPoint)) {
				// DEBUG System.out.println(&quot;    Going back before &quot; + pathSoFar.peekLast());
				pathFromLocalStart.removeLast();
			}
			// continue while loop with newly popped current
		} // while (current ...
		if (!current.equals(end)) {			
			// this task was interrupted by another one that already found the solution 
			// and should end now therefore:
			return null;
		} else {
			// Found the solution path:
			pathFromLocalStart.addLast(current);
			return pathFromLocalStart;
		}
	} // compute()
    } // class SolverTask

    @SuppressWarnings(&quot;serial&quot;)
    public class ParallelMaze  {
 	
	// for each cell in the maze: Has the solver visited it yet?
 	private final AtomicBoolean[][] visited;

	/**
	 * Atomically marks this point as visited unless visited before
	 * @return whether the point was visited for the first time, i.e. whether it could be marked
	 */
	boolean visit(Point p) {
		return  visited[p.getX()][p.getY()].compareAndSet(false, true);
	}

	public static void main(String[] args) {
		ForkJoinPool pool = new ForkJoinPool();
		ParallelMaze maze = new ParallelMaze(width, height, new Point(width-1, 0), new Point(0, height-1));
	    // Start initial task
		long startTime = System.currentTimeMillis();
		 // since SolverTask.compute() expects its starting point already visited, 
		// must do that explicitly for the global starting point:
		maze.visit(maze.start);
		maze.solution = pool.invoke(new SolverTask(maze.start, maze.end, maze));
		// One solution is enough: Stop all tasks that are still running
		pool.shutdownNow();
	    pool.awaitTermination(Integer.MAX_VALUE, TimeUnit.DAYS);
		long endTime = System.currentTimeMillis();
		System.out.println(&quot;Computed solution of length &quot; + maze.solution.size() + &quot; to maze of size &quot; + 
				width + &quot;x&quot; + height + &quot; in &quot; + ((float)(endTime - startTime))/1000 + &quot;s.&quot;);
    }



What determines the number of threads a Java ForkJoinPool creates?

I just finished reading this post: https://stackoverflow.com/questions/9276807/whats-the-advantage-of-a-java-5-threadpoolexecutor-over-a-java-7-forkjoinpool and felt that the answer is not straight enough.

Can you explain in simple language and examples, what are the **trade-offs** between Java 7&#39;s Fork-Join framework and the older solutions?

I also read the Google&#39;s #1 hit on the topic [Java Tip: When to use ForkJoinPool vs ExecutorService][1] from [javaworld.com][2] but the article doesn&#39;t answer the title question **when**, it talks about api differences mostly ...


  [1]: http://www.javaworld.com/article/2078440/enterprise-java/java-tip-when-to-use-forkjoinpool-vs-executorservice.html

  [2]: https://www.javaworld.com/

Java&#39;s Fork/Join vs ExecutorService - when to use which?

I came across a strange situation where using a parallel stream with a lambda in a static initializer takes seemingly forever with no CPU utilization. Here&#39;s the code:

	class Deadlock {
		static {
			IntStream.range(0, 10000).parallel().map(i -&gt; i).count();
			System.out.println(&quot;done&quot;);
		}
		public static void main(final String[] args) {}
	}

This appears to be a minimum reproducing test case for this behavior. If I:

* put the block in the main method instead of a static initializer,
* remove parallelization, or
* remove the lambda,

the code instantly completes. Can anyone explain this behavior? Is it a bug or is this intended?

I am using OpenJDK version 1.8.0_66-internal.

Content Type	Original Author	Original Content on Stackoverflow
Question	hotzen	View Question on Stackoverflow
Solution 1 - Mapreduce	tucuxi	View Answer on Stackoverflow
Solution 2 - Mapreduce	Per Quested Aronsson	View Answer on Stackoverflow

Difference between Fork/Join and Map/Reduce

Mapreduce Problem Overview

Mapreduce Solutions

Solution 1 - Mapreduce

Solution 2 - Mapreduce

How to test code built to save/restore Lifecycle of an Activity?

Proper way to Dispose of a BackGroundWorker

Attributions