Stdout buffer issue using node child_process

Javascriptnode.js

Javascript Problem Overview


I'm trying to execute curl using node child_process to get a JSON file (about 220Ko) from a shared folder in a local network. But it actually returns a buffer problem that I can't get throught. Here is my code:

var exec = require('child_process').exec;

var execute = function(command, callback){
    exec(command, function(error, stdout, stderr){ callback(error, stdout); });
};

execute("curl http://" + ip + "/file.json", function(err, json, outerr) {
    if(err) throw err;
    console.log(json);
})

And here is the error I get:

if(err) throw err;
          ^
Error: stdout maxBuffer exceeded.
    at Socket.<anonymous> (child_process.js:678:13)
    at Socket.EventEmitter.emit (events.js:95:17)
    at Socket.<anonymous> (_stream_readable.js:746:14)
    at Socket.EventEmitter.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:408:10)
    at emitReadable (_stream_readable.js:404:5)
    at readableAddChunk (_stream_readable.js:165:9)
    at Socket.Readable.push (_stream_readable.js:127:10)
    at Pipe.onread (net.js:526:21)

Javascript Solutions


Solution 1 - Javascript

You need to use and set the maxBuffer option when using child_process.exec. From the documentation:

> maxBuffer specifies the largest amount of data allowed on stdout or stderr - if this value is exceeded then the child process is killed.

The documentation also states that the default value of maxBuffer is 200KB.

As an example, the maximum buffer size is increased to 500KB in the following code:

var execute = function(command, callback){
    exec(command, {maxBuffer: 1024 * 500}, function(error, stdout, stderr){ callback(error, stdout); });
};

Additionally, you may want to read about http.get to see if it is capable of achieving what you are trying to do.

Solution 2 - Javascript

I had a similar issue and I fixed it moving from exec to spawn:

var child = process.spawn('<process>', [<arg1>, <arg2>]);

child.stdout.on('data', function (data) {
  console.log('stdout: ' + data);
});

child.stderr.on('data', function (data) {
  console.log('stderr: ' + data);
});

child.on('close', function (code) {
	console.log('child process exited with code ' + code);
});

Solution 3 - Javascript

Adding some explanation to the answers.

The exec command buffers the data before sending it to the parent process. It is generally suitable for the commands producing the smaller output. The above error occurred because the output generated by the execution of the command was larger than the max buffer size. One way to solve the above issue is to specify the buffer size as answered by Tim Cooper.

var execute = function(command, callback){
exec(command, {maxBuffer: 1024 * 500}, function(error, stdout, stderr){ 
 callback(error, stdout); });
};

Another solution is to use the spawn method which is generally faster compared to the exec and it does not buffer the data before sending. It sends the data as a stream hence the problem of buffer size never occurs. The code snippet used by Isampaio.

var child = process.spawn('<process>', [<arg1>, <arg2>]);
child.stdout.on('data', function (data) {
 console.log('stdout: ' + data);
});
child.stderr.on('data', function (data) {
 console.log('stderr: ' + data);
});
child.on('close', function (code) {
 console.log('child process exited with code ' + code);
});

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionYonnaledView Question on Stackoverflow
Solution 1 - JavascriptTim CooperView Answer on Stackoverflow
Solution 2 - JavascriptlsampaioView Answer on Stackoverflow
Solution 3 - JavascriptrbansalView Answer on Stackoverflow