Get the whole response body when the response is chunked?

Httpnode.js

Http Problem Overview


I'm making a HTTP request and listen for "data":

response.on("data", function (data) { ... })

The problem is that the response is chunked so the "data" is just a piece of the body sent back.

How do I get the whole body sent back?

Http Solutions


Solution 1 - Http

request.on('response', function (response) {
  var body = '';
  response.on('data', function (chunk) {
    body += chunk;
  });
  response.on('end', function () {
    console.log('BODY: ' + body);
  });
});
request.end();

Solution 2 - Http

Over at https://groups.google.com/forum/?fromgroups=#!topic/nodejs/75gfvfg6xuc, Tane Piper provides a good solution very similar to scriptfromscratch's, but for the case of a JSON response:

  request.on('response',function(response){
     var data = [];
     response.on('data', function(chunk) {
       data.push(chunk);
     });
     response.on('end', function() {
       var result = JSON.parse(data.join(''))
       return result
     });
   });`

This addresses the issue that OP brought up in the comments section of scriptfromscratch's answer.

Solution 3 - Http

I never worked with the HTTP-Client library, but since it works just like the server API, try something like this:

var data = '';
response.on('data', function(chunk) {
  // append chunk to your data
  data += chunk;
});

response.on('end', function() {
  // work with your data var
});

See node.js docs for reference.

Solution 4 - Http

In order to support the full spectrum of possible HTTP applications, Node.js's HTTP API is very low-level. So data is received chunk by chunk not as whole.
There are two approaches you can take to this problem:

  1. Collect data across multiple "data" events and append the results
    together prior to printing the output. Use the "end" event to determine
    when the stream is finished and you can write the output.
var http = require('http') ;
http.get('some/url' , function (resp) {
    var respContent = '' ;
    resp.on('data' , function (data) {
        respContent += data.toString() ;//data is a buffer instance
    }) ;
    resp.on('end' ,  function() {
        console.log(respContent) ;
    }) ;
}).on('error' , console.error) ;

2) Use a third-party package to abstract the difficulties involved in
collecting an entire stream of data. Two different packages provide a
useful API for solving this problem (there are likely more!): bl (Buffer
List) and concat-stream; take your pick!

var http = require('http') ;
var bl = require('bl') ;

http.get('some/url', function (response) {
    response.pipe(bl(function (err, data) {
        if (err) {
            return console.error(err)
        }
        data = data.toString() ;
        console.log(data) ;
    })) 
}).on('error' , console.error) ;

Solution 5 - Http

The reason it's messed up is because you need to call JSON.parse(data.toString()). Data is a buffer so you can't just parse it directly.

Solution 6 - Http

If you don't mind using the request library

var request = require('request');
request('http://www.google.com', function (error, response, body) {
  if (!error && response.statusCode == 200) {
    console.log(body) // Print the google web page.
  }
})

Solution 7 - Http

How about HTTPS chunked response? I've been trying to read a response from an API that response over HTTPS with a header Transfer-Encoding: chunked. Each chunk is a Buffer but when I concat them all together and try converting to string with UTF-8 I get weird characters.

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionajsieView Question on Stackoverflow
Solution 1 - HttpPero P.View Answer on Stackoverflow
Solution 2 - HttpJim DanzView Answer on Stackoverflow
Solution 3 - HttpschaermuView Answer on Stackoverflow
Solution 4 - HttpSalarView Answer on Stackoverflow
Solution 5 - HttpMartinView Answer on Stackoverflow
Solution 6 - HttptimView Answer on Stackoverflow
Solution 7 - HttpKamil OczkowskiView Answer on Stackoverflow