I need to do some parsing of large (5-10 Gb)logfiles in Javascript/Node.js (I&#39;m using Cube).

The logline looks something like:

    10:00:43.343423 I&#39;m a friendly log message. There are 5 cats, and 7 dogs. We are in state &quot;SUCCESS&quot;.

We need to read each line, do some parsing (e.g. strip out `5`, `7` and `SUCCESS`), then pump this data into Cube (https://github.com/square/cube) using their JS client.

Firstly, what is the canonical way in Node to read in a file, line by line?

It seems to be fairly common question online:

* http://www.quora.com/What-is-the-best-way-to-read-a-file-line-by-line-in-node-js
* https://stackoverflow.com/questions/6156501/read-a-file-one-line-at-a-time-in-node-js

A lot of the answers seem to point to a bunch of third-party modules:

* https://github.com/nickewing/line-reader
* https://github.com/jahewson/node-byline
* https://github.com/pkrumins/node-lazy
* https://github.com/Gagle/Node-BufferedReader

However, this seems like a fairly basic task - surely, there&#39;s a simple way within the stdlib to read in a textfile, line-by-line?

Secondly, I then need to process each line (e.g. convert the timestamp into a Date object, and extract useful fields).

What&#39;s the best way to do this, maximising throughput? Is there some way that won&#39;t block on either reading in each line, or on sending it to Cube?

Thirdly - I&#39;m guessing using string splits, and the JS equivalent of contains (IndexOf != -1?) will be a lot faster than regexes? Has anybody had much experience in parsing massive amounts of text data in Node.js?

Cheers,
Victor

All Logfile Analysis Solutions on Gang of Coders

Total of 1 Logfile Analysis Solutions

Parsing huge logfiles in Node.js - read in line-by-line