I know the `clear` command that &#39;clears&#39; the current screen, but it does this just by printing lots of newlines - the cleared contents just get scrolled up.

Is there a way to completely wipe all previous output from the terminal so that I can&#39;t reach it even by scrolling up?


How can I clear previous output in Terminal in Mac OS X?

I open several files in Vim by, for example, running

    vim a/*.php

which opens 23 files.

I then make my edit and run the following twice

    :q

which closes all my buffers.

**How can you close only one buffer in Vim?**



How do I close a single buffer (out of many) in Vim?

What are example applications for a &lt;a href=&quot;http://docs.oracle.com/javase/7/docs/api/java/nio/ByteBuffer.html&quot;&gt;`ByteBuffer`&lt;/a&gt; in Java? Please list any example scenarios where this is used.

What is the use of ByteBuffer in Java?

I see the word **&quot;BUFFER&quot;** everywhere, but I am unable to grasp what it exactly is.

 1. Would anybody please explain what is buffer in **layman&#39;s language**?
 2. When is it used?
 3. How is it used?

What does it mean by buffer?

To me it&#39;s not clear what&#39;s the difference between the two Linux memory concepts : `buffer` and `cache`. I&#39;ve read through [this post][1] and it seems to me  that the difference between them is the expiration policy:

1. buffer&#39;s policy is first-in, first-out
2. cache&#39;s policy is Least Recently Used.

Am I right?

In particular, I&#39;m looking at the two commands: `free` and `vmstat`

    james@utopia:~$ vmstat -S M
    procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
    r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
    5  0      0    173     67    912    0    0    19    59   75 1087 24  4 71  1
    james@utopia:~$ free -m
                 total       used       free     shared    buffers     cached
    Mem:          2007       1834        172          0         67        914
    -/+ buffers/cache:        853       1153
    Swap:         2859          0       2859


  [1]: https://stackoverflow.com/questions/3192579/buffer-and-cache-difference

What is the difference between buffer and cache memory in Linux?

How can I convert a NodeJS binary buffer into a JavaScript ArrayBuffer?

Convert a binary NodeJS Buffer to JavaScript ArrayBuffer

I have a method that creates a MessageDigest (a hash) from a file, and I need to do this to a lot of files (&gt;= 100,000). How big should I make the buffer used to read from the files to maximize performance?

Most everyone is familiar with the basic code (which I&#39;ll repeat here just in case):

    MessageDigest md = MessageDigest.getInstance( &quot;SHA&quot; );
    FileInputStream ios = new FileInputStream( &quot;myfile.bmp&quot; );
    byte[] buffer = new byte[4 * 1024]; // what should this value be?
    int read = 0;
    while( ( read = ios.read( buffer ) ) &gt; 0 )
        md.update( buffer, 0, read );
    ios.close();
    md.digest();

What is the ideal size of the buffer to maximize throughput? I know this is system dependent, and I&#39;m pretty sure its OS, FileSystem, *and* HDD dependent, and there maybe other hardware/software in the mix. 

(I should point out that I&#39;m somewhat new to Java, so this may just be some Java API call I don&#39;t know about.)

**Edit:** I do not know ahead of time the kinds of systems this will be used on, so I can&#39;t assume a whole lot. (I&#39;m using Java for that reason.)

**Edit:** The code above is missing things like try..catch to make the post smaller

How do you determine the ideal buffer size when using FileInputStream?

I need to save the whole output of [Screen][1] to a file to check later all the content.

The reason is that I&#39;m dumping a flash memory through a serial port, using Screen to interface with it. I would like to save it to a file to check memory structure.

I&#39;ve tried:

    $: screen /dev/ttyUSB0 115200 &gt;&gt; foo.txt
    $: screen /dev/ttyUSB0 115200 | tee foo.txt

And I&#39;ve also tried to use bufferfile from screen, but I don&#39;t understand how to use it.

Is there an easy way?

  [1]: https://en.wikipedia.org/wiki/GNU_Screen




Save Screen (program) output to a file

Usually, `stdout` is line-buffered. In other words, as long as your `printf` argument ends with a newline, you can expect the line to be printed instantly. This does not appear to hold when using a pipe to redirect to `tee`.

I have a C++ program, `a`, that outputs strings, always `\n`-terminated, to `stdout`.

When it is run by itself (`./a`), everything prints correctly and at the right time, as expected. However, if I pipe it to `tee` (`./a | tee output.txt`), it doesn&#39;t print anything until it quits, which defeats the purpose of using `tee`.

I know that I could fix it by adding a `fflush(stdout)` after each printing operation in the C++ program. But is there a cleaner, easier way? Is there a command I can run, for example, that would force `stdout` to be line-buffered, even when using a pipe?

Force line-buffering of stdout in a pipeline

I have a few questions about the socket library in C.  Here is a snippet of code I&#39;ll refer to in my questions.

    char recv_buffer[3000];
    recv(socket, recv_buffer, 3000, 0);

1. How do I decide how big to make recv_buffer?  I&#39;m using 3000, but it&#39;s arbitrary.  
2. what happens if `recv()` receives a packet bigger than my buffer?  
3. how can I know if I have received the entire message without calling recv again and have it wait forever when there is nothing to be received?
4. is there a way I can make a buffer not have a fixed amount of space, so that I can keep adding to it without fear of running out of space? maybe using `strcat` to concatenate the latest `recv()` response to the buffer?

I know it&#39;s a lot of questions in one, but I would greatly appreciate any responses.

How large should my recv buffer be when calling recv in the socket library

I have a JSON object and I&#39;m converting it to a `Buffer` and doing some process here. Later I want to convert the same buffer data to convert to valid JSON object.

I&#39;m working on Node V6.9.1

Below is the code I tried but I&#39;m getting `[object object]` when I convert back to JSON and cannot open this object.

    var obj = {
       key:&#39;value&#39;,
       key:&#39;value&#39;,
       key:&#39;value&#39;,
       key:&#39;value&#39;,
       key:&#39;value&#39;
    }

    var buf = new Buffer.from(obj.toString());

    console.log(&#39;Real Buffer &#39; + buf);  //This prints --&gt; Real Buffer &lt;Buffer 5b 6f 62 6a 65 63 74&gt;

    var temp = buf.toString();

    console.log(&#39;Buffer to String &#39; + buf);  //This prints --&gt; Buffer to String [object Object]

So I tried to print whole object using inspect way

    console.log(&#39;Full temp &#39; + require(&#39;util&#39;).inspect(buf, { depth: null }));  //This prints --&gt; &#39;[object object]&#39; [not printing the obj like declared above]

If i try to read it like an array

     console.log(buf[0]);  // This prints --&gt; [ 

I tried parsing also it throw `SyntaxError: Unexpected token o in JSON at position 2`


I need to view it as real object like I created (I mean like declared above).

Please help..



Convert a JSON Object to Buffer and Buffer to JSON Object back

I&#39;m not sure what will be in the char array after initialization in the following ways.    

1.`char buf[10] = &quot;&quot;;`  
2. `char buf[10] = &quot; &quot;;`  
3. `char buf[10] = &quot;a&quot;;`  

For case 2, I think `buf[0]` should be `&#39; &#39;`, `buf[1]` should be `&#39;\0&#39;`, and from `buf[2]` to `buf[9]` will be random content. For case 3, I think `buf[0]` should be `&#39;a&#39;`, `buf[1]` should be &#39;\0&#39;, and from `buf[2]` to `buf[9]` will be random content.

Is that correct?

And for the case 1, what will be in the `buf`? `buf[0] == &#39;\0&#39;` and from `buf[1]` to `buf[9]` will be random content?

C char array initialization

How do you clear the string buffer in Java after a loop so the next iteration uses a clear string buffer?

Clearing a string buffer/builder after loop

Assuming I have multiple files opened as buffers in Vim. The files have `*.cpp`, `*.h` and some are `*.xml`. I want to close all the XML files with `:bd *.xml`. However, Vim does not allow this (E93: More than one match...).

Is there any way to do this?

P.S. I know that `:bd file1 file2 file3` works. So can I somehow evaluate `*.xml` to `file1.xml file2.xml file3.xml`?

How to delete multiple buffers in Vim?

I&#39;m trying to write code to read a binary file into a buffer, then write the buffer to another file.  I have the following code, but the buffer only stores a couple of ASCII characters from the first line in the file and nothing else.

    int length;
    char * buffer;
    
    ifstream is;
    is.open (&quot;C:\\Final.gif&quot;, ios::binary );
    // get length of file:
    is.seekg (0, ios::end);
    length = is.tellg();
    is.seekg (0, ios::beg);
    // allocate memory:
    buffer = new char [length];
    // read data as a block:
    is.read (buffer,length);
    is.close();
     
    FILE *pFile;
    pFile = fopen (&quot;C:\\myfile.gif&quot;, &quot;w&quot;);
    fwrite (buffer , 1 , sizeof(buffer) , pFile );

Reading and writing binary file

I have a library that takes as input a `ReadableStream`, but my input is just a base64 format image. I could convert the data I have in a `Buffer` like so:

    var img = new Buffer(img_string, &#39;base64&#39;);

But I have no idea how to convert it to a `ReadableStream` or convert the `Buffer` I obtained to a `ReadableStream`.

Is there a way to do this?

Converting a Buffer into a ReadableStream in Node.js

The command to refresh a file from version on disk is `:e!`

How can I do the same for all files in the buffer? 

Background: I need that because I am using git with multiple branches with one vim open that contains a buffer. When I checkout a branch, I would like to have vim refresh.

Refresh all files in buffer from disk in vim

Scenario: I have opened Vim and pasted some text. I open a second tab with `:tabe` and paste some other text in there.

Goal: I would like a third tab with a output equivalent to writing both texts to files and opening them with `vimdiff`.

The closest I can find is &quot;diff the current buffer against a file&quot;, but not `diff`ing two open but unsaved buffers.

Diff two tabs in Vim

I am learning C++ and I found something that I can&#39;t understand:

&gt;  Output buffers can be explicitly flushed to force the buffer to be
&gt; written. By default, reading `cin` flushes `cout`; `cout` is also flushed
&gt; when the program ends normally.

So flushing the buffer (for example an output buffer): does this clear the buffer by deleting everything in it or does it clear the buffer by outputting everything in it? Or does flushing the buffer mean something completely different?

What does flushing the buffer mean?

I need to do a bunch of word replacements in a file and want to do it with a vi command, not an EX command such as `:%s///g`.

I know that this is the typical way one replaces the word at the current cursor position: `cw&lt;text&gt;&lt;esc&gt;` but is there a way to do this with the contents of the unnamed register as the replacement text and without overwriting the register?


Replace word with contents of paste buffer?

The following code uses SerialPort module to listen to data from a bluetooth connection.
 
I am expecting to see a stream of data in Hexadecimal format printed in console. But the console just shows some weird simbols. I want to know how can I decode and display the data in console. 

  
    var serialPort = new SerialPort(&quot;/dev/tty.EV3-SerialPort&quot;, {
      parser: SP.parsers.raw
    }, false); // this is the openImmediately flag [default is true]

    serialPort.open(function () {
     console.log(&#39;open&#39;);
     serialPort.on(&#39;data&#39;, function(data) {
       var buff = new Buffer(data, &#39;utf8&#39;); //no sure about this
      console.log(&#39;data received: &#39; + buff.toString());
     });  
    });



How to display nodejs raw Buffer data as Hex string

**Checking the [documentation][1] on memoryview:**



&gt; memoryview objects allow Python code to access the internal data of an
&gt; object that supports the buffer protocol without copying.
&gt; 
&gt; class **memoryview**(obj)
&gt; 
&gt; Create a memoryview that references obj. obj must support the
&gt; buffer protocol. Built-in objects that support the buffer protocol
&gt; include bytes and bytearray.


Then we are given the sample code:

    &gt;&gt;&gt; v = memoryview(b&#39;abcefg&#39;)
    &gt;&gt;&gt; v[1]
    98
    &gt;&gt;&gt; v[-1]
    103
    &gt;&gt;&gt; v[1:4]
    &lt;memory at 0x7f3ddc9f4350&gt;
    &gt;&gt;&gt; bytes(v[1:4])
    b&#39;bce&#39;

**Quotation over, now lets take a closer look:**

    &gt;&gt;&gt; b = b&#39;long bytes stream&#39;
    &gt;&gt;&gt; b.startswith(b&#39;long&#39;)
    True
    &gt;&gt;&gt; v = memoryview(b)
    &gt;&gt;&gt; vsub = v[5:]
    &gt;&gt;&gt; vsub.startswith(b&#39;bytes&#39;)
    Traceback (most recent call last):
      File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
    AttributeError: &#39;memoryview&#39; object has no attribute &#39;startswith&#39;
    &gt;&gt;&gt; bytes(vsub).startswith(b&#39;bytes&#39;)
    True
    &gt;&gt;&gt; 


**So what I gather from the above:**

We create a memoryview object to expose the internal data of a buffer object without
copying, however, in order to do anything useful with the object (by calling the methods
provided by the object), we have to create a copy!

Usually memoryview (or the old buffer object) would be needed when we have a large object,
and the slices can be large too. The need for a better efficiency would be present
if we are making large slices, or making small slices but a large number of times.

With the above scheme, I don&#39;t see how it can be useful for either situation, unless
someone can explain to me what I&#39;m missing here.


**Edit1:**

We have a large chunk of data, we want to process it by advancing through it from start to 
end, for example extracting tokens from the start of a string buffer until the buffer is consumed.In C term, this is advancing a pointer through the buffer, and the pointer can be passed
to any function expecting the buffer type. How can something similar be done in python?

People suggest workarounds, for example many string and regex functions take position 
arguments that can be used to emulate advancing a pointer. There&#39;re two issues with this: first 
it&#39;s a work around, you are forced to change your coding style to overcome the shortcomings, and
second: not all functions have position arguments, for example regex functions and `startswith` do, `encode()`/`decode()` don&#39;t. 

Others might suggest to load the data in chunks, or processing the buffer in small
segments larger than the max token.  Okay so we are aware of these possible
workarounds, but we are supposed to work in a more natural way in python without
trying to bend the coding style to fit the language - aren&#39;t we?

**Edit2:**

A code sample would make things clearer. This is what I want to do, and what I assumed memoryview would allow me to do at first glance. Lets use pmview (proper memory view) for the functionality I&#39;m looking for:


    tokens = []
    xlarge_str = get_string()
    xlarge_str_view =  pmview(xlarge_str)
    
    while True:
        token =  get_token(xlarge_str_view)
        if token: 
            xlarge_str_view = xlarge_str_view.vslice(len(token)) 
            # vslice: view slice: default stop paramter at end of buffer
    	    tokens.append(token)
        else:	
            break




  [1]: http://docs.python.org/dev/library/stdtypes.html#memoryview





What exactly is the point of memoryview in Python

I have the following program:

    int main(int argc, char *argv[])
    {
      char ch1, ch2;
      printf(&quot;Input the first character:&quot;); // Line 1
      scanf(&quot;%c&quot;, &amp;ch1); 
      printf(&quot;Input the second character:&quot;); // Line 2
      ch2 = getchar();
      
      printf(&quot;ch1=%c, ASCII code = %d\n&quot;, ch1, ch1);
      printf(&quot;ch2=%c, ASCII code = %d\n&quot;, ch2, ch2);
      
      system(&quot;PAUSE&quot;);	
      return 0;
    }

As the author of the above code have explained:
The program will not work properly because at Line 1, when the user presses Enter, it will leave in the input buffer 2 character: `Enter key (ASCII code 13)` and `\n (ASCII code 10)`. Therefore, at Line 2, it will read the `\n` and will not wait for the user to enter a character.

OK, I got this. But my first question is: Why the second `getchar()` (`ch2 = getchar();`) does not read the `Enter key (13)`, rather than `\n` character?

Next, the author proposed 2 ways to solve such probrems:

1. use `fflush()`

2. write a function like this:

        void
        clear (void)
        {    
          while ( getchar() != &#39;\n&#39; );
        }

This code worked actually. But I cannot explain myself how it works? Because in the while statement, we use `getchar() != &#39;\n&#39;`, that means read any single character except `&#39;\n&#39;`? if so, in the input buffer still remains the `&#39;\n&#39;` character?

How to clear input buffer in C?

I just learned about input/output using `BufferedReader`.

I wanted to know what exactly are the meanings of the term `Stream` and `Buffer`?

Also what does this line of code serves us:

    BufferedReader br=new BufferedReader(new InputStreamReader(System.in));


What exactly does &quot;Stream&quot; and &quot;Buffer&quot; mean in Java I/O?

I have a buffer with some binary data:

    var b = new Buffer ([0x00, 0x01, 0x02]);

and I want to append `0x03`.

How can I append more binary data? I&#39;m searching in the documentation but for appending data it must be a string, if not, an error occurs (*TypeError: Argument must be a string*):

    var b = new Buffer (256);
    b.write (&quot;hola&quot;);
    console.log (b.toString (&quot;utf8&quot;, 0, 4)); //hola
    b.write (&quot;, adios&quot;, 4);
    console.log (b.toString (&quot;utf8&quot;, 0, 11)); //hola, adios

Then, the only solution I can see here is to create a new buffer for every appended binary data and copy it to the major buffer with the correct offset:

    var b = new Buffer (4); //4 for having a nice printed buffer, but the size will be 16KB
    new Buffer ([0x00, 0x01, 0x02]).copy (b);
    console.log (b); //&lt;Buffer 00 01 02 00&gt;
    new Buffer ([0x03]).copy (b, 3);
    console.log (b); //&lt;Buffer 00 01 02 03&gt;

But this seems a bit inefficient because I have to instantiate a new buffer for every append.

Do you know a better way for appending binary data?


  [1]: https://github.com/Gagle/Node-BufferedWriter
  [2]: https://github.com/Gagle/Node-BufferedReader


**EDIT**

I&#39;ve written a [BufferedWriter][1] that writes bytes to a file using internal buffers. Same as [BufferedReader][2] but for writing.

A quick example:

    //The BufferedWriter truncates the file because append == false
    new BufferedWriter (&quot;file&quot;)
    	.on (&quot;error&quot;, function (error){
    		console.log (error);
    	})
    	
    	//From the beginning of the file:
    	.write ([0x00, 0x01, 0x02], 0, 3) //Writes 0x00, 0x01, 0x02
    	.write (new Buffer ([0x03, 0x04]), 1, 1) //Writes 0x04
    	.write (0x05) //Writes 0x05
    	.close (); //Closes the writer. A flush is implicitly done.
    
    //The BufferedWriter appends content to the end of the file because append == true
    new BufferedWriter (&quot;file&quot;, true)
    	.on (&quot;error&quot;, function (error){
    		console.log (error);
    	})
    	
    	//From the end of the file:
    	.write (0xFF) //Writes 0xFF
    	.close (); //Closes the writer. A flush is implicitly done.
    
    //The file contains: 0x00, 0x01, 0x02, 0x04, 0x05, 0xFF

**LAST UPDATE**

Use [concat](http://nodejs.org/api/buffer.html#buffer_class_method_buffer_concat_list_totallength).

How to append binary data to a buffer in node.js

I tried to copy text from IDEA with ideavim plugin, using default vim keybindings (**y**). But this text isn&#39;t copied in global buffer and i can paste it only in IDEA. &lt;br&gt;
How can I use copied piece of text in browser, for example?

Intellij IDEA with ideavim. Cannot copy text from another source

I&#39;ve tried [MiniBufExplorer](http://www.vim.org/scripts/script.php?script_id=159), but I usually end up with several windows showing it or close it altogether. What I&#39;d like is something like [LustyJuggler](http://www.vim.org/scripts/script.php?script_id=2050) with incremental search, the way I switch between buffers in Emacs. Surely there is a script like this?

How do you prefer to switch between buffers in Vim?

For what purpose I should use [`std::get_temporary_buffer`][1]? Standard says the following:

&gt; Obtains a pointer to storage sufficient to store up to n adjacent T objects.

I thought that the buffer will be allocated on the stack, but that is not true. According to the C++ Standard this buffer is actually not temporary. What advantages does this function have over the global function `::operator new`, which doesn&#39;t construct the objects either. Am I right that the following statements are equivalent?

    int* x;
    x = std::get_temporary_buffer&lt;int&gt;( 10 ).first;
    x = static_cast&lt;int*&gt;( ::operator new( 10*sizeof(int) ) );

Does this function only exist for syntax sugar? Why is there `temporary` in its name?

---

One use case was suggested in the [Dr. Dobb&#39;s Journal, July 01, 1996][2] for implementing algorithms:

&gt; If no buffer can be allocated, or if it is smaller than requested, the algorithm still works correctly, It merely slows down. 


  [1]: http://en.cppreference.com/w/cpp/memory/get_temporary_buffer
  [2]: http://www.drdobbs.com/184403203;jsessionid=JDREHEWS1OHWRQE1GHOSKHWATMY32JVN?pgno=7

Why do I need std::get_temporary_buffer?

I have a streaming time series, of which I am interested in keeping the last 4 elements, which means I want to be able to pop the first, and add to the end. Essentially what I need is a [**ring buffer**][1].

Which Java Collection is the best for this? Vector?


  [1]: https://en.wikipedia.org/wiki/Circular_buffer

Ring Buffer in Java

What is the difference between a byte array &amp; byte buffer ?   
  **Also, in what situations should one be preferred over the other?**

[my usecase is for a web application being developed in java].

When to use byte array &amp; when byte buffer?

&gt; **Possible Duplicate:**  
&gt; [Convert integer into byte array (Java)](https://stackoverflow.com/questions/1936857/convert-integer-into-byte-array-java)  

&lt;!-- End of automatically inserted text --&gt;

I need to store the length of a buffer, in a byte array 4 bytes large.

Pseudo code:

    private byte[] convertLengthToByte(byte[] myBuffer)
    {
        int length = myBuffer.length;
        
        byte[] byteLength = new byte[4];
    
        //here is where I need to convert the int length to a byte array
        byteLength = length.toByteArray;
    
        return byteLength;
    }

What would be the best way of accomplishing this? Keeping in mind I must convert that byte array back to an integer later.

Java - Convert int to Byte Array of 4 Bytes?

The [Cython documentation on typed memory views](http://docs.cython.org/src/userguide/memoryviews.html) list three ways of assigning to a typed memory view:

1. from a raw C pointer,
2. from a `np.ndarray` and
3. from a `cython.view.array`.

Assume that I don&#39;t have data passed in to my cython function from outside but instead want to allocate memory and return it as a `np.ndarray`, which of those options do I chose? Also assume that the size of that buffer is not a compile-time constant i.e. I can&#39;t allocate on the stack, but would need to `malloc` for option 1.

The 3 options would therefore looke something like this:

    from libc.stdlib cimport malloc, free
    cimport numpy as np
    from cython cimport view

    np.import_array()

    def memview_malloc(int N):
        cdef int * m = &lt;int *&gt;malloc(N * sizeof(int))
        cdef int[::1] b = &lt;int[:N]&gt;m
        free(&lt;void *&gt;m)

    def memview_ndarray(int N):
        cdef int[::1] b = np.empty(N, dtype=np.int32)
        
    def memview_cyarray(int N):
        cdef int[::1] b = view.array(shape=(N,), itemsize=sizeof(int), format=&quot;i&quot;)

What is surprising to me is that in all three cases, [Cython generates quite a lot of code](http://htmlpreview.github.com/?https://gist.github.com/kynan/910a2e4ce3332c9bd8c2/raw/a4131b4417d4b93bff0064e7a14b683e27564631/memview_bench.html) for the memory allocation, in particular a call to `__Pyx_PyObject_to_MemoryviewSlice_dc_int`. This suggests (and I might be wrong here, my insight into the inner workings of Cython are very limited) that it first creates a Python object and then &quot;casts&quot; it into a memory view, which seems unnecessary overhead.

A [simple benchmark](http://nbviewer.ipython.org/urls/gist.github.com/kynan/910a2e4ce3332c9bd8c2/raw/c4c935d34eb77e1ec0fb06bfa40ebce54a1c1812/Benchmark+Cython+typed+memory+view+allocation.ipynb) doesn&#39;t reveal much difference between the three methods, with 2. being the fastest by a thin margin.

Which of the three methods is recommended? Or is there a different, better option?

**Follow-up question:** I want to finally return the result as a `np.ndarray`, after having worked with that memory view in the function. Is a typed memory view the best choice or would I rather just use the old buffer interface as below to create an `ndarray` in the first place?

    cdef np.ndarray[DTYPE_t, ndim=1] b = np.empty(N, dtype=np.int32)

What is the recommended way of allocating memory for a typed memory view?

Suppose I&#39;m writing a couple of files to disk, between 2MB and 5GB.
What are sensible buffer values for the FileStream ?

Is it sensible to work with buffersizes of several megabytes, or should I stick to kilobyte-buffers ?

C# FileStream : Optimal buffer size for writing large files?

What&#39;s the default socket buffer size of linux? Is there any command to see it?

How to find the socket buffer size of linux

Should `fflush()` not be used to flush a buffer even if it is an output stream?

What is it useful for? How do we flush a buffer in general?

Flushing buffers in C

I have tested two writing configurations:

1. Fstream buffering:

    ```
    // Initialization
    const unsigned int length = 8192;
    char buffer[length];
    std::ofstream stream;
    stream.rdbuf()-&gt;pubsetbuf(buffer, length);
    stream.open(&quot;test.dat&quot;, std::ios::binary | std::ios::trunc)

    // To write I use :
    stream.write(reinterpret_cast&lt;char*&gt;(&amp;x), sizeof(x));
    ```

2. Manual buffering:

    ```
    // Initialization
    const unsigned int length = 8192;
    char buffer[length];
    std::ofstream stream(&quot;test.dat&quot;, std::ios::binary | std::ios::trunc);

    // Then I put manually the data in the buffer

    // To write I use :
    stream.write(buffer, length);
    ```

I expected the same result...

But my manual buffering improve performance by a factor of 10 to write a file of 100MB, and the fstream buffering does not change anything compared to the normal situation (without redefining a buffer).

Does someone has an explanation of this situation ?

EDIT :
Here are the news : a benchmark just done on a supercomputer (linux 64-bit architecture, lasts intel Xeon 8-core, Lustre filesystem and ... hopefully well configured compilers) 
![benchmark][1]
(and I don&#39;t explain the reason of the &quot;resonance&quot; for a 1kB manual buffer...)

EDIT 2 :
And the resonance at 1024 B (if someone has an idea about that, I&#39;m interested) :
![enter image description here][2]


  [1]: http://i.stack.imgur.com/NO8Pm.png
  [2]: http://i.stack.imgur.com/pknmv.png

std::fstream buffering vs manual buffering (why 10x gain with manual buffering)?

I am constructing an array of bytes in java and I don&#39;t know how long the array will be.  

I want some tool like Java&#39;s StringBuffer that you can just call .append(byte b) or .append(byte[] buf) and have it buffer all my bytes and return to me a byte[] when I&#39;m done.  Is there a class that does for bytes what StringBuffer does for Strings?  It does not look like the ByteBuffer class is what I&#39;m looking for.

Anyone have a good solution?



Byte array of unknown length in java

In Python you can use [StringIO][1] for a file-like buffer for character data. [Memory-mapped file][2] basically does similar thing for binary data, but it requires a file that is used as the basis. Does Python have a file object that is intended for binary data and is memory only, equivalent to Java&#39;s [ByteArrayOutputStream][3]?

The use-case I have is I want to create a ZIP file in memory, and [ZipFile][4] requires a file-like object.

  [1]: https://docs.python.org/library/struct.html
  [2]: https://docs.python.org/library/mmap.html
  [3]: http://java.sun.com/javase/6/docs/api/java/io/ByteArrayOutputStream.html
  [4]: https://docs.python.org/library/zipfile.html

Binary buffer in Python

I want to send a raw buffer using bluetooth connection. The content is a hex number. Currently I split the number manually to an byte array. Is there any function that can help me convert the number to byte array?

    //var data = 0x250001000192CD0000002F6D6E742F72;
    var data = new Buffer([0x25,0x00,0x01,0x00,0x01,0x92,0xCD,0x00,0x00,0x00,0x2F,0x6D,0x6E,0x74,0x2F,0x72]);
    serialPort.write(data);


Node.js convert hexadecimal number to byteArray

Should a buffer of bytes be signed char or unsigned char or simply a char buffer?
Any differences between C and C++?

Thanks.

Should a buffer of bytes be signed or unsigned char buffer?

The [MSDN site](http://msdn.microsoft.com/en-us/library/system.io.bufferedstream.aspx) states:

&gt; A buffer is a block of bytes in memory
&gt; used to cache data, thereby reducing
&gt; the number of calls to the operating
&gt; system. Buffers improve read and write
&gt; performance. A buffer can be used for
&gt; either reading or writing, but never
&gt; both simultaneously. The Read and
&gt; Write methods of BufferedStream
&gt; automatically maintain the buffer.

Should I use this class in every possible occasion?

When to use .NET BufferedStream class?

I&#39;m using log4net with AdoNetAppender. It&#39;s seems that the AdoNetAppender has a [Flush method][1]. Is there anyway I can call that from my code? 

I&#39;m trying to create an admin page to view all the entries in the database log, and I will like to setup log4net with bufferSize=100 (or more), then I want the administrator to be able to click an button on the admin page to force log4net to write the buffered log entries to the database (without shutting down log4net).

Is that possible?

  [1]: http://logging.apache.org/log4net/release/sdk/log4net.Appender.AdoNetAppenderMembers.html

Is there a way to programmably flush the buffer in log4net

I am writing a small I/O library to assist with a larger (hobby) project. A part of this library performs various functions on a file, which is read / written via the `FileStream` object. On each `StreamReader.Read(...)` pass, 

I fire off an event which will be used in the main app to display progress information. The processing that goes on in the loop is vaired, but is not too time consuming (it could just be a simple file copy, for example, or may involve encryption...).

My main question is: What is the best memory buffer size to use? Thinking about physical disk layouts, I could pick 2k, which would cover a CD sector size and is a nice multiple of a 512 bytes hard disk sector. Higher up the abstraction tree, you could go for a larger buffer which could read an entire FAT cluster at a time. I realise with today&#39;s PC&#39;s, I could go for a more memory hungry option (a couple of MiB, for example), but then I increase the time between UI updates and the user perceives a less responsive application.

As an aside, I&#39;m eventually hoping to provide a similar interface to files hosted on FTP / HTTP servers (over a local network / fastish DSL). What would be the best memory buffer size for those (again, a &quot;best-case&quot; tradeoff between perceived responsiveness vs. performance)?



File I/O with streams - best memory buffer size

I recently read a nice [post][1] on using `StringIO` in Ruby.  What the author doesn&#39;t mention, though, is that `StringIO` is just an &quot;I.&quot;  There&#39;s no &quot;O.&quot;  You _can&#39;t_ do this, for example:

    s = StringIO.new
    s &lt;&lt; &#39;foo&#39;
    s &lt;&lt; &#39;bar&#39;
    s.to_s
    # =&gt; should be &quot;foo\nbar&quot;
    # =&gt; really is &#39;&#39;`

Ruby really needs a StringBuffer just like the one Java has. StringBuffers serve two important purposes. First, they let you test the output half of what Ruby&#39;s StringIO does. Second, they are useful for building up long strings from small parts -- something that Joel reminds us over and over again is otherwise very very slow.

Is there a good replacement?

It&#39;s true that Strings in Ruby are mutable, but that doesn&#39;t mean we should always rely on that functionality.  If `stuff` is large, the performance and memory requirements of this, for example, is really bad.

    result = stuff.map(&amp;:to_s).join(&#39; &#39;)

The &quot;correct&quot; way to do this in Java is:

    result = StringBuffer.new(&quot;&quot;)
    for(String s : stuff) {
      result.append(s);
    }

Though my Java is a bit rusty.

  [1]: http://macdevelopertips.com/ruby/using-a-string-as-a-file-in-ruby.html

Why doesn&#39;t Ruby have a real StringBuffer or StringIO?

I read in multiple places that the default buffer size for a pipe is 4kB (for instance, [here][1]), and my `ulimit -a` tends to confirm that statement:

   

    $ ulimit -a
    core file size          (blocks, -c) 0
    data seg size           (kbytes, -d) unlimited
    scheduling priority             (-e) 0
    file size               (blocks, -f) unlimited
    pending signals                 (-i) 15923
    max locked memory       (kbytes, -l) 64
    max memory size         (kbytes, -m) unlimited
    open files                      (-n) 1024
    pipe size            (512 bytes, -p) 8 // 8 * 512B = 4kB
    POSIX message queues     (bytes, -q) 819200
    real-time priority              (-r) 0
    stack size              (kbytes, -s) 10240
    cpu time               (seconds, -t) unlimited
    max user processes              (-u) 1024
    virtual memory          (kbytes, -v) unlimited
    file locks                      (-x) unlimited

But when I use a little program to test the buffer size (by writing into the pipe until the write() blocks), I see a limit of 64kB!

See this program:

    #include &lt;stdio.h&gt;
    #include &lt;unistd.h&gt;
    #include &lt;limits.h&gt;
    
    int main(void)
    {
    	int tube[2];
    	char c = &#39;c&#39;;
    	int i;
    
    	fprintf(stdout, &quot;Tube Creation\n&quot;);
    	fprintf(stdout, &quot;Theoretical max size: %d\n&quot;, PIPE_BUF);
    	if( pipe(tube) != 0)
    	{
    		perror(&quot;pipe&quot;);
    		_exit(1);
    	}
    	fprintf(stdout, &quot;Writing in pipe\n&quot;);
    	for(i=0;; i++)
    	{
    		fprintf(stdout, &quot;%d bytes written\n&quot;, i+1);
    		if( write(tube[1], &amp;c, 1) != 1)
    		{
    			perror(&quot;Write&quot;);
    			_exit(1);
    		}
    	}
    	return 0;
    }

And its output:

    $ ./test_buf_pipe 
    Tube Creation
    Theoretical max size: 4096
    Writing in pipe
    1 bytes written
    2 bytes written
    3 bytes written
    4 bytes written
    [...]
    65535 bytes written
    [blocks here]

It strongly suggests that the pipe buffer size is actually 64k!
What is happening here??


  [1]: http://linux.die.net/man/7/pipe

Pipe buffer size is 4k or 64k?

How can I redirect or pipe the output of an ex command into my current buffer or a file?

For example, I want to read the contents of all the registers into the current buffer, which in ex mode is shown using `:registers`.

How to redirect ex command output into current buffer or file?

I have a relatively small file (some hundreds of kilobytes) that I want to be in memory for direct access for the entire execution of the code.

I don&#39;t know exactly the internals of Node.js, so I&#39;m asking if a `fs open` is enough or I have to read all file and copy to a `Buffer`? 



All Buffer Solutions on Gang of Coders

Total of 47 Buffer Solutions