Pipe buffer size is 4k or 64k?
CLinuxSizeBufferPipeC Problem Overview
I read in multiple places that the default buffer size for a pipe is 4kB (for instance, here), and my ulimit -a tends to confirm that statement:
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15923
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8 // 8 * 512B = 4kB
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
But when I use a little program to test the buffer size (by writing into the pipe until the write() blocks), I see a limit of 64kB!
See this program:
#include <stdio.h>
#include <unistd.h>
#include <limits.h>
int main(void)
{
int tube[2];
char c = 'c';
int i;
fprintf(stdout, "Tube Creation\n");
fprintf(stdout, "Theoretical max size: %d\n", PIPE_BUF);
if( pipe(tube) != 0)
{
perror("pipe");
_exit(1);
}
fprintf(stdout, "Writing in pipe\n");
for(i=0;; i++)
{
fprintf(stdout, "%d bytes written\n", i+1);
if( write(tube[1], &c, 1) != 1)
{
perror("Write");
_exit(1);
}
}
return 0;
}
And its output:
$ ./test_buf_pipe
Tube Creation
Theoretical max size: 4096
Writing in pipe
1 bytes written
2 bytes written
3 bytes written
4 bytes written
[...]
65535 bytes written
[blocks here]
It strongly suggests that the pipe buffer size is actually 64k! What is happening here??
C Solutions
Solution 1 - C
The other answers tell you that the pipe size is 64 KB. The reason why PIPE_BUF is 4KB is that PIPE_BUF is the largest size for which writes are guaranteed to be atomic. See http://pubs.opengroup.org/onlinepubs/9699919799/functions/write.html
Solution 2 - C
It's programmable now
As of Linux 2.6.35 you can use fcntl(2) with the F_SETPIPE_SZ operation to set the pipe buffer up to /proc/sys/fs/pipe-max-size. This is by default 1 MB; see proc(5).
Solution 3 - C
In my experience, the single write test produced a total size of 65536, yet when I wrote 2700 at a time, I could only write 16 times, and then the next attempt stalls. I figure that the 'atomic' write needs to be within one 4K block, and that for each of my writes, it goes to the next full block to satisfy the request. So, in effect, the useable pipe size depends on the size of your writes.
Solution 4 - C
Looks like the kernel use up to 16 buffers which adds up to 64k. See this link for an explanation of the ulimit output vs actual buffer size
Solution 5 - C
That's right. Since the 2.6.11 kernel, the pipe size in Linux is 64kB. Why ulimit reports it as 4Kb, I'm not sure, but that's wrong.