Try `stdbuf`, included in GNU coreutils and thus virtually any Linux distro.  This sets the buffer length for input, output and error to zero:

    stdbuf -i0 -o0 -e0 command



The command `unbuffer` from the `expect` package disables the output buffering:&lt;br /&gt;
[Ubuntu Manpage: unbuffer - unbuffer output](http://manpages.ubuntu.com/manpages/lucid/man1/expect_unbuffer.1.html)

Example usage:
&lt;pre&gt;&lt;code&gt;unbuffer hexdump file | ./my_script&lt;/code&gt;&lt;/pre&gt;

AFAIK, you can&#39;t do it without ugly hacks. Writing to a pipe (or reading from it) automatically turns on full buffering and there is nothing you can do about it :-(. &quot;Line buffering&quot; (which is what you want) is only used when reading/writing a terminal. The ugly hacks exactly do this: They connect a program to a pseudo-terminal, so that the other tools in the pipe read/write from that terminal in line buffering mode. The whole problem is described here: 

* http://www.pixelbeat.org/programming/stdio_buffering/

The page has also some suggestions (the aforementioned &quot;ugly hacks&quot;) what to do, i.e. using `unbuffer` or pulling some tricks with `LD_PRELOAD`.


You could also use the &lt;code&gt;script&lt;/code&gt; command to make the output of &lt;code&gt;hexdump&lt;/code&gt; line-buffered (&lt;code&gt;hexdump&lt;/code&gt; will be run in a pseudo terminal which tricks &lt;code&gt;hexdump&lt;/code&gt; into thinking its writing its stdout to a terminal, and not to a pipe).

    # cf. http://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe/
    stty -echo -onlcr
    script -q /dev/null hexdump file | ./my_script         # FreeBSD, Mac OS X
    script -q -c &quot;hexdump file&quot; /dev/null | ./my_script    # Linux
    stty echo onlcr

One should use grep or egrep &quot;--line-buffered&quot; options to solve this. no other tools needed.

I&#39;d like to know the maximum value of size_t on the system my program is running.
My first instinct was to use negative 1, like so:

    size_t max_size = (size_t)-1;

But I&#39;m guessing there&#39;s a better way, or a constant defined somewhere.


What is a portable method to find the maximum value of size_t?

In vim, how can I map &quot;save&quot; (`:w`) to &lt;kbd&gt;ctrl&lt;/kbd&gt;-&lt;kbd&gt;s&lt;/kbd&gt;.

I am trying &quot;map&quot; the command, but xterm freezes when I press &lt;kbd&gt;ctrl&lt;/kbd&gt;-&lt;kbd&gt;s&lt;/kbd&gt;. 

If I give &lt;kbd&gt;ctrl&lt;/kbd&gt;-&lt;kbd&gt;v&lt;/kbd&gt;,&lt;kbd&gt;ctrl&lt;/kbd&gt;-&lt;kbd&gt;s&lt;/kbd&gt; still I see only a `^`, not `^S`.

In vim how to map &quot;save&quot; to ctrl-s

Is there a way to run shell commands without output buffering?

For example, `hexdump file | ./my_script` will only pass input from hexdump to my_script in buffered chunks, not line by line.

Actually I want to know a general solution how to make any command unbuffered?

How to make output of any shell command unbuffered?

Is there a way to run shell commands without output buffering?
For example, <code>hexdump file | ./my_script</code> will only pass input from hexdump to my_script in buffered chunks, not line by line.
Actually I want to know a general solution how to make any command unbuffered?

How can I make python 3 (3.1) `print(&quot;Some text&quot;)` to stdout in UTF-8, or how to output raw bytes?

## Test.py

&lt;!-- language: lang-python --&gt;

    TestText = &quot;Test - āĀēĒčČ..šŠūŪžŽ&quot; # this is UTF-8
    TestText2 = b&quot;Test2 - \xc4\x81\xc4\x80\xc4\x93\xc4\x92\xc4\x8d\xc4\x8c..\xc5\xa1\xc5\xa0\xc5\xab\xc5\xaa\xc5\xbe\xc5\xbd&quot; # just bytes
    print(sys.getdefaultencoding())
    print(sys.stdout.encoding)
    print(TestText)
    print(TestText.encode(&quot;utf8&quot;))
    print(TestText.encode(&quot;cp1252&quot;,&quot;replace&quot;))
    print(TestText2)

Output (in CP1257 and I replaced chars to byte values `[x00]`):

&lt;!-- language: none --&gt;

    utf-8
    cp1257
    Test - [xE2][xC2][xE7][C7][xE8][xC8]..[xF0][xD0][xFB][xDB][xFE][xDE]  
    b&#39;Test - \xc4\x81\xc4\x80\xc4\x93\xc4\x92\xc4\x8d\xc4\x8c..\xc5\xa1\xc5\xa0\xc5\xab\xc5\xaa\xc5\xbe\xc5\xbd&#39;
    b&#39;Test - ??????..\x9a\x8a??\x9e\x8e&#39;
    b&#39;Test2 - \xc4\x81\xc4\x80\xc4\x93\xc4\x92\xc4\x8d\xc4\x8c..\xc5\xa1\xc5\xa0\xc5\xab\xc5\xaa\xc5\xbe\xc5\xbd&#39;

`print` is just too smart... :D There&#39;s no point using encoded text with `print` (since it always show only representation of bytes not real bytes) and it&#39;s impossible to output bytes at all, because print anyway and always encodes it in `sys.stdout.encoding`.

For example: `print(chr(255))` throws an error:

&lt;!-- language: none --&gt;

&gt;     Traceback (most recent call last):
&gt;       File &quot;Test.py&quot;, line 1, in &lt;module&gt;
&gt;         print(chr(255));
&gt;       File &quot;H:\Python31\lib\encodings\cp1257.py&quot;, line 19, in encode
&gt;         return codecs.charmap_encode(input,self.errors,encoding_table)[0]
&gt;     UnicodeEncodeError: &#39;charmap&#39; codec can&#39;t encode character &#39;\xff&#39; in position 0: character maps to &lt;undefined&gt;

By the way `print( TestText == TestText2.decode(&quot;utf8&quot;))` returns `False`, although print output is the same.

---
How does Python 3 determine `sys.stdout.encoding` and how can I change it?

I made a `printRAW()` function which works fine (actually it encodes output to UTF-8, so really it&#39;s not raw...):

&lt;!-- language: lang-python --&gt;

     def printRAW(*Text):
         RAWOut = open(1, &#39;w&#39;, encoding=&#39;utf8&#39;, closefd=False)
         print(*Text, file=RAWOut)
         RAWOut.flush()
         RAWOut.close()

     printRAW(&quot;Cool&quot;, TestText)

Output (now it print in UTF-8):

&lt;!-- language: none --&gt;

&gt;     Cool Test - āĀēĒčČ..šŠūŪžŽ

`printRAW(chr(252))` also nicely prints `&#252;` (in UTF-8, `[xC3][xBC]`) and without errors :)

Now I&#39;m looking for maybe better solution if there&#39;s any...

How to make python 3 print() utf8

In Python, how do you check if `sys.stdin` has data or not?

I found that `os.isatty(0)` can not only check if stdin is connected to a TTY device, but also if there is data available.

But if someone uses code such as

    sys.stdin = cStringIO.StringIO(&quot;ddd&quot;)

and after that uses `os.isatty(0)`, it still returns True.  What do I need to do to check if stdin has data?

How do I check if stdin has some data?

Is it possible to capture the stdout and stderr when using the [googletest][1] framework?

For example, I would like to call a function that writes errors to the console (stderr).
Now, when calling the function in the tests, I want to assert that no output appears there.

Or, maybe I want to test the error behaviour and want to assert that a certain string gets printed when I (deliberately) produce an error.


  [1]: http://code.google.com/p/googletest/

How to capture stdout/stderr with googletest?

I&#39;ve always been amazed/frustrated with how long it takes to simply output to the terminal with a print statement.  After some recent painfully slow logging I decided to look into it and was quite surprised to find that almost **all** the time spent is waiting for the terminal to process the results.

Can writing to stdout be sped up somehow?

I wrote a script (&#39;`print_timer.py`&#39; at the bottom of this question) to compare timing when writing 100k lines to stdout, to file, and with stdout redirected to `/dev/null`.  Here is the timing result:

    $ python print_timer.py
    this is a test
    this is a test
    &lt;snipped 99997 lines&gt;
    this is a test
    -----
    timing summary (100k lines each)
    -----
    print                         :11.950 s
    write to file (+ fsync)       : 0.122 s
    print with stdout = /dev/null : 0.050 s

Wow.  To make sure python isn&#39;t doing something behind the scenes like recognizing that I reassigned stdout to /dev/null or something, I did the redirection outside the script...

    $ python print_timer.py &gt; /dev/null
    -----
    timing summary (100k lines each)
    -----
    print                         : 0.053 s
    write to file (+fsync)        : 0.108 s
    print with stdout = /dev/null : 0.045 s

So it isn&#39;t a python trick, it is just the terminal.  I always knew dumping output to /dev/null sped things up, but never figured it was that significant!

It amazes me how slow the tty is.  How can it be that writing to physical disk is WAY faster than writing to the &quot;screen&quot; (presumably an all-RAM op), and is effectively as fast as simply dumping to the garbage with /dev/null?

[This link][1] talks about how the terminal will block I/O so it can _&quot;parse [the input], update its frame buffer, communicate with the X server in order to scroll the window and so on&quot;_... but I don&#39;t fully get it.  What can be taking so long?

I expect there is no way out (short of a faster tty implementation?) but figure I&#39;d ask anyway.


----------
UPDATE: after reading some comments I wondered how much impact my screen size actually has on the print time, and it does have some significance.  The really slow numbers above are with my Gnome terminal blown up to 1920x1200.  If I reduce it very small I get...

    -----
    timing summary (100k lines each)
    -----
    print                         : 2.920 s
    write to file (+fsync)        : 0.121 s
    print with stdout = /dev/null : 0.048 s

That is certainly better (~4x), but doesn&#39;t change my question.  It only _adds_ to my question as I don&#39;t understand why the terminal screen rendering should slow down an application writing to stdout.  Why does my program need to wait for screen rendering to continue?

Are all terminal/tty apps not created equal?  I have yet to experiment.  It really seems to me like a terminal should be able to buffer all incoming data, parse/render it invisibly, and only render the most recent chunk that is visible in the current screen configuration at a sensible frame rate.  So if I can write+fsync to disk in ~0.1 seconds, a terminal should be able to complete the same operation in something of that order (with maybe a few screen updates while it did it).

I&#39;m still kind of hoping there is a tty setting that can be changed from the application side to make this behaviour better for programmer.  If this is strictly a terminal application issue, then this maybe doesn&#39;t even belong on StackOverflow?

What am I missing?

----------


Here is the python program used to generate the timing:

    import time, sys, tty
    import os
    
    lineCount = 100000
    line = &quot;this is a test&quot;
    summary = &quot;&quot;
    
    cmd = &quot;print&quot;
    startTime_s = time.time()
    for x in range(lineCount):
        print line
    t = time.time() - startTime_s
    summary += &quot;%-30s:%6.3f s\n&quot; % (cmd, t)
    
    #Add a newline to match line outputs above...
    line += &quot;\n&quot;
    
    cmd = &quot;write to file (+fsync)&quot;
    fp = file(&quot;out.txt&quot;, &quot;w&quot;)
    startTime_s = time.time()
    for x in range(lineCount):
        fp.write(line)
    os.fsync(fp.fileno())
    t = time.time() - startTime_s
    summary += &quot;%-30s:%6.3f s\n&quot; % (cmd, t)
    
    cmd = &quot;print with stdout = /dev/null&quot;
    sys.stdout = file(os.devnull, &quot;w&quot;)
    startTime_s = time.time()
    for x in range(lineCount):
        fp.write(line)
    t = time.time() - startTime_s
    summary += &quot;%-30s:%6.3f s\n&quot; % (cmd, t)
    
    print &gt;&gt; sys.stderr, &quot;-----&quot;
    print &gt;&gt; sys.stderr, &quot;timing summary (100k lines each)&quot;
    print &gt;&gt; sys.stderr, &quot;-----&quot;
    print &gt;&gt; sys.stderr, summary


  [1]: http://www.linusakesson.net/programming/tty/index.php

Why is printing to stdout so slow? Can it be sped up?

Setting the default output encoding in Python 2 is a well-known idiom:

    sys.stdout = codecs.getwriter(&quot;utf-8&quot;)(sys.stdout)

This wraps the `sys.stdout` object in a codec writer that encodes output in UTF-8.

However, this technique does not work in Python 3 because `sys.stdout.write()` expects a `str`, but the result of encoding is `bytes`, and an error occurs when `codecs` tries to write the encoded bytes to the original `sys.stdout`.

What is the correct way to do this in Python 3?


How to set sys.stdout encoding in Python 3?

Is there a reason to use `endl` with `cout` when I can  just use `\n`? My C++ book says to use endl, but I don&#39;t see why. Is `\n` not supported as widely as `endl`, or am I missing something?



Why use endl when I can use a newline character?

I need to create a CSV and upload it to an S3 bucket. Since I&#39;m creating the file on the fly, it would be better if I could write it directly to S3 bucket as it is being created rather than writing the whole file locally, and then uploading the file at the end.

Is there a way to do this? My project is in Python and I&#39;m fairly new to the language. Here is what I tried so far:

    import csv
    import csv
    import io
    import boto
    from boto.s3.key import Key


    conn = boto.connect_s3()
    bucket = conn.get_bucket(&#39;dev-vs&#39;)
    k = Key(bucket)
    k.key = &#39;foo/foobar&#39;

    fieldnames = [&#39;first_name&#39;, &#39;last_name&#39;]
    writer = csv.DictWriter(io.StringIO(), fieldnames=fieldnames)
    k.set_contents_from_stream(writer.writeheader())

I received this error: BotoClientError: s3 does not support chunked transfer

**UPDATE: I found a way to write directly to S3, but I can&#39;t find a way to clear the buffer without actually deleting the lines I already wrote. So, for example:**

    conn = boto.connect_s3()
    bucket = conn.get_bucket(&#39;dev-vs&#39;)
    k = Key(bucket)
    k.key = &#39;foo/foobar&#39;

    testDict = [{
        &quot;fieldA&quot;: &quot;8&quot;,
        &quot;fieldB&quot;: None,
        &quot;fieldC&quot;: &quot;888888888888&quot;},
        {
        &quot;fieldA&quot;: &quot;9&quot;,
        &quot;fieldB&quot;: None,
        &quot;fieldC&quot;: &quot;99999999999&quot;}]

    f = io.StringIO()
    fieldnames = [&#39;fieldA&#39;, &#39;fieldB&#39;, &#39;fieldC&#39;]
    writer = csv.DictWriter(f, fieldnames=fieldnames)
    writer.writeheader()
    k.set_contents_from_string(f.getvalue())

    for row in testDict:
        writer.writerow(row)
        k.set_contents_from_string(f.getvalue())

    f.close()

Writes 3 lines to the file, however I&#39;m unable to release memory to write a big file. If I add:

    f.seek(0)
    f.truncate(0)
to the loop, then only the last line of the file is written. Is there any way to release resources without deleting lines from the file?

Can you upload to S3 using a stream rather than a local file?

````
BufferingHelper.EnableRewind();
````
Above is an extension method for HttpRequest object in ASP.NET Core 2.2. It is no more there in ASP.NET Core 3.0 (atleast with this name). I want to know it&#39;s alternate in ASP.NET Core 3.0. I am not sure if
````
HttpRequestRewindExtensions.EnableBuffering();
````
is the alternate.

Content Type	Original Author	Original Content on Stackoverflow
Question	bodacydo	View Question on Stackoverflow
Solution 1 - Stdout	lambshaanxy	View Answer on Stackoverflow
Solution 2 - Stdout	Siu Ching Pong -Asuka Kenji-	View Answer on Stackoverflow
Solution 3 - Stdout	Nordic Mainframe	View Answer on Stackoverflow
Solution 4 - Stdout	vroc	View Answer on Stackoverflow
Solution 5 - Stdout	Binghoo Dang	View Answer on Stackoverflow

How to make output of any shell command unbuffered?

Stdout Problem Overview

Stdout Solutions

Solution 1 - Stdout

Solution 2 - Stdout

Solution 3 - Stdout

Solution 4 - Stdout

Solution 5 - Stdout

In vim how to map "save" to ctrl-s

What is a portable method to find the maximum value of size_t?

Attributions