Jenkins console output not in realtime
Continuous IntegrationJenkinsContinuous Integration Problem Overview
Pretty new to Jenkins and I have simple yet annoying problem. When I run job (Build) on Jenkins I am triggering ruby command to execute my test script.
Problem is Jenkins is not displaying output in real time from console. Here is trigger log.
Building in workspace /var/lib/jenkins/workspace/foo_bar
No emails were triggered.
[foo_bar] $ /bin/sh -xe /tmp/hudson4042436272524123595.sh
+ ruby /var/lib/jenkins/test-script.rb
Basically it hangs on this output until build is complete than it just shows full output. Funny thing is this is not consistent behavior, sometimes it works as it should. But most of the time there is no real time console output.
Jenkins version: 1.461
Continuous Integration Solutions
Solution 1 - Continuous Integration
To clarify some of the answers.
ruby
orpython
or any sensible scripting language will buffer the output; this is in order to minimize the IO; writing to disk is slow, writing to a console is slow...- usually the data gets
flush()
'ed automatically after you have enough data in the buffer with special handling for newlines. e.g. writing a string without newline thensleep()
would not write anything until after thesleep()
is complete (I'm only usingsleep
as an example, feel free to substitute with any other expensive system call).
e.g. this would wait 8 seconds, print one line, wait 5 more seconds, print a second line.
from time import sleep
def test():
print "ok",
time.sleep(3)
print "now",
time.sleep(5)
print "done"
time.sleep(5)
print "again"
test()
-
for
ruby
,STDOUT.sync = true
, turns theautoflush
on; all writes toSTDOUT
are followed byflush()
. This would solve your problem but result in more IO.STDOUT.sync = true
-
for
python
, you can usepython -u
or the environment variablePYTHONUNBUFFERED
to makestdin/stdout/stout
not buffered, but there are other solutions that do not changestdin
orstderr
export PYTHONUNBUFFERED=1
-
for
perl
, you haveautoflush
autoflush STDOUT 1;
Solution 2 - Continuous Integration
Make sure your script is flushing its stdout and stderr. In my case I had a buffering issue similar to what you describe but I was using python. The following python code fixed it for me:
import sys
sys.stdout.flush()
I'm not a Ruby coder, but Google reveals the following:
$stdout.flush
Solution 3 - Continuous Integration
It seems to me that python -u
works as well.
E.g. In batch command
python -u foo.py
Solution 4 - Continuous Integration
Easiest solution here is to turn on syncing buffer to output. Something that @Craig wrote about in his answer but one line solution that will cover whole script, and not require you to flush buffer many times.
Just write
STDOUT.sync = true
Logic behind is simple, to avoid using IO operations many times output is buffered. To disable this use
STDOUT.sync = false
This is Ruby solution ofc.
Solution 5 - Continuous Integration
Each of the other answers is specific to one program or another, but I found a more general solution here:
https://unix.stackexchange.com/a/25378
You can use stdbuf
to alter the buffering behavior of any program.
In my case, I was piping output from a shell script through tee
and grep
to split lines into either the console or a file based on content. The console was hanging as described by OP. This solved it:
./slowly_parse.py login.csv |tee >(grep -v LOG: > out.csv) | stdbuf -oL -eL grep LOG:
Eventually I discovered I could just pass --line-buffered
to grep for the same result:
./slowly_parse.py login.csv |tee >(grep -v LOG: > out.csv) | grep --line-buffered LOG:
Solution 6 - Continuous Integration
The other answers are correct in saying that you need to ensure standard output is not buffered.
The other thing to be aware of is that Jenkins itself does line by line buffering. If you have a slow-running process that emits single characters (for example, an nunit test suite summary that prints a .
for a successful test and an E
for an error) you will not see anything until the end of line.
[True for my Jenkins 1.572 running on a Windows box.]
Solution 7 - Continuous Integration
For some commands, including tee
a the best choice for unbuffering is a program called unbuffer
from expect
package.
Usage example:
instead of
somecommand | tee /some/path
do
somecommand | unbuffer -p tee /some/path
Sources and more info:
Solution 8 - Continuous Integration
The Operating-System is buffering output-data by nature, to save CPU, and so does Jenkins.
Looks like you are using a shell-command to run your Ruby script -
I suggest running your Ruby script directly via the dedicated plugin:
(may need to install it)
Solution 9 - Continuous Integration
Python buffered its output traces and print it at the end of script to minimize writing on console as writing to console is slow.
You can use following command after your traces. It will flush all traces to console, which are queued before that command.
> sys.stdout.flush()