How to use the output from python in the bash pipeline right away?

In this code, you encounter standard output stream buffering (i.e., information transmitted to the output stream is transmitted only when its volume is large enough).

In order for the print statement in python 2.x (or the print function in python 3.x) to clear the output buffer and pass the information to the operating system, use the flush method of the sys.stdout object:

 #!/usr/bin/env python # -*- coding: utf-8 -*- import sys import time print(u"Привет!") sys.stdout.flush() time.sleep(10) 

Example:

 $ PYTHONIOENCODING=utf-8 your_script | grep вет 

You can not add flush() calls, but use the -u option, which disables buffers on standard I / O streams:

 $ PYTHONIOENCODING=utf-8 python -u your_script.py | grep вет 

PYTHONIOENCODING is used here to correctly specify the encoding in which the program output is performed. It has nothing to do with the encoding of the source code.

Short : $ unbuffer ./script.py | grep pattern $ unbuffer ./script.py | grep pattern

If you run the script without redirecting its output to grep, then the output should appear immediately without waiting for the buffer to be filled:

 $ ./script.py Привет! 

If the standard output is sent to the terminal (it is assumed that the person is looking at it), then the output is buffered line by line (the buffer is cleared after each new line).

If the output is redirected to a pipe (consumed by another program — grep in this case), the buffer is not cleared until it is completely full (typical size is 4KB-8KB) —can be for performance reasons to reduce the number of system calls ( write(2) ). Therefore, the output in the example is not shown until the script has completed (when all buffers are cleared during normal output). Picture showing stdio buffers inside processes and pipe buffer in the kernel for command1 | command2 command1 | command2 commands :

This is a common behavior among programs that use C stdio, such as CPython implementation of Python 2. See the second reason in Q: Why not just use a pipe (popen ())?

There are several ways to clear the buffer or remove the buffering of the standard output in general in Python:

• pass the -u command line parameter: python -u ... You can set this parameter in shebang: #!/usr/bin/python -u and run the script directly: ./script.py .

• set the PYTHONUNBUFFERED environment variable:

   $ PYTHONUNBUFFERED=nonempty python ... 

• replace sys.stdout with an unbuffered stream (for example, using os.fdopen() )

• pass flush=True parameter to print() function
• call sys.stdout.flush()

See How to flush output of Python print?

Unlike Python 2, on Python 3, text streams use a buffer for strings even if -u passed — a difference can occur if the output does not contain a new line, for example:

 import sys import time for i in range(10): print(i) sys.stdout.write('before sleep ') time.sleep(1) print('after sleep') 

If you pass -u , the numbers are printed at once in both versions (since print(i) prints a new line). In Python 3, 'before sleep' delayed until the next iteration (when the next new line is printed), despite the -u parameter.

Ways that require changing the source code ( sys.stdout manipulation) can be fragile (you can forget to substitute flush=True in the right place) or not work at all (if the output is in uncontrolled code) and they are not always needed. It is often better to configure the environment in which the script is run to enable the desired buffering mode, instead of editing the code.

If your script starts other programs that can print to standard output, you can try to replace the C stdio buffer or use pty to make the program think that it outputs to the terminal, not pipe:

• $ unbuffer python ...
• $ stdbuf -oL -eL python ...
• $ script -q -c 'python ...' /dev/null | grep ...

See Turn off buffering in pipe .