Python non-blocking read with subprocess.Popen

non_blocking_read.py

import fcntl
import os
from subprocess import *

def non_block_read(output):
    fd = output.fileno()
    fl = fcntl.fcntl(fd, fcntl.F_GETFL)
    fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
    try:
        return output.read()
    except:
        return ""

############
# Use case #
############

sb = Popen("echo test; sleep 10000", shell=True, stdout=PIPE)
sb.kill()
sb.poll() # return -9
#sb.stdout.read() # Will block and will block forever cause nothing will come out since the job is done
non_block_read(sb.stdout) # will return '' instead of hanging for ever

Note that fcntl isn't available on Windows.

Thank you you saved my life :)

Note that fcntl isn't available on Windows. =_= ...

Using shell=True "makes a program vulnerable to shell injection, a serious security flaw which can result in arbitrary command execution. For this reason, the use of shell=True is strongly discouraged" . Is it possible to adapt this approach to shell=False?

Just look at the code: the shell=True is only there so that the example (which is in shell notation) can work:

sb = Popen("echo test; sleep 10000", shell=True, stdout=PIPE)

If you don't use shell syntax, or rely on any other shell features, then you don't need shell=True.

This is much simpler:

p = Popen("echo test; sleep 10000", shell=True, stdout=PIPE)
for x in iter(p.stdout.readline, b''):
    print(x)

Thank you so much. I've been trying just about everything to get this working with a read -p "Type something" command inside a script (which outputs to stderr without a newline making readline() not work) and this is the only thing that has worked properly.

This is much simpler:

p = Popen("echo test; sleep 10000", shell=True, stdout=PIPE)
for x in iter(p.stdout.readline, b''):
    print(x)

p.stdout.readline() blocks. That is why this gist is about non_blocking_read

Readline version (strips '\n')

def non_block_readline(output) -> str:
    fd = output.fileno()
    fl = fcntl.fcntl(fd, fcntl.F_GETFL)
    fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
    try:
        return output.readline().strip("\n")
    except:
        return ""

Since Python 3.5 os.set_blocking can do exactly the same thing:

import os
from subprocess import *

sb = Popen("echo test; sleep 10000", shell=True, stdout=PIPE)
os.set_blocking(sb.stdout.fileno(), False)  # That's what you are looking for
sb.kill()
sb.poll() # return -9
sb.stdout.read()  # This is not going to block. When the pipe is empty it returns an empty string.