python – How do I execute a program or call a system command?

How to execute a program or call a system command from Python

Simple, use subprocess.runwhich returns a CompletedProcess object:

>>> from subprocess import run
>>> from shlex import split
>>> completed_process = run(split('python --version'))
Python 3.8.8
>>> completed_process
CompletedProcess(args=['python', '--version'], returncode=0)

(run wants a list of lexically parsed shell arguments – this is what you’d type in a shell, separated by spaces, but not where the spaces are quoted, so use a specialized function, splitto split up what you would literally type into your shell)

Why?

As of Python 3.5, the documentation recommends subprocess.run:

The recommended approach to invoking subprocesses is to use the run() function for all use cases it can handle. For more advanced use cases, the underlying Popen interface can be used directly.

Here’s an example of the simplest possible usage – and it does exactly as asked:

>>> from subprocess import run
>>> from shlex import split
>>> completed_process = run(split('python --version'))
Python 3.8.8
>>> completed_process
CompletedProcess(args=['python', '--version'], returncode=0)

run waits for the command to successfully finish, then returns a CompletedProcess object. It may instead raise TimeoutExpired (if you give it a timeout= argument) or CalledProcessError (if it fails and you pass check=True).

As you might infer from the above example, stdout and stderr both get piped to your own stdout and stderr by default.

We can inspect the returned object and see the command that was given and the returncode:

>>> completed_process.args
['python', '--version']
>>> completed_process.returncode
0

Capturing output

If you want to capture the output, you can pass subprocess.PIPE to the appropriate stderr or stdout:

>>> from subprocess import PIPE
>>> completed_process = run(shlex.split('python --version'), stdout=PIPE, stderr=PIPE)
>>> completed_process.stdout
b'Python 3.8.8n'
>>> completed_process.stderr
b''

And those respective attributes return bytes.

Pass a command list

One might easily move from manually providing a command string (like the question suggests) to providing a string built programmatically. Don’t build strings programmatically. This is a potential security issue. It’s better to assume you don’t trust the input.

>>> import textwrap
>>> args = ['python', textwrap.__file__]
>>> cp = run(args, stdout=subprocess.PIPE)
>>> cp.stdout
b'Hello there.n  This is indented.n'

Note, only args should be passed positionally.

Full Signature

Here’s the actual signature in the source and as shown by help(run):

def run(*popenargs, input=None, timeout=None, check=False, **kwargs):

The popenargs and kwargs are given to the Popen constructor. input can be a string of bytes (or unicode, if specifying encoding or universal_newlines=True) that will be piped to the subprocess’s stdin.

The documentation describes timeout= and check=True better than I could:

The timeout argument is passed to Popen.communicate(). If the timeout expires, the child process will be killed and waited for. The TimeoutExpired exception will be re-raised after the child process has terminated.

If check is true, and the process exits with a non-zero exit code, a CalledProcessError exception will be raised. Attributes of that exception hold the arguments, the exit code, and stdout and stderr if they were captured.

and this example for check=True is better than one I could come up with:

>>> subprocess.run("exit 1", shell=True, check=True)
Traceback (most recent call last):
  ...
subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1

Expanded Signature

Here’s an expanded signature, as given in the documentation:

subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None, 
shell=False, cwd=None, timeout=None, check=False, encoding=None, 
errors=None)

Note that this indicates that only the args list should be passed positionally. So pass the remaining arguments as keyword arguments.

Pope

When use Popen instead? I would struggle to find use-case based on the arguments alone. Direct usage of Popen would, however, give you access to its methods, including poll‘send_signal’, ‘terminate’, and ‘wait’.

Here’s the Popen signature as given in the source. I think this is the most precise encapsulation of the information (as opposed to help(Popen)):


def __init__(self, args, bufsize=-1, executable=None,
             stdin=None, stdout=None, stderr=None,
             preexec_fn=None, close_fds=True,
             shell=False, cwd=None, env=None, universal_newlines=None,
             startupinfo=None, creationflags=0,
             restore_signals=True, start_new_session=False,
             pass_fds=(), *, user=None, group=None, extra_groups=None,
             encoding=None, errors=None, text=None, umask=-1, pipesize=-1):

But more informative is the Popen documentation:

subprocess.Popen(args, bufsize=-1, executable=None, stdin=None, stdout=None, 
stderr=None, preexec_fn=None, close_fds=True, shell=False, cwd=None,
env=None, universal_newlines=None, startupinfo=None, creationflags=0, 
restore_signals=True, start_new_session=False, pass_fds=(), *, group=None, 
extra_groups=None, user=None, umask=-1, encoding=None, errors=None, 
text=None)

Execute a child program in a new process. On POSIX, the class uses os.execvp()-like behavior to execute the child program. On Windows, the class uses the Windows CreateProcess() function. The arguments to Popen are as follows.

Understanding the remaining documentation on Popen will be left as an exercise for the reader.

Leave a Comment