1 Introduction
The Subplot library runcmd for Python provides scenario steps
and their implementations for running Unix commands and examining the
results. The library consists of a bindings file lib/runcmd.yaml and
implementations in Python in lib/runcmd.py or in the Rust subplotlib
step library. There is no Bash version.
This document explains the acceptance criteria for the library and how
they're verified. It uses the steps and functions from the
lib/runcmd library. The scenarios all have the same structure: run a
command, then examine the exit code, standard output (stdout for
short), or standard error output (stderr) of the command.
The scenarios use the Unix commands true and false to
generate exit codes, and echo to produce stdout. To generate
stderr, they use the little helper script below.
#!/bin/sh echo "$@" 1>&2
#!/bin/sh echo $HOME
2 Check exit code
These scenarios verify the exit code. To make it easier to write scenarios in language that flows more naturally, there are a couple of variations.
2.1 Successful execution
2.2 Successful execution in a sub-directory
2.3 Execution in a sub-directory does not affect HOME
2.4 Failed execution
2.5 Failed execution in a sub-directory
3 Check we can prepend to $PATH
This scenario verifies that we can add a directory to the beginning of
the PATH environment variable, so that we can have runcmd invoke a
binary from our build tree rather than from system directories. This
is especially useful for testing new versions of software that's
already installed on the system.
#!/bin/sh echo "custom ls, not system ls"
4 Check output has what we want
These scenarios verify that stdout or stderr do have something we want to have.
4.1 Check stdout is exactly as wanted
Note that the string is surrounded by double quotes to make it clear to the reader what's inside. Also, C-style string escapes are understood.
4.2 Check stderr is exactly as wanted
4.3 Check stdout using sub-string search
Exact string comparisons are not always enough, so we can verify a sub-string is in output.
4.4 Check stderr using sub-string search
4.5 Check stdout using regular expressions
Fixed strings are not always enough, so we can verify output matches a regular expression. Note that the regular expression is not delimited and does not get any C-style string escaped decoded.
4.6 Check stderr using regular expressions
5 Check output doesn't have what we want to avoid
These scenarios verify that the stdout or stderr do not have something we want to avoid.
5.1 Check stdout is not exactly something
5.2 Check stderr is not exactly something
5.3 Check stdout doesn't contain sub-string
5.4 Check stderr doesn't contain sub-string
5.5 Check stdout doesn't match regular expression
5.6 Check stderr doesn't match regular expressions
5.7 Setting and clearing of environment variables
A default scenario environment won't contain extra values
unless they're passed in deliberately with SUBPLOT_ENV_* names.
We can set variables to ensure they end up set
And they will remain set until the scenario finishes
Unless we clear the variable explicitly
6 Check redirection
6.1 Stdin comes from a named file
hello, world