1 Introduction

Subplot is software to help capture and communicate acceptance criteria for software and systems, and how they are verified, in a way that's understood by all project stakeholders. This is the user guide for Subplot.

The acceptance criteria are expressed as scenarios, which roughly correspond to use cases. The scenario as accompanied by explanatory text to explain things to the reader. Scenarios use a given/when/then sequence of steps, where each step is implemented by code provided by the developers of the system under test or by Subplot itself. This is very similar to the Cucumber tool, but with more emphasis on producing a standalone document.

1.1 Acceptance criteria and acceptance tests

We define the various concepts relevant to Subplot as follows:

• Acceptance criteria: What the stakeholders require of the system for them to be happy with it and use it.

• Stakeholder: Someone with a keen interest in the success of a system. They might be a paying client, someone who uses the system, or someone involved in developing the system. Depending on the system and project, some stakeholders may have a bigger say than others.

• Acceptance test: How stakeholders verify that the system fulfills the acceptance criteria, in an automated way. Some criteria may not be possible to verify automatically.

• Scenario: In Subplot, the acceptance criteria are written as freeform prose, with diagrams, etc. The scenarios, which are embedded blocks of Subplot scenario language, capture the mechanisms of verifying that criteria are met - the acceptance tests - showing step by step how to determine that the software system is acceptable to the stakeholders.

1.2 A basic workflow for using Subplot

We recommend the following initial approach to using Subplot, which you can vary based on your particular needs and circumstances.

1. Start with a small acceptance document that you think expresses some useful requirements.
2. Write some acceptance criteria and have them agreed among the stakeholders.
3. Write scenarios to verify that the criteria are met, and have those scenarios agreed by the stakeholders.
4. Write bindings and test functions, so that as the code is written it can be tested against the acceptance criteria.
5. Iterate on this in short cycles to maximise discussion and stakeholder buy-in.

You definitely want to keep the subplot document source code in version control. You certainly need to have people who can write technical text that's aimed at all your stakeholders.

1.3 Subplot architecture

Subplot reads an input document, in Markdown, and generates a typeset output document, as HTML, for all stakeholders to understand. Subplot also generates a test program, in Python, that verifies the acceptance criteria are met, for developers and testers and auditors to verify the system under test meets its acceptance criteria. The generated program uses code written by the Subplot user to implement the verification steps. The diagram below illustrates this and shows how data flows through the system.

Subplot generated HTML itself.

Subplot actually consists mainly of two separate programs: subplot docgen for generating output documents, and subplot codegen for generating the test program. There are a couple of additional tools (subplot metadata for reporting meta data about a Subplot document, and subplot extract for extracting embedded files from a subplot document.

Thus a more detailed architecture view is shown below.

1.4 A fairy tale of acceptance testing

The king was upset. This naturally meant the whole court was in a tizzy and chattering excitedly at each other, while trying to avoid the royal wrath.

"Who will rid me of this troublesome chore?" shouted the king, and quaffed a flagon of wine. "And no killing of priests, this time!"

The grand hall's doors were thrown open. The grand wizard stood in the doorway, robe, hat, and staff everything, but quite still. After the court became silent, the wizard strode confidently to stand before the king.

"What ails you, my lord?"

The king looked upon the wizard, and took a deep breath. It does not do to shout at wizards, for they control dragons, and even kings are tasty morsels to the great beasts.

"I am tired of choosing what to wear every day. Can't you do something?"

The wizard stoked his long, grey beard. He turned around, looked at the magnificent outfits worn by members of the court. He turned back, and looked at the king.

"I believe I can fix this. Just to be clear, your beef is with having to choose clothing, yes?"

"Yes", said the king, "that's what I said. When will you be done?"

The wizard raised his staff and brought it back down again, with a loud bang.

"Done" said the wizard, smugly.

The king was amazed and started smiling, until he noticed that everyone, including himself, was wearing identical burlap sacks and nothing on their feet. His voice was high, whiny, like that of a little child.

"Oh no, that's not at all what I wanted! Change it back! Change it back now!"

The moral of this story is to be clear and precise in your acceptance criteria, or you might get something other than what you really, really wanted.

1.5 Motivation for Subplot

Keeping track of requirements and acceptance criteria is necessary for all but the simplest of software projects. Having all stakeholders in a project agree to them is crucial, as is that all agree how it is verified that the software meets the acceptance criteria. Subplot provides a way for documenting the shared understanding of what the acceptance criteria are and how they can be checked automatically.

Stakeholders in a project may include:

• those who pay for the work to be done; this may be the employer of the developers for in-house projects ("customer")
• those who use the resulting systems, whether they pay for it or not ("user")
• those who install and configure the systems and keep them functional ("sysadmin")
• those who support the users ("support")
• those who test the project for acceptability ("tester")
• those who develop the system in the first place ("developer")

The above list is incomplete and simplistic, but suffices as an example.

All stakeholders need to understand the acceptance criteria, and how the system is evaluated against the criteria. In the simplest case, the customer and the developer need to both understand and agree so that the developer knows when the job is done, and the customer knows when they need to pay their bill.

However, even when the various stakeholder roles all fall upon the same person, or only on people who act as developers, the Subplot tooling can be useful. A developer would understand acceptance criteria expressed only in code, but doing so may take time and energy that are not always available. The Subplot approach aims to encourage hiding unnecessary detail and documenting things in a way that is easy to understand with little effort.

Unfortunately^*^, this does mean that for a Subplot output document to be good and helpful, writing it will require effort and skill. No tool can replace that.

* - Or fortunately, if you're the person whose job it is to write the documents of course.

2 Terminology

In Subplot we use the following terminology:

• A need (or "want" or "ask" or "request"): what a stakeholder wants from the system at the level they are concerned. This is often fuzzy, vague, and unclear and there may not be any way to verify that is is met.

  This is what a client tells a salesperson during the sales process, or an open source user says in a wishlist bug report.

  For example: a client might ask for the backup program to de-duplicate data in the backups; this is too vague to be implemented with any confidence. It leaves open too many details: what kind of data should be de-duplicated? At what granularity? Should it happen for data across users, and if so, how much can users know about each others' data? Should the de-duplication happen on the client machines, on the server, or only in storage?

• Acceptance criteria: needs expressed in a way that are clear, unambiguous, and specific, and with a specified way for verifying the criteria are met.

  This is what is nailed down in the contract so that it's clear when payment is due; or what an open source project commits to supporting.

• Stakeholder: anyone whose opinion about the system matters and who gets to affect what the acceptance criteria are, for their level of involvement. Stakeholders can be involved at various levels in a project. While the levels will vary between projects, they might be something like:

  • very high level: I am the CEO of the client, and I have final say on what we're willing to pay for, and I set the acceptance criteria that the system can back up all our important files on production systems; this will be verified by backing up all our production data.

  • medium level: I am the CTO of the client and I add the following acceptance criteria:

    • backed up data can be restored, verified by restoring a backup and comparing to the original data
    • backups are encrypted, verified by checking the backup for cleartext data
    • a full backup and restore of a terabyte of data takes at most one hour in a setup similar to our intended production systems, verified by measuring the backup and restore of production data

  • low level: I am the security analyst at the client and I add the acceptance criteria that the encryption uses SHA3 for integrity checking and the Ed25519 elliptic curve for encryption and digital signatures; these will be verified by third party inspection

• Requirements: since acceptance criteria are about the stakeholders accepting the software that gets offered to them, the development organization may use these criteria as a basis from which to derive more granular and specific statements about what the project should do and how to check that it does so. These are the traditional requirements and unit/integration tests which most software developers work with.

3 Subplot input language

Subplot reads upwards of four input files, each in a different format:

• The "subplot document" file in YAML,
• The document file(s) in GitHub Flavored Markdown.
• The bindings file(s), in YAML.
• The functions file(s), in Python, Rust, or some other language.

Subplot interprets marked parts of the input markdown documents specially. These are fenced code blocks tagged with the scenario, file, or example classes.

3.1 Scenario language

The scenarios are core to Subplot. They express what the detailed acceptance criteria are and how they're verified. The scenarios are meant to be understood by both all human stakeholders and the Subplot software. As such, they are expressed in a somewhat stilted language that resembles English, but is just formal enough that it can also be understood by a computer.

A scenario is a sequence of steps. A step can be setup to prepare for an action, an action, or an examination of the effect an action had. For example, a scenario to verify that a backup system works might look like the following:

~~~scenario
given a backup server
when I make a backup
and I restore the backup
then the restored data is identical to the original data
~~~

This is not magic. The three kinds of steps are each identified by the first word in the step.

• given means it's a step to set up the environment for the scenario
• when means it's a step with the action that the scenario verifies
• then means it's a step to examine the results of the action

The and keyword is special in that it means the step is the same kind as the previous step. In the example, on line 4, it means the step is a when step.

Each step is implemented by a bit of code, provided by the author of the subplot document. The step is bound to the code via a binding file, via the text of the step: if the text is like this, then call that function. Bindings files are described in detail shortly below.

The three kinds of steps exist to make scenarios easier to understand by humans. Subplot itself does not actually care if a step is setup, action, or examination, but it's easier for humans reading the scenario, or writing the corresponding code, if each step only does the kind of work that is implied by the kind of step it's bound to.

3.1.1 Using Subplot's language effectively

Your subplot scenarios will be best understood when they use the subplot language in a consistent fashion, within and even across different projects. As with programming languages, it's possible to place your own style on your subplots. Indeed, there is no inherent internal implementation difference between how given, when and then steps are processed (other than that given steps often also have cleanup functions associated with them).

Nonetheless we have some recommendations about using the Subplot language, which reflect how we use it in Subplot and related projects.

When you are formulating your scenarios, it is common to try and use phraseology along the lines of if this happens then that is the case but this is not language which works well with subplot. Scenarios describe what will happen in the success case. As such we don't construct scenarios which say if foo happens then the case fails, instead we say when I do the thing then foo does not happen. This is a subtle but critical shift in the construction of your test cases which will mean that they map more effectively to scenarios.

Scenarios work best when they describe how some entity (human or otherwise) actually goes about successfully achieving their goal. They start out by setting the scene for the goal (given) they go on to describe the actions/activity undertaken in order for the goal to be achieved (when) and they describe how the entity knows that the goal has been achieved (then). By writing in this active goal-oriented fashion, your scenarios will flow better and be easier for all stakeholders to understand.

In general you should use given statements where you do not wish to go into the detail of what it means for the statement to have been run, you simply wish to inform the reader that some precondition is met. These statements are often best along the lines of given a setup which works or given a development enviroment or somesuch.

The when statements are best used to denote active steps. These are the steps which your putative actors or personae use to achieve their goals. These often work best in the form when I do the thing or when the user does the thing.

The then statements are the crux of the scenario, they are the validation steps. These are the steps which tell the reader of the scenario how the actor knows that their action (the when steps) has had the desired outcome. This could be of the form then some output is present or then it exits successfully.

With all that in mind, a good scenario looks like

given the necessary starting conditions
when I do the required actions
then the desired outcome is achieved

Given all that, however, it's worth considering some pitfalls to avoid when writing your scenarios.

It's best to avoid overly precise or overly technical details in your scenario language (unless that's necessary to properly describe your goal etc.) So it's best to say things like then the output file is valid JSON rather than then the output file contains {"foo": "bar", "baz": 7}. Obviously if the actual values are important then again, statements such as then the output file has a key "foo" which contains the value "bar" or similar.

Try not to change "person" or voice in your scenarios unless there are multiple entities involved in telling your stories. For example, if you have a scenario statement of when I run fooprogram do not also have statements in the passive such as when fooprogram is run. It's reasonable to switch between when and then statements (then the output is good) but try not to have multiple then statements which switch it up, such as then I have an output file, and the output file is ok.

If you're likely to copy-paste your scenario statements around, do not use and as a scenario keyword, even though it's valid to do so. Instead start all your scenario statements with the correct given, when, or then. The typesetter will deal with formatting that nicely for you.

3.2 Document markup

Subplot parses Markdown input files using GitHub-flavored Markdown.

Subplot supports most of the major features of gfm including tables, task lists, _sub- and ^super- script, strike-through, and heading attributes.

Subplot extends Markdown by treating certain certain tags for fenced code blocks specially. A scenario, for example, would look like this:

```scenario
given a standard setup
when peace happens
then everything is OK
```

The scenario tag on the code block is recognized by Subplot, which will typeset the scenario (in output documents) or generate code (for the test program) accordingly. Scenario blocks do not need to be complete scenario. Subplot will collect all the snippets into one block for the test program. Snippets under the same heading belong together; the next heading of the same or a higher level ends the scenario.

For scenario blocks you may not use any attributes. All attributes are reserved for Subplot. Subplot doesn't define any attributes yet, but by reserving all of them, it can add them later without it being a breaking change.

For embedding test data files in the Markdown document, Subplot understands the file tag:

~~~{#filename .file}
This data is accessible to the test program as 'filename'.
~~~

The .file attribute is necessary, as is the identifier, here #filename. The generated test program can access the data using the identifier (without the #).

Subplot also understands the dot and roadmap tags, and can use the Graphviz dot program, or the roadmap Rust crate, to produce diagrams. These can useful for describing things visually.

When typesetting files, Subplot will automatically number the lines in the file so that documentation prose can refer to sections of embedded files without needing convoluted expressions of positions. However if you do not want that, you can annotate the file with .noNumberLines.

For example…

~~~{#numbered-lines.txt .file}
This file has numbered lines.

This is line number three.
~~~

~~~{#not-numbered-lines.txt .file .noNumberLines}
This file does not have numbered lines.

This is still line number three, but would it be obvious?
~~~

…renders as:

This file has numbered lines.

This is line number three.

This file does not have numbered lines.

This is still line number three, but would it be obvious?

3.2.1 Use embedded file

This scenario makes sure the sample files are used in a scenario so that they don't cause warnings.

3.3 Document metadata

Document metadata is read from the Subplot Document (YAML file). This can used to set the document title, authors, date (version), and more. Crucially for Subplot, the bindings and functions files are named in the metadata block, rather than Subplot deriving them from the input file name.

title: "Subplot"
authors:
- The Subplot project
date: work in progress
markdowns:
- subplot.md
bindings:
- subplot.yaml
impls:
  python:
    - subplot.py

There can be more than one bindings or functions file: use a YAML list.

3.4 Bindings file

The bindings file binds scenario steps to code functions that implement the steps. The YAML file is a list of objects (also known as dicts or hashmaps or key/value pairs), specifying a step kind (given, when, then), a pattern matching the text of the step and optionally capturing interesting parts of the text. Each binding may contain a type map which tells subplot the types of the captures in the patterns so that they can be validated to some extent, and a binding will list some number of implementations, each of which is specified by the name of the language (template) it is for, and then the name of a function that implements the step, optionally with the name of a function to call to clean up a scenario which includes that step.

There are some flexibilities in bindings, futher details can be found below:

1. Patterns can be simple or full-blown Perl-compatible regular expresssions (PCRE).
2. Bindings may have type maps. Without a type map, all captures are considered to be short strings (words).
3. Bindings may have as many or as few implementations as needed. A zero impl binding will work for docgen but will fail to codegen. This can permit document authors to prepare bindings without knowing how an engineer might implement it.

- given: "a standard setup"
  impl:
    python:
      function: create_standard_setup
- when: "{thing} happens"
  impl:
    python:
      function: make_thing_happen
  types:
    thing: word
- when: "I say (?P<sentence>.+) with a smile"
  regex: true
  impl:
    python:
      function: speak
- then: "everything is OK"
  impl:
    python:
      function: check_everything_is_ok

In the example above, there are four bindings and they all provide Python implementation functions:

• A binding for a "given a standard setup" step. The binding captures no part of the text, and causes the create_standard_setup function to be called.
• A binding for a "when" step consisting of one word followed by "happens". For example, "peace", as in "then peace happens". The word is captured as "thing", and given to the make_thing_happen function as an argument when it is called.
• A binding for a "when" followed by "I say", an arbitrary sentence, and then "with a smile", as in "when I say good morning to you with a smile". The function speak is then called with capture named "sentence" as "good morning to you".
• A binding for a "then everything is OK" step, which captures nothing, and calls the check_everything_is_ok function.

3.5 Step functions and cleanup

A step function must be atomic: either it completes successfully, or it cleans up any changes it made before returning an indication of failure.

A cleanup function is only called for successfully executed step functions.

For example, consider a step that creates and starts a virtual machine. The step function creates the VM, then starts it, and if both actions succeeds, the step succeeds. A cleanup function for that step will stop and delete the VM. The cleanup is only called if the step succeeded. If the step function manages to create the VM, but not start it, it's the step function's responsibility to delete the VM, before it signals failure. The cleanup function won't be called in that case.

3.5.1 Simple patterns

The simple patterns are of the form {name} and match a single word consisting of printable characters. This can be varied by adding a suffix, such as {name:text} which matches any text. The following kinds of simple patterns are supported:

• {name} or {name:word} – a single word. As a special case, the form {name:file} is also supported. It is also a single word, but has the added constraint that it must match an embedded file's name.
• {name:text} – any text
• {name:int} – any whole number, including negative
• {name:uint} – any unsigned whole number
• {name:number} – any number
• {name:escapedword} – a single word, in which backslash escapes are processed during the generation of the test suite.
• {name:escapedtext} – any text, in which backslash escapes are processed during the generation of the test suite.

A pattern uses simple patterns by default, or if the regex field is set to false. To use regular expressions, regex must be set to true. Subplot complains if typical regular expression characters are used, when simple patterns are expected, unless regex is explicitly set to false.

3.5.2 Regular expression patterns

Regular expression patterns are used only if the binding regex field is set to true.

The regular expressions use PCRE syntax as implemented by the Rust regex crate. The (?P<name>pattern) syntax is used to capture parts of the step. The captured parts are given to the bound function as arguments, when it's called.

3.5.3 The type map

Bindings may also contain a type map. This is a dictionary called types and contains a key-value mapping from capture name to the type of the capture. Valid types are listed above in the simple patterns section. In addition to simple patterns, the type map can be used for regular expression bindings as well.

When using simple patterns, if the capture is given a type in the type map, and also in the pattern, then the types must match, otherwise subplot will refuse to load the binding.

Typically the type map is used by the code generators to, for example, distinguish between "12" and 12 (i.e. between a string and what should be a number). This permits the generated test suites to use native language types directly. The file type, if used, must refer to an embedded file in the document; subplot docgen will emit a warning if the file is not found, and subplot codegen will emit an error.

3.5.4 The implementation map

Bindings can contain an impl map which connects the binding with zero or more language templates. If a binding has no impl entries then it can still be used to docgen a HTML document from a subplot document. This permits a workflow where requirements owners / architects design the validations for a project and then engineers implement the step functions to permit the validations to work.

Shipped with subplot are a number of libraries such as files or runcmd and these libraries are polyglot in that they provide bindings for all supported templates provided by subplot.

Here is an example of a binding from one of those libraries:

- given: file {embedded_file}
  impl:
    rust:
      function: subplotlib::steplibrary::files::create_from_embedded
    python:
      function: files_create_from_embedded
  types:
    embedded_file: file

3.5.5 Embedded file name didn't match

title: Bad filenames in matched steps do not permit codegen
markdowns: [badfilename.md]
bindings: [b.yaml]
impls:
  python: [f.py]

# Bad filename

```scenario
given file missing.md
```

3.5.6 Bindings file strictness - given when then

The bindings file is semi-strict. For example you must have only one of given, when, or then in your binding.

title: Bad bindings cause everything to fail
markdowns: [badbindingsgwt.md]
bindings: [badbindingsgwt.yaml]

# Bad bindings
```scenario
given we won't reach here
```

- given: we won't reach here
  then: we won't reach here

3.5.7 Bindings file strictness - unknown field

The bindings file is semi-strict. For example, you must not have keys in the bindings file which are not known to Subplot.

title: Bad bindings cause everything to fail
markdowns: [badbindingsuf.md]
bindings: [badbindingsuf.yaml]

# Bad bindings
```scenario
given we won't reach here
```

- given: we won't reach here
  function: old_school_function

3.6 Functions file

Functions implementing steps are supported in Python. The language is chosen by setting the template field in the document YAML metadata to python.

The functions files are not parsed by Subplot at all. Subplot merely copies them to the output. All parsing and validation of the file is done by the programming language being used.

The conventions for calling step functions vary by language. All languages support a "dict" abstraction of some sort. This is most importantly used to implement a "context" to store state in a controlled manner between calls to step functions. A step function can set a key to a value in the context, or retrieve the value for a key.

Typically, a "when" step does something, and records the results into the context, and a "then" step checks the results by inspecting the context. This decouples functions from each other, and avoids having them use global variables for state.

3.6.1 Python

The context is implemented by a dict-like class.

The step functions are called with a ctx argument that has the current state of the context, and each capture from a step as a keyword argument. The keyword argument name is the same as the capture name in the pattern in the bindings file.

The contents of embedded files are accessed using a function:

• get_file(filename)

Example:

import json

def exit_code_is(ctx, wanted=None):
    assert_eq(ctx.get("exit"), wanted)

def json_output_matches_file(ctx, filename=None):
    actual = json.loads(ctx["stdout"])
    expected = json.load(open(filename))
    assert_dict_eq(actual, expected)

def file_ends_in_zero_newlines(ctx, filename=None):
    content = open(filename, "r").read()
    assert_ne(content[-1], "\n")

3.7 Comparing the scenario runners

Currently Subplot ships with three scenario runner templates. The Python and Rust templates. Given that, this comparison is only considered correct against the version of Rust at the time of publishing a Subplot release. Newer versions of Rust may introduce additional functionality which we do not list here. Finally, we do not list features here which are considered fundamental, such as "runs all the scenarios" or "supports embedded files" since no template would be considered for release if it did not do these things. These are the differentiation points.

Feature	Python	Rust
Isolation model	Subprocess	Threads
Parallelism	None	Threading
Passing environment variables	CLI	Prefixed env vars
Execution order	Randomised	Fixed order plus threading peturbation
Run specific scenarios	Simple substring check	Either exact or simple substring check
Diagnostic logging	Supports comprehensive log file	Writes captured output to stdout/stderr on failure
Stop-on-failure	Stops on first failure unless told not to	Runs all tests unless told not to
Data dir integration	Cleans up each scenario unless told to save it	Cleans up each scenario with no option to save failure state

4 An overview of acceptance criteria and their verification

• discuss acceptance criteria vs requirements; functional vs non-functional requirements; automated vs manual testing
• discuss stakeholders
• discuss different approaches for verifying that a system meets it criteria
• discuss how scenarios can be used to verify acceptance criteria

5 Simple example project

• discuss how to use Subplot for some simple, but not simplistic, software project
• discuss different kinds of stakeholders a project may have

6 Authoring Subplot documents

• discuss that it may be necessary to have several documents for different audiences, at different levels of abstraction (cf. the FOSDEM safety devroom talk)
• discuss writing style to target all the different stakeholders
• discuss mechanics and syntax
  • Markdown and supported features
  • scenarios, embedded files, examples
  • bindings
  • step implementations in various languages
  • embedded markup for diagrams
  • running docgen

7 Extended example project

• discuss how to use Subplot for a significant project, but keep it sufficiently high level that it doesn't get too long and tedious to read

8 Appendix: Implementing scenario steps in Bash

• this appendix will explain how to implement scenario steps using the Bash shell

9 Appendix: Implementing scenario steps in Python

• this appendix will explain how to implement scenario steps using the Python language

10 Appendix: Implementing scenario steps in Rust

• this appendix will explain how to implement scenario steps using the Rust language

11 Appendix: Scenario

This is currently necessary so that codegen won't barf.