XReq

XReq - Executable Requirements for DO-178B

XReq is a tool designed to help testing and verifying projects. It has been specifically adapted for the DO-178B context but can be used by a much wider audience. To help DO-178B projects, it brings together the tests (HLT/LLT) with their requirements, therefore helping traceability of the tests.

Tests are written in English (or any other natural language) with a very simple formalism to keep a structure. The text for the requirement appears first, then appears the text for the test cases. XReq transforms the test cases written in English to actual Ada/C/C++ code using a simple matching process with a set of definitions. This compiled test is then run against the project to actually test the project and to determine if the tests pass or fails.

XReq was inspired from the Behaviour Driven Development, a process used by the agile community and consisting in writing the requirements and test cases before actually digging in the project. These executable requirements are then used all along the process of implementing the feature to test its validity, and later, to test regressions. By providing XReq, we hope to bring more robust methods in critical or embedded projects.

Executable Requirements

The executable requirements are the files you write. They can either be written with the feature vocabulary, or the requirement vocabulary (more suited for DO-178B projects). It is a text file containing English sentences. The file itself has two parts:

  • The description of the requirement (or feature), as specific as possible. It must be an exhaustive desription of the requirement. The structure is not important to XReq, you can choose whatever formalism you prefer.

  • Test cases (or scenarios), each describing step by step a test case for the requirement. This is the part of the requirement that XReq will compile and that will be executed.

Test cases are composed of various steps that will be executed in order. There are three kinds of steps, each introduced by a different keyword. A special keyword And has a special meaning. It is used as a link-word to repeat the keyword used on the previous step.

  • Given steps are used to initialize the system. Set the preconditions.

  • When steps are actions applied to the system. The system state is going to change from the initial state to a defined state.

  • Then steps tests the state of the system after executing actions. It check that the system is in the expected state and generally errors are detected at this point.

Let's look at an example of a simple calculator, the requirement is in a file called calculate.requirement:

Requirement: Calculate
  In order to do simple calculations
  As a user
  I want to be able to use simple operators (add, subtract, multiply and divide)

  Test Case: Add two numbers
    Given the calculator has just been powered on
     When I press the button "1"
      And I press the button "+"
      And I press the button "2"
      And I press the button "="
     Then I should see "3" on the screen

  Test Case: Subtract two numbers
    Given the calculator has just been powered on
     When I press the button "1"
      And I press the button "-"
      And I press the button "2"
      And I press the button "="
     Then I should see "-1" on the screen

Step Definitions

In order to allow the computer to run each of these steps, you need to write step definitions. You can also use existing sets of step definitions. XReq will then match each step with the appropriate step definition using regular expression matching.

Step definitions are simple procedures with a common signature gathered in a package. Regular expressions can be associated to procedures with a special formalism that is included in comments.

The step definitions for the previous requirement can be expressed in a package step_definitions/calculator_steps.ads:

with XReqLib.General; use XReqLib.General;

package Calculator_Steps is

   --  @given ^the calculator has just been powered on$
   procedure Given_the_calculator_has_just_been_powered_on (Args : in out Arg_Type);

   --  @when  ^I press the button "([^"]*)"$
   procedure When_I_press_the_button (Args : in out Arg_Type);

   --  @then  ^I should see "([^"]*)" on the screen$
   procedure Then_I_should_see (Args : in out Arg_Type);

end Calculator_Steps;

The special argument Arg_Type contains matches from the regular expression, and additional parameters that are included in the requirement.

XReq Invocation

For Ada packages, XReq can generate a stub of the step definition package (or add step definitions to an existing package). To do so, just XReq with:

xreq --fill-steps-in Calculator_Steps calculate.requirement

When step definitions are provided, XReq can compile the requirement in an Ada package (in the tests directory) with the following command line:

xreq calculate.requirement

It can additionally generate a test suite, and a GPR project file to compile it using:

xreq --executable test_suite calculate.requirement

And to force XReq to build the test suite, you can add the option --make:

xreq --make --executable test_suite calculate.requirement

The options of XReq can be obtained with the --help switch. See the help page. The generated test suite can also have different options.

Run the test suite

When running the test suite, each step in each test case will be executed in order and errors reported. Steps that do not report an error are green, failed steps are red and skipped steps (because a previous step failed) are in blue.

The test suite can generate a colored report on the command line, or a HTML report in a file. You can choose the report type with the --format option and its destination with the --output option.

For instance, let's say we haven't implemented yet our step definitions, if we execute our test suite, we are going to see:

$ xreq -m -x test_suite calculate.requirement
$ tests/test_suite

Requirement: Calculate
  In order to do simple calculations
  As a user
  I want to be able to use simple operators (add, subtract, multiply and divide)

  Test Case: Add two numbers
    Given the calculator has just been powered on
      STEPS.GIVEN_THE_CALCULATOR_HAS_JUST_BEEN_POWERED_ON.NOT_YET_IMPLEMENTED:
      Procedure Given_the_calculator_has_just_been_powered_on not implemented
    When I press the button "1"
    And I press the button "+"
    And I press the button "2"
    And I press the button "="
    Then I should see "3" on the screen

  Test Case: Subtract two numbers
    Given the calculator has just been powered on
      STEPS.GIVEN_THE_CALCULATOR_HAS_JUST_BEEN_POWERED_ON.NOT_YET_IMPLEMENTED:
      Procedure Given_the_calculator_has_just_been_powered_on not implemented
    When I press the button "1"
    And I press the button "-"
    And I press the button "2"
    And I press the button "="
    Then I should see "-1" on the screen

2 scenario (2 failed)
12 steps (2 failed, 10 skipped)
Finished in 0s