Testing statecharts

Like any executable software artefacts, statecharts can and should be tested during their development.

One possible appproach is to test the execution of a statechart by hand. The Sismic interpreter stores and returns several values that can be inspected during the execution, including the active configuration, the list of entered or exited states, etc. The functional tests in tests/test_interpreter.py on the GitHub repository are several examples of this kind of tests.

This way of testing statecharts is, however, quite cumbersome, especially if one would also like to test a statechart’s contracts (i.e., its invariants and behavioral pre- and postconditions).

To overcome this, Sismic provides a module sismic.testing that makes it easy to test statecharts using statecharts themselves!

Using statecharts to encode (un)desirable properties

Remark: in the following, the term statechart under test refers to the statechart that is to be tested, while the term property statechart refers to a statechart that expresses conditions or invariants that should be satisfied by the statechart under test.

While contracts can be used to verify assertions on the context of a statechart during its execution, property statecharts can be used to test specific behavior of a statechart under test.

A property statechart defines a property that should (or not) be satisfied by other statecharts. A property statechart is like any other statechart, in the sense that neither their syntax nor their semantics differs from any other statechart. The difference comes from the events it receives and the role it plays. If the run of a statechart property ends in a final state, it signifies that the property was verified. In the case of a desirable property, this means that the test succeed. In the case of an undesirable property, this means that the test failed.

Note

This is more a convention than a requirement, but you should follow it.

The run of such a property statechart is driven by a specific sequence of events and pauses, which represents what happens during the execution of a statechart under test.

For example, such a sequence contains event consumed events, state entered events, state exited events, ... In particular, the following events are generated:

  • A execution started event is sent at the beginning.
  • each time a step begins, a step started event is created.
  • each time an event is consumed, a event consumed event is created. the consumed event is available through the event attribute.
  • each time a state is exited, an state exited event is created. the name of the state is available through the state attribute.
  • each time a transition is processed, a transition processed event is created. the source state name and the target state name (if any) are available respectively through the source and target attributes. The event processed by the transition is available through the event attribute.
  • each time a state is entered, an state entered event is created. the name of the state is available through the state attribute.
  • each time a step ends, a step ended event is created.
  • A execution stopped event is sent at the end.
  • each time an event is fired from within the statechart, a sent event is created. the sent event is available through the event attribute.

The sequence does follow the interpretation order:

  1. an event is possibly consumed
  2. For each matching transition
    1. states are exited
    2. transition is processed
    3. states are entered
    4. internal events are sent
    5. statechart is stabilized (some states are exited and/or entered, some events are sent)

Using statechart to check properties on a trace

The trace of an interpreter is the list of its executed macro steps. The trace can be built upon the values returned by each call to execute() (or execute_once()), or can be automatically built using sismic.interpreter.helpers.log_trace() function.

Function teststory_from_trace() provides an easy way to construct a story for statechart properties from the trace obtained by executing a statechart under test.

sismic.testing.teststory_from_trace(trace)

Return a test story based on the given trace, a list of macro steps. See documentation to see which are the events that are generated.

Notice that this function adds a pause if there is any delay between pairs of consecutive steps.

Parameters:trace (List[MacroStep]) – a list of MacroStep instances
Return type:Story
Returns:A story

Notice that using this function, the property statechart can not access the context of the statechart under test.

To summarize, if you want to test the trace of a statechart under test tested, you need to:

  1. construct a property statechart tester that expresses the property you want to test.
  2. execute tested (using a story or directly by sending events) and log its trace.
  3. generate a new story from this trace with teststory_from_trace().
  4. tell this story to an interpreter of the property statechart tester.

If tester ends in a final configuration, ie. tester.final holds, then the test is considered successful. The semantic of successful depends on the desirability of the checked property.

The following property statechart examples are relative to this statechart. They show the specification of some testers in YAML, and how to execute them.

Note that these statechart properties are currently used as unit tests for Sismic.

7th floor is never reached

This property statechart ensures that the 7th floor is never reached. It stores the current floor based on the number of times the elevator goes up and goes down.

statechart:
  name: Test that the elevator never reachs 7th floor
  preamble: floor = 0
  root state:
    name: root
    initial: standing
    states:
      - name: standing
        transitions:
          - event: state entered
            guard: event.state == 'moving'
            target: moving
          - guard: floor == 7
            target: fail
      - name: moving
        transitions:
          - event: state entered
            guard: event.state == 'movingUp'
            action: floor += 1
          - event: state entered
            guard: event.state == 'movingDown'
            action: floor -= 1
          - event: state exited
            guard: event.state == 'moving'
            target: standing
      - name: fail
        type: final

It can be tested as follows:

    def test_7th_floor_never_reached(self):
        story = Story([Event('floorSelected', floor=8)])
        trace = story.tell(self.tested)  # self.tested is an interpreter for our elevator

        test_story = teststory_from_trace(trace)

        with open('docs/examples/elevator/tester_elevator_7th_floor_never_reached.yaml') as f:
            tester = Interpreter(io.import_from_yaml(f))
        test_story.tell(tester)
        self.assertFalse(tester.final)

You can even simulate a failure:

    def test_7th_floor_never_reached_fails(self):
        story = Story([Event('floorSelected', floor=4), Pause(2), Event('floorSelected', floor=7)])
        trace = story.tell(self.tested)  # self.tested is an interpreter for our elevator

        test_story = teststory_from_trace(trace)

        with open('docs/examples/elevator/tester_elevator_7th_floor_never_reached.yaml') as f:
            tester = Interpreter(io.import_from_yaml(f))
        test_story.tell(tester)
        self.assertTrue(tester.final)

Elevator moves after 10 seconds

This property statechart checks that the elevator automatically moves after some idle time if it is not on the ground floor. The test sets a timeout of 12 seconds, but it should work for any number strictly greater than 10 seconds.

statechart:
  name: Test that the elevator goes to ground floor after 10 seconds (timeout set to 12 seconds)
  preamble: floor = 0
  root state:
    name: root
    initial: active
    states:
      - name: active
        parallel states:
          - name: guess floor
            transitions:
              - event: state entered
                guard: event.state == 'movingUp'
                action: floor += 1
              - event: state entered
                guard: event.state == 'movingDown'
                action: floor -= 1
          - name: check timeout
            initial: standing
            states:
              - name: standing
                transitions:
                  - event: state entered
                    guard: event.state == 'moving'
                    target: moving
                  - guard: after(12) and floor != 0
                    target: timeout
              - name: moving
                transitions:
                  - event: state exited
                    guard: event.state == 'moving'
                    target: standing
              - name: timeout
                type: final

We check this tester using several stories, as follows:

    def test_elevator_moves_after_10s(self):
        stories = [
            Story([Event('floorSelected', floor=4)]),
            Story([Event('floorSelected', floor=0)]),
            Story([Event('floorSelected', floor=4), Pause(10)]),
            Story([Event('floorSelected', floor=0), Pause(10)]),
            Story([Event('floorSelected', floor=4), Pause(9)]),
            Story([Event('floorSelected', floor=0), Pause(9)]),
        ]

        for story in stories:
            with self.subTest(story=story):
                # Reopen because we need to reset it
                with open('docs/examples/elevator/elevator.yaml') as f:
                    sc = io.import_from_yaml(f)
                tested = Interpreter(sc)

                test_story = teststory_from_trace(story.tell(tested))

                with open('docs/examples/elevator/tester_elevator_moves_after_10s.yaml') as f:
                    tester = Interpreter(io.import_from_yaml(f))
                test_story.tell(tester)
                self.assertFalse(tester.final)

Using statecharts to check properties at runtime

Sismic provides a convenience class to allow property statechart to check properties at runtime. Class ExecutionWatcher can be used to associate a statechart tester with a statechart under test:

class sismic.testing.ExecutionWatcher(tested_interpreter)

This can be used to associate a property statechart with a statechart under test. An instance of this class is built upon an Interpreter instance (the tested one).

It provides a method, namely watch_with which takes a property statechart (and a set of optional parameters that can be used to tune the interpreter that will be built upon this property statechart) and returns the resulting Interpreter instance for this tester.

If started (using start), whenever something happens during the execution of the interpreter under test, events are automatically sent to every associated statechart properties. Their internal clock are synchronized, and the context of the statechart under test is also exposed to the property statechart, ie. if x is a variable in the context of a statechart under test, then context.x is dynamically exposed to every associated property statechart.

Parameters:tested_interpreter (Interpreter) – Interpreter to watch
start()

Send a started event to the statechart properties, and starts watching the execution of the statechart under test.

Return type:None
stop()

Send a stopped event to the statechart properties, and stops watching the execution of the statechart under test.

Return type:None
watch_with(property_statechart, fails_fast=False, interpreter_klass=<class 'sismic.interpreter.interpreter.Interpreter'>, **kwargs)

Watch the execution of the tested interpreter with given sproperty statechart.

interpreter_klass is a callable that accepts a Statechart instance, an initial_context parameter and any additional parameters provided to this method. This callable must return an Interpreter instance

Parameters:
  • property_statechart (Statechart) – a property statechart (instance of Statechart)
  • fails_fast (bool) – If True (default is False), the execution of the statechart under test will raise an AssertionError as soon as given property statechart reaches a final state.
  • interpreter_klass (Callable[..., Interpreter]) – a callable that accepts a Statechart instance, an initial_context and any additional (optional) parameters provided to this method.
Return type:

Interpreter

Returns:

the interpreter instance that wraps given property statechart.

To summarize, if you want to test (at runtime) the execution of a statechart under test tested, you need to:

  1. create an ExecutionWatcher with tested.
  2. construct at least one property statechart tester that expresses the property you want to test.
  3. associate each tester to the watcher with watch_with().
  4. start watching with start().
  5. execute tested (using a story or directly by sending events).
  6. stop watching with stop().

If tester ends in a final configuration, ie. tester.final holds, then the test is considered successful. Again, the semantic of a successful run depends on the desirability of the property.

Destination should be reached

This property statechart ensures that every chosen destination is finally reached.

statechart:
  name: Test that destinations are reached
  preamble: |
    destinations = []  # List of destinations
  root state:
    name: root
    initial: check
    states:
      - name: check
        transitions:
          - event: execution stopped
            guard: len(destinations) > 0
            target: fail
        parallel states:
          - name: wait floor selection
            transitions:
              - event: event consumed
                guard: event.event.name == 'floorSelected'
                action: destinations.append(event.event.floor)
          - name: wait doors open
            transitions:
              - event: state entered
                guard: event.state == 'doorsOpen'
                action: |
                  # Current floor, deduced from tested statechart's context
                  floor = context.current

                  # Remove floor from destination if it exists
                  try:
                    destinations.remove(floor)
                  except ValueError:
                    pass
      - name: fail
        type: final

It can be tested as follows:

from sismic.io import import_from_yaml
from sismic.interpreter import Interpreter
from sismic.testing import ExecutionWatcher
from sismic.model import Event

# Load statecharts
with open('examples/elevator/elevator.yaml') as f:
    elevator_statechart = import_from_yaml(f)
with open('examples/elevator/tester_elevator_destination_reached.yaml') as f:
    tester_statechart = import_from_yaml(f)

# Create the interpreter and the watcher
interpreter = Interpreter(elevator_statechart)
watcher = ExecutionWatcher(interpreter)

# Add the tester and start watching
tester = watcher.watch_with(tester_statechart)
watcher.start()

# Send the elevator to 4th
interpreter.queue(Event('floorSelected', floor=4)).execute(max_steps=2)
assert tester.context['destinations'] == [4]

interpreter.execute()
assert tester.context['destinations'] == []

# Stop watching. The statechart ends in a final state only if a failure occured
watcher.stop()

assert not tester.final