test

import test

A testing framework: suites, assertions, mocks, snapshots, lifecycle hooks, and reports worth reading.

import test { * }

describe('Calculator', @{

  it('adds', @{
    expect(2 + 2).to_be(4)
  })

  it('divides', @{
    expect(10 / 4).to_be_close_to(2.5)
  })

})

Run it like any other script:

$ zuri run tests/calculator.zu

Calculator
  ✓ adds
  ✓ divides

  PASS

  2 passed  •  2 total
  suites 1   time 1ms

Importing

import test { * } is the form to use in a test file. It brings describe, it, expect, mock, the hooks and run into scope, which is what makes a test read as a test rather than as a series of calls into a module.

import test is there for anything that is not a test file, and qualifies the same names: test.conduct('tests'), test.run(). The two can be combined when a test file wants both.

Declaring

describe(name, body) groups tests and may be nested as deeply as the subject calls for. it(name, body) declares one. Neither runs anything: they build a tree, and the tree is run once the file has finished declaring everything in it. That is what makes focusing, filtering and counting possible at all.

The run happens on its own, through os.at_exit(). Call run() explicitly when you want to pass options or read the Summary back; doing so takes over, and nothing runs a second time.

Focusing and skipping

describe_only('the one I am working on', @{ ... })
describe_skip('not now', @{ ... })

it_only('just this', @{ ... })
it_skip('known broken', @{ ... })
it_todo('handle an empty payload')

As soon as anything is marked only, everything not marked only and not inside something marked only is skipped. skip always wins over only.

it('name') with no body is a todo as well: it is reported as one and counts towards nothing.

Table-driven tests

it_each([
  [1, 1, 2],
  [2, 3, 5],
  [10, -4, 6],
], 'adds $0 and $1 to make $2', @(left, right, total) {
  expect(left + right).to_be(total)
})

Each row becomes one test. $0, $1 and so on in the name are replaced with that row’s values, and $# with the row number. describe_each() does the same for whole suites.

Per-test options

it('reaches the network', @{ ... }, { retries: 2, tags: ['slow'] })
it('reproduces issue 412', @{ ... }, { failing: true })
it('stays fast', @{ ... }, { timeout: 50 })

Hooks

describe('Database', @{
  before_all(@{ db.connect() })
  after_all(@{ db.close() })

  before_each(@{ db.begin() })
  after_each(@{ db.rollback() })

  it('inserts a row', @{ ... })
})

before_all runs once, immediately before the first test in its suite that is actually going to run, and not at all for a suite everything was filtered out of. after_all runs after the last one, and runs even when before_all failed, so it must cope with a setup that got only part of the way. before_each runs outermost first and after_each innermost first, and after_each runs even when the test failed.

Asserting

expect(value) gives an object carrying every matcher, each of which negates through .not and returns itself so several read as one:

expect(port).to_be_int().to_be_between(1024, 65535)
expect(items).not.to_be_empty()

The full list is in test.expect. In short:

• equality: to_be, to_equal, to_be_same_as, to_be_close_to, to_be_within, to_match_object
• truthiness: to_be_true, to_be_false, to_be_truthy, to_be_falsy, to_be_nil, to_be_defined
• types: to_be_a, to_be_string, to_be_number, to_be_int, to_be_float, to_be_bigint, to_be_bool, to_be_list, to_be_dict, to_be_bytes, to_be_function, to_be_callable, to_be_iterable, to_be_class, to_be_instance, to_be_instance_of
• numbers: to_be_greater_than, to_be_greater_than_or_equal, to_be_less_than, to_be_less_than_or_equal, to_be_between, to_be_positive, to_be_negative, to_be_zero, to_be_divisible_by, to_be_even, to_be_odd, to_be_nan, to_be_finite, to_be_infinite
• text: to_contain, to_contain_ignoring_case, to_equal_ignoring_case, to_start_with, to_end_with, to_match, to_be_blank
• collections: to_have_length, to_be_empty, to_contain_equal, to_contain_all, to_contain_any, to_contain_none, to_contain_exactly, to_have_key, to_have_keys, to_have_value, to_have_property, to_be_sorted, to_have_unique_items
• errors: to_raise, to_raise_instance_of, to_raise_with_message, to_not_raise
• mocks: to_have_been_called, to_have_been_called_times, to_have_been_called_with, to_have_been_last_called_with, to_have_been_nth_called_with, to_have_returned, to_have_returned_with, to_have_raised
• output: to_print, to_print_exactly, to_print_nothing
• snapshots: to_match_snapshot
• anything else: to_satisfy

fail(message) fails outright, for the branch that should never be reached. assertions(n) and has_assertions() fail a test that did not assert what it said it would, which is how a test proves that the callback it put its assertions in actually ran.

Mocking

var send = mock()
notify(send.fn)

expect(send).to_have_been_called_times(1)
expect(send).to_have_been_called_with('hello')

spy_on(object, 'key') replaces a function in place and restores it after the test, whether the test passed or not. See test.mock for what can and cannot be spied on.

Snapshots

expect(render(invoice)).to_match_snapshot()

The first run records the value beside the test file and passes; later runs compare against it. See test.snapshot for the file format, how to update them, and why a brand new snapshot fails on CI.

Running

run() takes an options dictionary:

filter and reporter can also come from ZURI_TEST_FILTER and ZURI_TEST_REPORTER, which is what lets one command be pointed at one test without editing the file.

run() returns a Summary; summary.exit_code() is 0 when everything passed. A file that never calls run() gets the same status: the automatic run sets it through os.set_exit_code() when anything failed, so CI needs nothing added.

More than one file

conduct(directory) finds every test file under a directory, runs each in a process of its own, and reports them together. A file that hangs or crashes takes nothing else down with it. See test.conduct.

zuri test is that call behind a command line, and is all a project needs to run its tests directory:

$ zuri test
$ zuri test pricing      # just tests/pricing.zu

Call conduct() yourself when the run wants to be under the project’s own control:

# tests/index.zu
import os
import test

test.conduct(os.dir_name(__file__))

The test API

Every public name in test, wherever it is declared. Each links to the page that documents it.

Submodules

Functions

describe()

test.describe(name: string, body: function) -> Suite

Groups tests, and may be nested.

describe('Stack', @{
  describe('push()', @{
    it('grows the stack', @{ ... })
  })
})

Parameters

• name (string)
• body (function)

Returns Suite

describe_only()

test.describe_only(name: string, body: function) -> Suite

Runs only this suite, and anything else marked only.

Parameters

• name (string)
• body (function)

Returns Suite

describe_skip()

test.describe_skip(name: string, body: function) -> Suite

Skips this suite. Its tests are still declared, still counted, and still listed; they simply do not run.

Parameters

• name (string)
• body (function)

Returns Suite

it()

test.it(name: string, body: ?function, options: ?dict) -> Case

Declares one test.

it('returns the total', @{
  expect(total(cart)).to_be(42)
})

Parameters

• name (string)
• body (?function) — left out, this is a todo.
• options (?dict) — retries, failing, timeout, tags. See the module documentation for what each does.

Returns Case

it_only()

test.it_only(name: string, body: function, options: ?dict) -> Case

Runs only this test, and anything else marked only.

Parameters

• name (string)
• body (function)
• options (?dict)

Returns Case

it_skip()

test.it_skip(name: string, body: ?function, options: ?dict) -> Case

Skips this test, while still listing it in the report.

Parameters

• name (string)
• body (?function)
• options (?dict)

Returns Case

it_todo()

test.it_todo(name: string) -> Case

Notes a test that has not been written yet.

It is listed in the report and counts towards nothing, which is the point: a todo is a reminder that survives being committed.

Parameters

• name (string)

Returns Case

it_each()

test.it_each(table: list, name: string, body: function, options: ?dict)

Declares one test per row of a table.

it_each([
  ['', 0],
  ['a', 1],
  ['hello', 5],
], 'says $0 is $1 characters long', @(text, length) {
  expect(text.length()).to_be(length)
})

Each row is spread across body’s parameters, so a three-column table calls a three-parameter function. A row that is not a list counts as a single-column row.

Parameters

• table (list) — the rows.
• name (string) — $0, $1, … are replaced with that row’s values, and $# with the row number, counting from one.
• body (function)
• options (?dict) — applied to every test the table declares.

Returns — list: the declared tests.

describe_each()

test.describe_each(table: list, name: string, body: function)

Declares one suite per row of a table.

The row is spread across body’s parameters exactly as it_each() does, so the whole suite is written once against whatever the row holds.

describe_each([
  ['json', json_codec],
  ['yaml', yaml_codec],
], '$0 codec', @(name, codec) {
  it('round-trips a dictionary', @{
    expect(codec.decode(codec.encode({ a: 1 }))).to_equal({ a: 1 })
  })
})

Parameters

• table (list)
• name (string)
• body (function)

Returns — list: the declared suites.

before_all()

test.before_all(body: function)

Runs once before the first test in the enclosing suite that is going to run.

Parameters

• body (function)

after_all()

test.after_all(body: function)

Runs once after the last test in the enclosing suite, and only if before_all ran. It runs when before_all raised as well, since a setup that failed part way may already hold what needs releasing, so it must cope with whatever the setup got as far as.

Parameters

• body (function)

before_each()

test.before_each(body: function)

Runs before every test in the enclosing suite and everything nested inside it, outermost suite first.

Parameters

• body (function)

after_each()

test.after_each(body: function)

Runs after every test in the enclosing suite and everything nested inside it, innermost suite first, whether the test passed or not.

Parameters

• body (function)

run()

test.run(options: ?dict) -> Summary

Runs everything declared so far, and reports it.

Calling this is optional. A file that declares tests and never says anything else runs them as it ends, and fails the process when they fail. Call it when you want to pass options or read the result:

run({ reporter: 'dot', bail: 1 })

var summary = run({ filter: 'parser' })
echo summary.passed

Parameters

• options (?dict) — see the module documentation for every option and its default.

Returns Summary

Note: calling it takes over from the automatic run, so nothing is reported twice. Calling it twice does run everything twice.

conduct()

test.conduct(directory: ?string, options: ?dict)

Discovers test files under directory and runs each in a process of its own.

Parameters

• directory (?string) — 'tests' by default.
• options (?dict) — see {test.conduct}.

Returns — dict: the combined result.

declared()

test.declared() -> Suite

The tests declared so far, as a tree, without running any of them.

Returns Suite

reset()

test.reset()

Forgets every declaration and every loaded snapshot file.

For a process that runs more than one suite, and for the tests of this framework itself.

use_color()

test.use_color(on: bool)

Forces colour in the report on or off, overriding what the environment said.

Parameters

• on (bool)

2021, Richard Ore and Zuri contributors