plunit.pl -- Unit Testing

Unit testing environment for SWI-Prolog and SICStus Prolog. For usage, please visit https://www.swi-prolog.org/pldoc/package/plunit.

set_test_options(+Options)

Specifies how to deal with test suites. Defined options are:

load(+Load): Whether or not the tests must be loaded. Values are never, always, normal (only if not optimised)
run(+When): When the tests are run. Values are manual, make or make(all).
format(+Mode): Currently one of tty or log. tty uses terminal control to overwrite successful tests, allowing the user to see the currently running tests and output from failed tests. This is the default of the output is a tty. log prints a full log of the executed tests and their result and is intended for non-interactive usage.
output(+When): If always, emit all output as it is produced, if never, suppress all output and if on_failure, emit the output if the test fails.
show_blocked(+Bool): Show individual blocked tests during the report.
occurs_check(+Mode): Defines the default for the occurs_check flag during testing.
cleanup(+Bool): If true (default =false), cleanup report at the end of run_tests/1. Used to improve cooperation with memory debuggers such as dmalloc.
jobs(Num): Number of jobs to use for concurrent testing. Default is one, implying sequential testing.
timeout(+Seconds): Set timeout for each individual test. This acts as a default that may be overuled at the level of units or individual tests. A timeout of 0 or negative is handled as inifinite.

loading_tests[private]

True if tests must be loaded.

begin_tests(+UnitName:atom) is det

begin_tests(+UnitName:atom, Options) is det

Start a test-unit. UnitName is the name of the test set. the unit is ended by :- end_tests(UnitName).

end_tests(+Name) is det

Close a unit-test module.

To be done: - Run tests/clean module?; - End of file?

make_unit_module(+Name, -ModuleName) is det[private]

unit_module(+Name, -ModuleName) is det[private]

expand_test(+Name, +Options, +Body, -Clause) is det[private]

Expand test(Name, Options) :- Body into a clause for 'unit test'/4 and 'unit body'/2.

expand(+Term, -Clauses) is semidet[private]

valid_options(:Pred, +Options) is det[private]

Verify Options to be a list of valid options according to Pred.

Errors: - type_error or instantiation_error.

test_option(+Option) is semidet[private]

True if Option is a valid option for test(Name, Options).

test_option(+Option) is semidet[private]

True if Option is a valid option for :-

begin_tests(Name,
Options)

reify_tmo(:Goal, -Result, +Options) is det[private]

reify(:Goal, -Result) is det[private]

Call Goal and unify Result with one of true, false or throw(E).

capture_output(:Goal, -Output) is semidet[private]

capture_output(:Goal, -Output, +Options) is semidet[private]

Arguments:

Output - is a pair Msgs-String, where Msgs is a boolean that is true if there were messages that require a non-zero exit status and Output contains the output as a string.

got_messages(:Goal, -Result)[private]

run_tests is semidet

run_tests(+TestSet) is semidet

run_tests(+TestSet, +Options) is semidet

Run tests and report about the results. The predicate run_tests/0 runs all known tests that are not blocked. The predicate run_tests/1 takes a specification of tests to run.

The predicate run_tests/2 is synchronized. Concurrent testing may be achieved using the relevant options. See set_test_options/1. Options are passed to set_test_options/1. In addition the following options are processed:

summary(-Summary)

Unify Summary do a dict holding the keys below. The value of these keys is an integer describing the number of tests. If this option is given, run_tests/2 does not fail if some tests failed.

total
passed
failed
timeout
blocked

Arguments:

TestSet - is either a single specification or a list of specifications. Each single specification is either the name of a test-unit or a term <test-unit>:<test>, denoting a single test within a unit. If TestSet is all, all known tests are executed.

report_and_cleanup(+Ref, +Time, +Options)[private]

Undo changes to the environment (trapping assertions), report the results and cleanup.

run_units_and_check_errors(+Units, +Options) is semidet[private]

Run all test units and succeed if all tests passed.

runnable_tests(+Spec, -Plan) is det[private]

Change a Unit+Test spec into a plain Unit:Tests lists, where blocked tests or tests whose condition fails are already removed. Each test in Tests is a term @(Test,Line), which serves as a unique identifier of the test.

count_tests(+Units0, -Units, -Count) is det[private]

Count the number of tests to run. A forall(Generator, Test) counts as a single test. During the execution, the concrete tests of the forall are considered "sub tests".

run_unit(+Unit) is det[private]

Run a single test unit. Unit is a term Unit:Tests, where Tests is a list of tests to run.

run_tests_in_files(+Files:list) is det[private]

Run all test-units that appear in the given Files.

make_run_tests(+Files)[private]

Called indirectly from make/0 after Files have been reloaded.

run_test(+Unit, +Test) is det[private]

Run a single test.

run_test(+Unit, +Name, +Line, +UnitOptions, +Options, +Body)[private]

Deals with forall(Generator, Test)

run_test_once6(+Unit, +Name, +Progress, +Line, +UnitOptions, +Options, +Body)[private]

Inherit the timeout and occurs_check option (Global -> Unit -> Test).

run_test_once(+Unit, +Name, Progress, +Line, +Options, +Body)[private]

Deal with occurs_check, i.e., running the test multiple times with different unification settings wrt. the occurs check.

report_result(+Result, +Progress, +Output, +Options) is det[private]

run_test_6(+Unit, +Name, +Line, +Options, :Body, -Result) is det[private]

6th step of the tests. Deals with tests that must be ignored (blocked, conditions fails), setup and cleanup at the test level. Result is one of:

failure(Unit, Name, Line, How, Time)

How is one of:

succeeded
Exception
time_limit_exceeded(Limit)
cmp_error(Cmp, E)
wrong_answer(Cmp)
failed
no_exception
wrong_error(Expect, E)
wrong_answer(Expected, Bindings)

success(Unit, Name, Line, Determinism, Time)

setup_failed(Unit, Name, Line)

run_test_7(+Unit, +Name, +Line, +Options, :Body, -Result) is det[private]

This step deals with the expected outcome of the test. It runs the actual test and then compares the result to the outcome. There are two main categories: dealing with a single result and all results.

non_det_test(+Expected, +Unit, +Name, +Line, +Options, +Body, -Result)[private]

Run tests on non-deterministic predicates.

result_vars(+Expected, -Vars) is det[private]

Create a term v(V1, ...) containing all variables at the left side of the comparison operator on Expected.

nondet_compare(+Expected, +Bindings, +Unit, +Name, +Line) is semidet[private]

Compare list/set results for non-deterministic predicates.

bug: - Sort should deal with equivalence on the comparison operator.
To be done: - Properly report errors

cmp(+CmpTerm, -Left, -Op, -Right) is det[private]

call_det(:Goal, -Det) is nondet[private]

True if Goal succeeded. Det is unified to true if Goal left no choicepoints and false otherwise.

match_error(+Expected, +Received) is semidet[private]

True if the Received errors matches the expected error. Matching is based on subsumes_term/2.

setup(+Module, +Context, +Options) is semidet[private]

Call the setup handler and fail if it cannot run for some reason. The condition handler is similar, but failing is not considered an error. Context is one of

unit(Unit): If it is the setup handler for a unit
test(Unit, Name, Line): If it is the setup handler for a test

condition(+Module, +Context, +Options) is semidet[private]

Evaluate the test or test unit condition.

call_ex(+Module, +Goal)[private]

Call Goal in Module after applying goal expansion.

cleanup(+Module, +Options) is det[private]

Call the cleanup handler and succeed. Failure or error of the cleanup handler is reported, but tests continue normally.

failure(+Unit, +Name, +Progress, +Line, +How, +Time, +Output, +Options) is det[private]

Test failed. Report the error.

assert_cyclic(+Term) is det[private]

Assert a possibly cyclic unit clause. Current SWI-Prolog assert/1 does not handle cyclic terms, so we emulate this using the recorded database.

To be done: - Implement cycle-safe assert and remove this.

setup_jobs(+Count) is det[private]

Setup threads for concurrent testing.

job_wait(?Unit) is det[private]

Wait for all test jobs to finish.

begin_test(+Unit, +Test, +Line, +Progress) is det[private]

end_test(+Unit, +Test, +Line, +Progress) is det[private]

Maintain running/5 and report a test has started/is ended using a silent message:

plunit(begin(Unit:Test, File:Line, Progress))
plunit(end(Unit:Test, File:Line, Progress))

See also: - message_hook/3 for intercepting these messages

running_tests is det

Print the currently running test.

current_test(?Unit, ?Test, ?Line, ?Body, ?Options) is nondet

True when a test with the specified properties is loaded.

current_test_unit(?Unit, ?Options) is nondet

True when a Unit is a current unit test declared with Options.

test_summary(?Unit, -Summary) is det[private]

True when Summary is a dict that reports the main statistics about the executed tests.

report(+Time, +Options) is det[private]

Print a summary of the tests that ran.

test_report(+What) is det

Produce reports on test results after the run. Currently only supports fixme for What.

unit_file(+Unit, -File) is det[private]

unit_file(?Unit, ?File) is nondet[private]

True when the test unit Unit is defined in File.

load_test_files(+Options) is det

Load .plt test-files related to loaded source-files. Options is currently ignored.

info(+Term)[private]

Runs print_message(Level, Term), where Level is one of silent or informational (default).

progress(+UnitTest, +Progress, +Result, +Time) is det[private]

Test Unit:Name completed in Time. Result is the result and is one of

passed
failed
assertion
nondet
fixme(passed)
fixme(nondet)
fixme(failed)
forall(end,Nth,FTotal): Pseudo result for completion of a forall(Gen,Test) set. Mapped to forall(FTotal, FFailed)

test_name_summary(+Term, +MaxLen, -Summary) is det[private]

Given the test id, generate string that summarizes this in MaxLen characters.

progress_string(+Progress, -S) is det[private]

True when S is a string representation for the test progress.

progress_tag(+Status, -Tag, -Keep, -Style) is det[private]

Given a progress status, determine the status tag, whether we must preserve the line and the Style we must use to print the status line.

jobs_redraw is det[private]

Redraw the job window.

job_format(+Style, +Fmt, +Args) is det[private]

job_format(+Job, +Style, +Fmt, +Args, +Save) is det[private]

Point should be below the status window. Format Fmt+Args in the line Job using Style and return to the position below the window.

job_finish(+Style, +Fmt, +Args) is det[private]

job_finish(+Job, +Style, +Fmt, +Args) is det[private]

Complete the status line for Job. This redraws the original status line when we are using a job window.

feedback is semidet[private]

provide feedback using the tty format, which reuses the current output line if the test is successful.

user:message_hook(+Severity, +Message, +Lines) is semidet[multifile]

Redefine printing some messages. It appears SICStus has no way to get multiple messages at the same line, so we roll our own. As there is a lot pre-wired and checked in the SICStus message handling we cannot reuse the lines. Unless I miss something ...

Re-exported predicates

The following predicates are exported from this file while their implementation is defined in imported modules or non-module files loaded by this module.

begin_tests(+UnitName:atom) is det

begin_tests(+UnitName:atom, Options) is det

Start a test-unit. UnitName is the name of the test set. the unit is ended by :- end_tests(UnitName).

run_tests is semidet

run_tests(+TestSet) is semidet

run_tests(+TestSet, +Options) is semidet

Run tests and report about the results. The predicate run_tests/0 runs all known tests that are not blocked. The predicate run_tests/1 takes a specification of tests to run.

summary(-Summary)

Unify Summary do a dict holding the keys below. The value of these keys is an integer describing the number of tests. If this option is given, run_tests/2 does not fail if some tests failed.

total
passed
failed
timeout
blocked

Arguments:

run_tests is semidet

run_tests(+TestSet) is semidet

run_tests(+TestSet, +Options) is semidet

Run tests and report about the results. The predicate run_tests/0 runs all known tests that are not blocked. The predicate run_tests/1 takes a specification of tests to run.

summary(-Summary)

Unify Summary do a dict holding the keys below. The value of these keys is an integer describing the number of tests. If this option is given, run_tests/2 does not fail if some tests failed.

total
passed
failed
timeout
blocked

Arguments: