Expect-test is a framework for writing tests in OCaml, similar to Cram.
Expect-tests mimic the (now less idiomatic) inline test framework in providing a let%expect_test construct.
The body of an expect-test can contain output-generating code, interleaved with [%expect] extension expressions to denote the expected output.
When run, expect-tests pass iff the output matches the expected output. If a test fails, the inline_tests_runner outputs a diff and creates a file with the suffix ".corrected" containing the actual output.
Here is an example expect-test in foo.ml:
open! Core
let%expect_test "addition" =
printf "%d" (1 + 2);
[%expect {| 4 |}]
;;
When the test runs, the inline_tests_runner creates foo.ml.corrected with contents:
open! Core
let%expect_test "addition" =
printf "%d" (1 + 2);
[%expect {| 3 |}]
;;
inline_tests_runner also outputs:
------ foo.ml
++++++ foo.ml.corrected
File "foo.ml", line 6, characters 0-1:
|open! Core
|
|let%expect_test "addition" =
| printf "%d" (1 + 2);
-| [%expect {| 4 |}]
+| [%expect {| 3 |}]
|;;
|
Diffs are shown in color if the -use-color flag is passed to the inline test runner executable.
Each [%expect] block matches all the output generated since the previous [%expect] block (or the beginning of the test). In this way, when multiple [%expect] blocks are interleaved with test code, they can help show which part of the test produced which output.
The following test:
let%expect_test "interleaved" =
let l = [ "a"; "b"; "c" ] in
printf "A list [l]\n";
printf "It has length %d\n" (List.length l);
[%expect {| A list [l] |}];
List.iter l ~f:print_string;
[%expect
{|
It has length 3
abc
|}]
;;
is rewritten as
let%expect_test "interleaved" =
let l = [ "a"; "b"; "c" ] in
printf "A list [l]\n";
printf "It has length %d\n" (List.length l);
[%expect
{|
A list [l]
It has length 3
|}];
List.iter l ~f:print_string;
[%expect {| abc |}]
;;
When there is "trailing" output at the end of a let%expect_test (output that has yet to be matched by some [%expect] block), a new [%expect] block is appended to the test with the trailing output:
let%expect_test "trailing output" =
print_endline "Hello";
[%expect {| Hello |}];
print_endline "world"
;;
becomes:
let%expect_test "trailing output" =
print_endline "Hello";
[%expect {| Hello |}];
print_endline "world";
[%expect {| world |}]
;;
You might have noticed that the contents of the [%expect] blocks are not exactly the program output; in some of the examples above, they contain a different number of leading and trailing newlines, and are indented to match the code indentation. We say the contents of a block [%expect str] (where str is some string literal) match the output at that block if the output, after we format it to standardize indentation and other whitespace, is identical to the contents of str after it has been similarly formatted .
The formatting applied depends on the type of delimiter used in str (i.e. whether it a "quoted string" or a {xxx| delimited string |xxx}). To summarize:
[%expect {| |}] or [%expect ""].[%expect {| output |}] or [%expect "output"].{| delimited strings |}, the least-indented line with content (the "left margin" of the output) is aligned to be two spaces past the indentation of the [%expect] block."quoted string", the least-indented line is indented by exactly one space (this plays the nicest with ocamlformat's existing decisions about how to format string literals).Here is an example containing several cases of output that are subject to distinct formatting rules and how they appear in [%expect] and [%expect_exact] blocks:
let%expect_test "matching behavior --- no content" =
printf " ";
[%expect {| |}];
printf " ";
[%expect ""];
printf " ";
[%expect_exact {| |}];
printf " ";
[%expect_exact " "]
;;
let%expect_test "matching behavior --- one line of content" =
printf "\n This is one line\n\n";
[%expect {| This is one line |}];
printf "\n This is one line\n\n";
[%expect "This is one line"];
printf "\n This is one line\n\n";
[%expect_exact
{|
This is one line
|}];
printf "\n This is one line\n\n";
[%expect_exact "\n This is one line\n\n"]
;;
let%expect_test "matching behavior --- multiple lines of content" =
printf
{|
Once upon a midnight dreary,
while I pondered, weak and weary,
Over many a quaint and curious
volume of forgotten lore |};
[%expect
{|
Once upon a midnight dreary,
while I pondered, weak and weary,
Over many a quaint and curious
volume of forgotten lore
|}];
printf
{|
Once upon a midnight dreary,
while I pondered, weak and weary,
Over many a quaint and curious
volume of forgotten lore |};
[%expect
" \n\
\ Once upon a midnight dreary,\n\
\ while I pondered, weak and weary,\n\
\ Over many a quaint and curious\n\
\ volume of forgotten lore\n\
\ "];
printf
{|
Once upon a midnight dreary,
while I pondered, weak and weary,
Over many a quaint and curious
volume of forgotten lore |};
[%expect_exact
{|
Once upon a midnight dreary,
while I pondered, weak and weary,
Over many a quaint and curious
volume of forgotten lore |}];
printf
{|
Once upon a midnight dreary,
while I pondered, weak and weary,
Over many a quaint and curious
volume of forgotten lore |};
[%expect_exact
"\n\
Once upon a midnight dreary,\n\
\ while I pondered, weak and weary,\n\
Over many a quaint and curious\n\
\ volume of forgotten lore "]
;;
Expect-test is by default permissive about this formatting, so that a [%expect] block that is correct modulo formatting is accepted. However, passing -expect-test-strict-indentation=true to the ppx driver makes the test runner issue corrections for blocks that do not satisfy the indentation rules. For example, the following:
let%expect_test "bad formatting" =
printf "a\n b";
[%expect
{|
a
b |}]
;;
is corrected to:
let%expect_test "bad formatting" =
printf "a\n b";
[%expect
{|
a
b
|}]
;;
(to add the required indentation and trailing newline)
Expects reached from multiple placesA [%expect] extension can be encountered multiple times if it is in e.g. a functor or a function:
let%expect_test "function" =
let f output =
print_string output;
[%expect {| hello world |}]
in
f "hello world";
f "hello world"
;;
The test passes if the [%expect] block matches the output each time it is encountered, as described in the section on matching behavior.
If the outputs are not consistent, then the corrected file contains a report of all of the outputs that were captured, in the order that they were captured at runtime.
For example, calling f in the snippet above with inconsistent arguments will produce:
let%expect_test "function" =
let f output =
print_string output;
[%expect
{|
(* expect_test: Test ran multiple times with different test outputs *)
============================ Output 1 / 4 ============================
hello world
============================ Output 2 / 4 ============================
goodbye world
============================ Output 3 / 4 ============================
once upon
a midnight dreary
============================ Output 4 / 4 ============================
hello world
|}]
in
f "hello world";
f "goodbye world";
f "once upon\na midnight dreary";
f "hello world"
;;
Every [%expect] and [%expect_exact] block in a let%expect_test must be reached at least once if that test is ever run. Failure for control flow to reach a block is not treated like recording empty output at a block. The extension expression [%expect.unreachable] is used to indicate that some part of an expect test shouldn't be reached; if control flow reaches that point anyway, the corrected file replaces the [%expect.unreachable] with a plain old expect containing the output collected until that point. Conversely, if control flow never reaches some [%expect] block, that block is turned into a [%expect.unreachable]. For example:
let%expect_test "unreachable" =
let interesting_bool = 3 > 5 in
printf "%b\n" interesting_bool;
if interesting_bool
then [%expect {| true |}]
else (
printf "don't reach\n";
[%expect.unreachable])
;;
becomes:
let%expect_test "unreachable" =
let interesting_bool = 3 > 5 in
printf "%b\n" interesting_bool;
if interesting_bool
then [%expect.unreachable]
else (
printf "don't reach\n";
[%expect
{|
false
don't reach
|}])
;;
Note that, for an expect block that is sometimes reachable and sometimes not, that block passes if the output captured at that block matches every time the block is encountered. For example, the following test passes:
module Test (B : sig
val interesting_opt : int option
end) =
struct
let%expect_test "sometimes reachable" =
match B.interesting_opt with
| Some x ->
printf "%d\n" x;
[%expect {| 5 |}]
| None -> [%expect {| |}]
;;
end
module _ = Test (struct
let interesting_opt = Some 5
end)
module _ = Test (struct
let interesting_opt = None
end)
module _ = Test (struct
let interesting_opt = Some 5
end)
When an exception is raised by the body of an expect-test, the inline_test_runner shows it (and, if relevant, any output generated by the test that had not yet been captured) in a [@@expect.uncaught_exn] attribute attached to the corresponding let%expect_test. [%expect] blocks in the test are treated according to the usual rules: those reached before the exception is raised capture output as usual, and those that "would have" been reached after are marked as unreachable:
let%expect_test "exception" =
Printexc.record_backtrace false;
printf "start!";
[%expect {| |}];
let sum = 2 + 2 in
if sum <> 3
then (
printf "%d" sum;
failwith "nope");
printf "done!";
[%expect {| done! |}]
;;
becomes:
let%expect_test "exception" =
Printexc.record_backtrace false;
printf "start!";
[%expect {| start! |}];
let sum = 2 + 2 in
if sum <> 3
then (
printf "%d" sum;
failwith "nope");
printf "done!";
[%expect.unreachable]
[@@expect.uncaught_exn
{|
(Failure nope)
Trailing output
---------------
4
|}]
;;
Unlike [%expect] blocks, which might be reached on some runs of a test and not others, a test with an [@@expect.uncaught_exn] attribute must raise every time it is run. Changing the None branch of the functorized test from before to raise gives:
module Test' (B : sig
val interesting_opt : int option
end) =
struct
let%expect_test "sometimes raises" =
match B.interesting_opt with
| Some x ->
printf "%d\n" x;
[%expect {| 5 |}]
| None -> failwith "got none!"
[@@expect.uncaught_exn
{|
(* expect_test: Test ran multiple times with different uncaught exceptions *)
=============================== Output 1 / 3 ================================
<expect test ran without uncaught exception>
=============================== Output 2 / 3 ================================
(Failure "got none!")
=============================== Output 3 / 3 ================================
<expect test ran without uncaught exception>
|}]
;;
end
module _ = Test' (struct
let interesting_opt = Some 5
end)
module _ = Test' (struct
let interesting_opt = None
end)
module _ = Test' (struct
let interesting_opt = Some 5
end)
The extension point [%expect.output] evaluates to a string with the output that would have been captured had an [%expect] node been there instead.
One idiom for testing non-deterministic output is to capture the output using [%expect.output] and post-process it:
(* Suppose we want to test code that attaches a timestamp to everything it prints *)
let print_message s = printf "%s: %s\n" (Time_float.to_string_utc (Time_float.now ())) s
let%expect_test "output capture" =
(* A simple way to clean up the non-determinism is to 'X' all digits *)
let censor_digits s = String.map s ~f:(fun c -> if Char.is_digit c then 'X' else c) in
print_message "Hello";
[%expect.output] |> censor_digits |> print_endline;
[%expect {| XXXX-XX-XX XX:XX:XX.XXXXXXZ: Hello |}];
print_message "world";
[%expect.output] |> censor_digits |> print_endline;
[%expect {| XXXX-XX-XX XX:XX:XX.XXXXXXZ: world |}]
;;
Other uses of [%expect.output] include:
t_of_sexp and performing tests on the resulting structure.[%expect] block.The [%expectation] extension point provides programmatic control over test expectations, allowing for dynamic handling of test outputs.
The public API for this functionality is available in expect_test_helpers_core.
An expectation becomes "active" when execution encounters an [%expectation] block. Only one expectation can be active at a time. The active expectation remains active until explicitly resolved through one of the following functions:
commit: Accepts the test result for the currently active expectation. If the test passes, no diff is produced. If it fails, the expectation is updated in the corrected file.skip: Ignores the test result, making no changes to the expectation block even if it fails. The expectation still consumes its input.reset: Like skip, but also restores the collected input, allowing subsequent expect blocks to capture it.Each [%expectation] must be followed by exactly one call to commit, skip, or reset before the next [%expect], [%expect_exact], [%expectation], or the end of the test.
While an expectation is active, these functions can be called to examine its state:
is_successful: Reports whether the actual output matches the expected output (accounting for whitespace normalization)expected: Returns the expected output stringactual: Returns the actual output string captured so farThis functionality is particularly valuable for testing asynchronous systems where state changes don't occur immediately. Consider a test with the following pattern:
Standard expect tests lack the waiting mechanism in step 2, which can lead to flaky tests, especially in UI testing where timing varies. The [%expectation] extension allows implementation of a polling pattern:
let animate state ~target ~step = Int.min (state + step) target
let progress_bar state ~target =
state := animate !state ~target ~step:1;
for i = 0 to 9 do
printf (if i < !state then "#" else ".")
done
;;
let%expect_test "progress_bar_state_change" =
let progress = ref 0 in
let timeout = ref 100 in
let target = 6 in
while !timeout > 0 do
timeout := !timeout - 1;
progress_bar progress ~target;
[%expectation {| ######.... |}];
if Expectation.is_successful () || !timeout = 0
then Expectation.commit ()
else Expectation.skip ()
done
;;
This approach combines the waiting (2) and verification (3) steps, reducing flakiness by polling until the actual state matches the expected state or a timeout occurs.
Expect-test exposes hooks for configuring how the bodies of expect tests are run, which can be used to set up and tear down test environments, sanitize output, or embed [%expect] expressions in a monadic computation, like a Deferred.t.
Each let%expect_test reads these configurations from the module named Expect_test_config in the scope of that let binding. The default module in scope defines no-op hooks that the user can override. To do so, first include the existing Expect_test_config, then override a subset of the following interface:
module type Expect_test_config = sig
(** The type of the expression on the RHS of a [let%expect_test]
binding is [unit IO.t] *)
module IO : sig
type 'a t
val return : 'a -> 'a t
end
(** Run an IO operation until completion *)
val run : (unit -> unit IO.t) -> unit
(** [sanitize] can be used to map all output strings, e.g. for cleansing. *)
val sanitize : string -> string
(** This module type actually contains other definitions, but they
are for internal testing of [ppx_expect] only. *)
end
For example, Async exports an Expect_test_config equivalent to:
module Expect_test_config = struct include Expect_test_config module IO = Async_kernel.Deferred let run f = Async_unix.Thread_safe.block_on_async_exn f end
If we want to consistently apply the same sanitization to all of the output in our expect test, like we did in the timestamp example above, we can override Expect_test_config.sanitize. This cleans up the testing code and removes the need to use [%expect.output].
(* Suppose we want to test code that attaches a timestamp to everything it prints *)
let print_message s = printf "%s: %s\n" (Time_float.to_string_utc (Time_float.now ())) s
module Expect_test_config = struct
include Expect_test_config
(* A simple way to clean up the non-determinism is to 'X' all digits *)
let sanitize s = String.map s ~f:(fun c -> if Char.is_digit c then 'X' else c)
end
let%expect_test "sanitization" =
print_message "Hello";
[%expect {| XXXX-XX-XX XX:XX:XX.XXXXXXZ: Hello |}];
print_message "world";
[%expect {| XXXX-XX-XX XX:XX:XX.XXXXXXZ: world |}]
;; Debugging deadlocks & hanging tests
One common pitfall while using expect-test is that it hurts the inspectability of code that fails to terminate. Normally, adding prints to understand how code behaves interacts well with expect-test; when the tests finish, the test framework produces a corrected file, where the output has been inserted to the interleaved [%expect] extension points. However, if a test never finishes, no corrected file is produced at all, and all of the useful debug output produced by the program seemingly vanishes into oblivion.
As of December 2024, it is possible to see the output collected by a running expect-test program in real-time using the -verbose flag1.
For example, the output of the following OCaml program (test_loops.ml)
let rec loop () = loop () let%expect_test "doesn't finish" = print_endline "about to enter an infinite loop"; loop () ;;
can be seen by running
$ ./inline_tests_runner -verbose -only-test test_loops.ml
File "test_loops.ml", line X, characters X-X: doesn't finish
about to enter an infinite loop
despite the fact that the test never finishes or produces a corrected file.
When -verbose is used, running the inline_tests_runner executable to completion still produces a corrected file, so the -verbose flag need not be incompatible with normal workflows.
Follow the same rules as for ppx_inline_test.
This behavior is only supported on inline_tests_runner executables built for native code on Linux systems. It is not supported on Windows or inline_tests_runners compiled to WASM or javascript. ↩
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4