Tabularised feature comparison between Catch, doctest and lest C++ test frameworks.
Skip to the feature comparison.
Note: This is an initial draft, it is incomplete and likely contains errors.
Ideas, additions, corrections, signaling omissions, etc. welcome!
The frameworks in this comparison share the following properties:
Advent of Catch Expression-decomposing assertions and function-level fixtures (sections) were introduced by Kevlin Henney in his talk Rethinking Unit Testing in C++ at ACCU London in May 2010 [1]. At that time, Phil Nash was working on a testing framework called YACUTS (Yet Another C++ Unit Test System) that would bring similar capabilities, be it a little more intrusively [2]. Kevlin's talk inspired Phil to develop Catch [3].
Kevlin's emphasis on simplicity in testing and the desire to understand how expression decomposition works inspired Martin Moene to write lest [4]. The code of lest is intentionally kept small, so that it should be fairly easy to gain insight in how things work.
Accompanying test code with the code to test was one of the drivers for Viktor Kirilov to write doctest [5]. Others are to enable using doctest with many different compilers and to shine in compile-time and run-time performance. At CppCon 2017 Viktor gave a presentation that explains many techniques used in doctest and Catch [23].
Feature doctest Catch-1 Catch-2 lest Notes License MIT BSL-1.0 BSL-1.0 BSL-1.0 C++ standard related C++ standard C++98 C++98 C++11 C++11 Can work without exceptions ✓ - - - reduced functionality Requires RTTI - - - - Fields of use Desktop ✓ ✓ ✓ ✓ Embedded (tiny) ? ? ? ? Embedded (largish) ? ? ? ✓ Application, tool ? ✓ ✓ ? Library, framework ? ✓ ✓ ✓ Operating System ? ? ? ✓ lest: IncludeOS Documentation of framework see [22] First contact ? ? ? ? new users Education ? ? ? ? new & existing users Support ? ? ? ? experienced users Troubleshooting ? ? ? ? annoyed users Internals ? ? ? ? fellow developers Reference ? ? ? ? everyone Try it online ✓ ✓ ✓ ✓ on wandbox Forum - ✓ ✓ - Catch: Google group Chat ✓ - - - doctest: Gitter StackOverflow tag - ✓ ✓ - see [19] Test organisation Suites of tests ✓ - - - Fixtures (sections) ✓ ✓ ✓ ✓ doctest: sub case Fixtures (class-based) ✓ ✓ ✓ - Type-parameterised tests ✓ ✓ ✓ - Value-parameterised tests - - - - Auto-registration of tests ✓ ✓ ✓ ✓ Tabularised tests - - - ✓ array of lambdas Allow tests accompany code ✓ ✓ ✓ - Catch: not recommended... Allow tests in header files ✓ ✓ ✓ - ...to avoid long compilation times Assertions BDD style scenarios ✓ ✓ ✓ ✓ Assert expressions ✓ ✓ ✓ ✓ Assert exceptions ✓ ✓ ✓ ✓ Assert abortion (death) - - - - Assert assertions (death) - - - - Expression decomposition ✓ ✓ ✓ ✓ see [2] Floating point comparison, approx ✓ ✓ ✓ ✓ Floating point comparison, ULP - - ✓ - see stf [10] Hamcrest matchers (built-in) - some some • see [12][13][14][15] Hamcrest matchers (user-definable) - ✓ ✓ - Macros with and without prefix ✓ ✓ ✓ ✓ CATCH_CHECK(), CHECK() Other test facilities Mocking support • • • • via 3rd party Logging facility ✓ ✓ ✓ - Logging levels - - - - Checkpoints - - - - see Boost.Test [11] Test decorators ✓ - - - see [16] Test data generators - • - - Cach-1: undocumented Property-based testing - - - - Obtain name of current test ? • ? - Catch-1: undocumented API / seams Reporting user-defined types ✓ ✓ ✓ ✓ Reporter API - ✓ ✓ - Event listeners - ✓ ✓ - Run-time context ✓ ✓ ✓ - Reporting formats Console, multi-line ✓ ✓ ✓ - Console, single-line (compact) - ✓ ✓ ✓ similar to compiler error JUnit - ✓ ✓ - result at end XML - ✓ ✓ - streaming TeamCity - ✓ ✓ - TAP - ✓ ✓ - Automake - ✓ ✓ - Reporting options Colourised output (run-time) ✓ ✓ ✓ Colourised output (compile-time) ✓ - - ✓ Literal suffix u, l, f - - - ✓ compile-time Test execution Concurrent execution of tests - - - - Isolated execution of tests - - - - Limit test execution time ✓ - - - doctest: via test decorator Signal tests without assertions - ✓ ✓ - Supports floating point exceptions - - - - Supports Posix signals ✓ ✓ ✓ - Supports Windows SEH ✓ ✓ ✓ - Compile-time control Improve compilation speed ✓ ✓ ✓ - reduced functionality Omit macros without prefix ✓ ✓ ✓ ✓ Omit std::wstring - - - ✓ Omit std::cout, cerr, clog - ✓ ✓ - Specify terminal color system ✓ ✓ ✓ - none, ansi, win32 Specify terminal width - ✓ ✓ - Control handling of POSIX signals ✓ ✓ ✓ - Control handling of SE on Windows ✓ ✓ ✓ - Enable leak checking on Windows - ✓ ✓ - uses CRT Debug Heap Enable use of __COUNTER__ - ✓ ✓ - for internal unique names Run-time control, commandline Help screen ✓ ✓ ✓ ✓ List selected suites ✓ - - - List selected tests ✓ ✓ ✓ ✓ List tags of selected tests - ✓ ✓ ✓ Select tests by suite ✓ - - - Select tests by name ✓ ✓ ✓ ✓ Select tests by tag ✓ ✓ ✓ ✓ Select tests by file ✓ ✓ ✓ - Catch: via filename as tags Select section by name ✓ ✓ ✓ - Select supports regexp ✓ - - ✓ Omit assertions expected to throw - ✓ ✓ - Count selected tests ✓ - - ✓ Abort at N-th failure ✓ ✓ ✓ ✓ lest: N is 1 Break into debugger ✓ ✓ ✓ - Report passing tests ✓ ✓ ✓ ✓ Time duration of tests ✓ ✓ ✓ ✓ Control order of tests ✓ ✓ ✓ ✓ Repeat tests - ✓ ✓ ✓ IDE integration see [17] Output-window-compatible reports ✓ ✓ ✓ ✓ click through to source C++ Builder ? ? ? ? CLion ✓ ✓ ✓ - Code::Blocks ? ? ? ? CodeLite ? ? ? ? Eclipse CDT ? ? ? ? KDevelop ? ? ? ? Qt Creator ? ? ✓ ? Visual-MinGW ? ? ? ? Visual Studio • • • - Catch and doctest: via ReSharper C++ Visual Studio Code - - - - see issue Platforms AIX / XL ? 11.1.0 ? ? Catch-1: 2013 macOS / clang ? ? ? ? macOS / gcc 4.4-6 ? ? ? source: Travis macOS / XCode / clang 6.4-8.3 7.3-8 7.3-8 ? source: Travis macOS / XCode / gcc 6.4-7.3 7.3-8 7.3-8 ? source: Travis Linux / clang 3.4-4 3.4-3.8 3.5-3.8 3.6-3.7 source: Travis Linux / gcc 4.4-6 4.4-6 4.8-6 4.9-5 source: Travis Solaris / SunPro ? 5.12.0 ? ? Catch-1: 2013 Windows / clang ? ? ? 3.2+ Windows / gcc 7 ? ? 4.8.1+ Windows / Visual C++ 9-15 10-15 14-15 12-15 Catch-1: possibly from 8 Windows / Cygwin / clang ? ? ? ? Windows / Cygwin / gcc ? ? ? ? Windows / msys2 / clang ? ? ? ? Windows / msys2 / gcc ? ? ? ? Quality of project note: challenge 42 Code of conduct - ✓ ✓ - Follows CII best practices ✓ partly partly partly see [21] Quality of code Unit testing ✓ ✓ ✓ ✓ details below CI testing ✓ ✓ ✓ ✓ details below CI code analysis ✓ - - - details below Testing of framework Unit tests ✓ ✓ ✓ ✓ Tests itself ✓ ✓ ✓ ✓ Baseline tests ✓ ✓ ✓ - CI testing Travis ✓ ✓ ✓ ✓ Appveyor ✓ ✓ ✓ ✓ Coverage scan ✓ - ✓ - Coverity scan ✓ - - - CI static code analysis clang-tidy ✓ - - - CppCheck ✓ - - - CppCoreCheck - - - - PVS-Studio - - - - OCLint ✓ - - - ReSharper C++ - - - - Visual Studio ✓ - - - Code-related Compile-time performance, relative ☆☆☆ ☆☆ ? ? see [20] Run-time performance, relative ☆☆☆ ☆☆ ? ? see [20] Compiler-warning friendly ☆☆☆ ☆☆ ☆☆ ☆☆ Compiles with -Werror -Wall -Wextra ✓ ✓ ✓ ✓ Can provide canned main() ✓ ✓ ✓ - Only depends on C++ std library ✓ ✓ ✓ ✓ Single-file header-only ✓ ✓ ✓ ✓ Size, LOC 4,500 9,100 8,800 1,000 see [18] Distribution of framework GitHub single-file download ✓ ✓ ✓ ✓ from the landing page conan ✓ ✓ ✓ - hunter ✓ ? - - vcpkg ✓ ✓ ✓ - Linux and other (search) search → search search on Repology Feature doctest Catch-1 Catch-2 lest Notes[1] Kevlin Henney. Rethinking Unit Testing in C++ (Video). Skills Matter, May 2010.
[2] Phil Nash. The Ultimate C++ Unit Test Framework. Blog, May 2010.
[3] Phil Nash. Unit Testing in C++ and Objective-C just got easier. Blog announcement, December 2010.
[4] Martin Moene. lest errors escape testing. Blog announcement, June 2013.
[5] Viktor Kirilov. doctest – the Lightest C++ Unit Testing Framework. Announcement in Overload 137, February 2017.
[6] Phil Nash. Catch-1 on GitHub. 2010.
[7] Phil Nash. Catch-2 on GitHub. 2017.
[8] Viktor Kirilov. doctest on GitHub. 2014.
[9] Martin Moene. lest on GitHub. 2013.
[10] Joel Falcou. Design rationale for using ULP with stf.
[11] Gennadiy Rozental and Raffi Enficiaud. Checkpoints in Boost.Test.
[12] Matchers in Catch-1.
[13] Matchers in Catch-2.
[14] Matchers in lest: via hamlest.
[15] Hamcrest 1.3 Quick Reference (PDF).
[16] Decorators in doctest.
[17] Wikipedia. List of C++ IDEs.
[18] CodeBlocks IDE. Non-empty, non-comment lines, via File > Properties.
[19] StackOverflow. Questions tagged catch-unit-test.
[20] Viktor Kirilov. doctest Benchmarks. May 2017.
[21] Linux Foundation (LF). CII Best Practices.
[22] Jacob Kaplan-Moss. Writing great documentation.
[23] Viktor Kirilov. Mix Tests and Production Code With Doctest - Implementing and Using the Fastest Modern C++ Testing Framework (slides). CppCon 2017.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4