Having a good battery of tests for your code may be a requirement for many Python projects. In practice, writing unit tests is hard and can take a lot of time and effort. Therefore, some developers don’t like to write them. However, with large language models (LLMs) and tools like ChatGPT, you can quickly create robust and complete sets of tests for your Python code.
In Python, you can use multiple different tools for writing tests. The most commonly used tools include doctest
, unittest
, and pytest
. ChatGPT can be of great help in writing tests with any of these tools.
In this tutorial, you’ll:
doctest
unittest
tests, fixtures, and suitespytest
tests and fixturesTo get the most out of this tutorial, you should set up a ChatGPT account and know the basics of interacting with this tool using prompt engineering. You should also know the basics of how to test code in Python.
Get Your Code: Click here to download the free sample code you’ll use to write unit tests for your Python code using ChatGPT.
Benefits of Using ChatGPT for Testing Python CodeHaving good and up-to-date unit tests for your code is a must for any Python project. Poorly tested code or code without tests may end up being unreliable and weak. With automated tests, you can ensure and show that your code works correctly in different scenarios. So, having tests is important from the technical and commercial point of view.
Writing good tests is hard and can take a lot of time. That’s why some developers don’t like to write them at all. Using large language models (LLM) like ChatGPT can be a viable alternative for providing your projects and code with proper tests.
Some of the benefits of using ChatGPT to write tests for your Python code include the following:
With ChatGPT, you can generate unit tests for your Python code in almost no time. However, you must note that even though the generated tests can look good, you should still review and possibly refine the generated tests.
Note: For this tutorial, you’ll use a free ChatGPT account that’s powered by the GPT-3.5 LLM. You can use this model for free with the default web interface by OpenAI. If you use a paid account with GPT-4, then your results might be more accurate and faster.
Additionally, you’ll use an individual chat thread for each major section in the tutorial. So you’ll use different threads for the doctest
, unittest
, and pytest
sections.
Using a single chat thread to run all the examples in this tutorial may cause the chat to generate significantly different responses because of the past context, which includes previous prompts and responses.
In the following sections, you’ll learn the basics of using ChatGPT as an assistant for creating coherent unit tests for your Python projects using different test frameworks and libraries.
Writingdoctest
Tests With ChatGPT
Python’s doctest
module provides a lightweight testing framework that you can use for test automation. It can read the test cases from your project’s documentation and your code’s docstrings. This framework comes with the Python interpreter, so you’ll have it at your disposal with any Python installation, which is great.
Note: To dive deeper into how to use the doctest
module for testing your code, check out Python’s doctest
: Document and Test Your Code at Once.
With doctest
, you can write code that provides examples of how your code works, simulating REPL sessions. These examples work as documentation and also as test cases or unit tests. Each doctest
test needs a statement or expression and an expected result.
You can use ChatGPT to help you out with writing doctest
tests. First, you need some code to test. The following function tries to tackle the FizzBuzz challenge, where you return "fizz"
for numbers divisible by three, "buzz"
for those divisible by five, and "fizz buzz"
for those divisible by both three and five.
Go ahead and create a fizzbuzz.py
file and add the following code to it:
This function works okay for numbers divisible by 3
or 5
. However, there is a slight issue with numbers that are divisible by both. Even though you know this, go ahead and start a new chat thread. Then, prompt ChatGPT to write doctest
test for the function:
You: Write
doctest
tests for the following function:ChatGPT:
Certainly! The function you’ve provided implements a basic FizzBuzz algorithm but with a slight logical error. The check for divisibility by
15
should come before checks for3
and5
, as every number divisible by15
is also divisible by both3
and5
, but your current order will never reach the"fizz buzz"
case. Here’s a corrected version of your function with the logical order fixed, followed bydoctest
tests for it:To use these
doctest
tests, you’ll need to include them in the docstring of your function as shown above. Then, you can run the tests with thedoctest
module by adding the following to your script:This code will automatically run the tests defined in the docstrings when you execute the Python file. Remember, the corrected order of conditions in the function ensures that numbers divisible by both
3
and5
return"fizz buzz"
, which is crucial for the FizzBuzz logic.
Wow! That great! The chat not only wrote the doctest
tests for you. It also fixed the issue, explained important concepts around doctest
, and provided instructions on how to run the tests. Note that the tests cover both regular and edge cases, which is pretty convenient.
Note: When you run the prompts from this tutorial in your own ChatGPT instance, you’ll get different answers.
It’s vital that you understand that the generated responses can include mistakes, issues, and incorrect concepts. So, you must carefully review each output and correct any problem you find.
Think of ChatGPT as a productivity booster tool rather than a source of strictly correct information or code.
Go ahead and run the following command from your terminal window to execute the tests and see their results:
After running the above command, you’ll see a verbose output with detailed information about every doctest
test. The -v
command-line option is responsible for generating the verbose output. As you can see, all the tests passed, so your function now works okay. The LLM has done a great job.
unittest
Tests With ChatGPT
The unittest
framework is another standard library tool that you can use to write unit tests for your code. This testing framework uses an object-oriented approach and supports some essential concepts that facilitate code testing:
In the following sections, you’ll write some quick prompts for ChatGPT to provide test cases, fixtures, and test suites for you.
Writing Test CasesThe unittest
module provides the TestCase
base class, which you can use to write test cases through subclassing. Unlike doctest
, unittest
is a tool for creating elaborate test cases in independent modules that you can group in a tests/
folder under your project’s root directory or in any other location.
Again, you can ask ChatGPT to write unittest
test cases for you. Here’s a quick prompt to do this. Remember to start a new chat thread before running the prompt:
You: Write
unittest
test cases for the following function:ChatGPT:
To write
unittest
test cases for thefizzbuzz()
function, you need to follow these steps:
- Import the
unittest
module.- Define a test class that inherits from
unittest.TestCase
.- Write test methods within this class to test different aspects of the
fizzbuzz()
function.
- Each test method should start with the word
test
.- Use assertions like
self.assertEqual()
to check the function’s output against expected results.- Finally, call
unittest.main()
to run the tests when the script is executed.Here’s how you can structure your test cases for the
fizzbuzz()
function:To run these tests, save the code in a
.py
file and execute it with Python. Theunittest
framework will automatically discover the test methods inTestFizzBuzz
and execute them, reporting the results.
This response is also great! The chat gives you a great explanation of how to organize your test cases when you’re using unittest
. Then, it generates the code for the tests.
Note that every test includes multiple assertions with different arguments and outputs. This approach might be appropriate when you need to test multiple related properties on an object. In your example, this approach works fine because the combined assertions test for a single return value.
Using multiple assertions helps reduce boilerplate code. In general, this approach has the following pros:
This approach also has its cons:
Finally, ChatGPT gives you quick instructions on running the tests. You already have a fizzbuzz.py
file containing the fizzbuzz()
function. So go ahead and create the file called test_fizzbuzz_unittest.py
and paste only the test-related code:
In this file, you’ve copied the test-related code as the chat generated it. Note that you’ve imported the fizzbuzz()
function from your fizzbuzz.py
file for the tests to run. Now, go ahead and run the following command to execute the tests:
This command runs the test from the test_fizzbuzz_unittest.py
using the command-line interface of unittest
. The -v
command-line option generates a verbose output that takes advantage of the tests’ docstrings to provide context on what each test is about.
Because the code in the chat response includes the name-main idiom wrapping the unittest.main()
function call, you can also run the tests by running the test_fizzbuzz_unittest.py
as a regular Python script.
A test fixture is a preparation that you need to perform for one or more tests and any associated cleanup actions. The setup code may involve the creation of temporary files, objects, databases, datasets, connections, and so on.
To illustrate how fixtures work in unittest
, say that you’ve created a readers.py
file that implements a JSONReader
class that looks something like the following:
You need to write unittest
tests for your class. These tests will need preparation. For example, you can write a setup fixture to create a temporary file with some sample content. Then, you can remove the file after the tests run in the teardown stage.
Here’s a quick prompt that you can use with ChatGPT to write the fixtures and the appropriate tests:
You: Write
unittest
test cases with fixtures for the following class:ChatGPT:
To write
unittest
test cases with fixtures for theJSONReader
class, you’ll need to follow these steps:
- Use setup and teardown methods to prepare and clean up test fixtures. These can be
.setUp()
(runs before each test method) and.tearDown()
(runs after each test method) for resource management.- In the setup method, create a temporary JSON file that the
JSONReader
class will read.- Write test methods to verify that the
JSONReader
class correctly reads JSON content from a file.- Use the teardown method to clean up any resources, such as deleting the temporary file created for testing.
Here’s how you can implement these steps in a
unittest
test case:This example demonstrates how to:
- Use
.setUp()
to prepare a temporary JSON file that theJSONReader
class will use for reading.- Write a basic test (
.test_read_json()
) to verify that theJSONReader
can correctly read JSON data from a file.- Write a test (
.test_file_not_found()
) to check how theJSONReader
behaves when the file does not exist, expecting aFileNotFoundError
.- Use
.tearDown()
to clean up by deleting the temporary file after each test.To run these tests, save the code in a
.py
file and execute it with Python. Theunittest
framework will automatically discover and run the tests, reporting the results.
In its response, the chat first lists the required steps to write test cases and fixtures for the target class. Then, it generates the test code. The .setUp()
method creates the required temporary file with some sample data using the tempfile
module from the standard library. The .tearDown()
method removes the temporary file as a clean-up action. In general, the code looks good and well organized.
Note that you’ll find some coding style issues in the generated code. For example, instead of formal docstrings, the test functions start with a comment. Additionally, some attributes may not have descriptive names and, in some cases, you’ll find unneeded constructs like using a throwaway variable in the last test. You can manually fix these issues to fulfill your coding style requirements.
You already have a readers.py
file containing your JSONReader
class. Go ahead and create a file called test_readers_unittest.py
and copy the generated code. Remember to remove the class and import it from your readers.py
file. Then, run the following command to execute the tests:
Great! The tests passed, and the generated code works correctly.
Creating Test SuitesUsing unittest
, you can create test suites for grouping tests. This way, you can selectively run a set of tests on your code, which can be convenient in those situations where your tests take a lot of time to run. In this case, you may need a shortcut that allows you to run only the test that relates to the code that you’re working on.
To illustrate how to create tests and group them with test suites using ChatGPT, say that you have a module called calculations.py
that defines basic arithmetic and statistics operations:
You need to generate unittest
tests for all the functions in this module. Here’s a quick prompt that you can use to complete this task:
You: Write
unittest
test cases for the following module:ChatGPT:
To test the module you provided, which contains functions for basic arithmetic operations and statistical calculations, you can follow the structure outlined below. This structure includes tests for each function, ensuring that they handle both typical cases and edge cases, such as division by zero or calculating the mode of a dataset with multiple modes:
This test suite covers each function in the provided module. The arithmetic operations are tested with positive, negative, and zero values. The statistical operations are tested with lists of various lengths and compositions, including edge cases:
- For
divide()
, it tests division by zero, expecting aValueError
.- For
mean()
, it tests with lists of both even and odd lengths.- For
median()
, it tests both odd and even number of elements in the list, including a test where the data isn’t initially sorted.- For
mode()
, it tests both when there is a single mode and multiple modes in the list.To run these tests, ensure the functions from the module are either defined in the same file or properly imported into the test file, and then execute the test file with Python. The
unittest
framework will automatically discover and run the tests, reporting the results.
Great! You have test cases for all the functions in your calculations.py
module. The different tests are grouped into two TestCase
subclasses. The first one groups the basic arithmetic functions and the second one groups the statistical functions. This is a great way to structure your tests.
There’s a tiny detail that you must note in this generated code. The chat is importing the Counter
class from the collections
module. This import isn’t needed because the test code doesn’t use the class. So you can safely remove this import.
Before trying to run the tests, go ahead and import all the functions from your calculations.py
module. Then run the following command, which executes all the tests from the test_calculations.py
file:
All the tests passed! In this case, you didn’t use the -v
command-line option so you don’t get the verbose output.
Now, say that you want to run your test independently. For example, you might want to run the tests on the arithmetic functions separately from the tests on statistical functions. You can use unittest
test suites for this purpose.
Here’s a prompt to ask the chat to help you out. Remember to stay in the same chat thread:
You: Write
unittest
test suites to group the arithmetic functions and the statistical functions.ChatGPT:
To group the tests for arithmetic and statistical functions into separate test suites using Python’s
unittest
framework, you can use theTestSuite
class:This script defines two test classes,
TestArithmeticOperations
andTestStatisticalOperations
, each containing tests relevant to their category. Theload_tests()
function is used to create a test suite that combines tests from both classes, managing a total of 10 tests. This function is a hook provided byunittest
for customizing test loading and suite creation. It’s called automatically when usingunittest.main()
.To run the tests, ensure that the functions from your module are either defined in the same file or properly imported into the test file, and then execute the test file with Python. The
unittest
framework will discover theload_tests()
function, use it to construct the test suite, and run the tests, reporting the results.
In the above response, the chat generated two TestCase
subclasses, which are similar to the ones created before. Then, it generates the load_tests()
function. In this example, the function defines a test suite, including the two test cases.
If you run the code as is, then you’ll be executing all the tests. However, if you comment out one of the calls to .addTests()
in load_tests()
function, then you’ll be executing specific tests. To try these updated tests, go ahead and replace the content of your test_calculations.py
with the above code. Remember to import the functions from calculations.py
.
Then, comment out the line that adds the TestArithmeticOperations
and run the following command:
When you comment out the line that adds the arithmetic tests, your code only runs the statistical tests.
Using test suites has several benefits. Some of the most relevant include the following:
Test suites are a powerful feature of unittest
that enhance the organization, execution, and maintenance of tests, making them particularly useful in complex testing scenarios.
pytest
Tests With ChatGPT
If you’ve dived deep into Python tests, then you’ll know the pytest
framework, which is a popular testing framework for Python. It allows you to test all kinds of Python applications and libraries. It’s well-known for its simplicity, flexibility, and powerful features, which include some of the following:
test_*.py
or *_test.py
and runs all functions and methods whose names start with test_*
.In short, pytest
is a full-featured and flexible testing framework that simplifies the process of writing, organizing, and running tests in Python. In the following sections, you’ll use ChatGPT to write pytest
tests, create parametrized tests, and write test fixtures.
In pytest
, you can write tests as regular functions. You don’t need to subclass a base class or use specific methods from a framework’s API. So, writing tests with pytest
is quick and flexible.
To write some pytest
tests with ChatGPT, go ahead and start a new chat thread. Then issue the following prompt:
You: Write
pytest
test cases for the following function:ChatGPT:
Certainly! To write
pytest
test cases for thefizzbuzz()
function, you first need to make sure you havepytest
installed. If it’s not installed, you can install it usingpip install pytest
.Here are some example test cases that cover all branches of the
fizzbuzz()
function:To run these tests, you’ll save them in a file (let’s say
test_fizzbuzz.py
) and then executepytest
in the terminal in the directory where your test file is located. Pytest will automatically discover tests that are prefixed withtest_
and execute them.
In this response, you have five separate test functions. Each function tests for a different scenario. It’s important to note that you don’t need to import pytest
at the beginning of your module for the code to work.
Note that ChatGPT has generated functions with a single assertion each. This approach differs from what it did with the unittest
tests. In general, using a single assertion per test has the following pros:
Single-assertion tests also have some cons:
As ChatGPT says, you need to install pytest
to be able to run these tests. So, go ahead and run the following command in your Python environment:
This command will download pytest
from the Python Package Index (PyPI) and install it in your current Python environment.
Now, you’re ready to run the generated tests. Go ahead and create a file called test_fizzbuzz_pytest.py
and add the above code to it. Remember to import your fizzbuzz()
function. You can also remove the pytest
import. Once you have the file in place, run the following command:
Your tests work as expected. They test four different scenarios, and your code successfully passes all of them.
Parametrizing TestsAs you already learned, pytest
allows you to parametrize test functions. This parametrization is pretty handy because it frees you from writing one test function for every set of parameters. So, you can write fewer test functions and have more tests at the same time. This reduces the repetitive boilerplate code and increases your productivity.
You’ll use the @pytest.mark.parametrize decorator to define multiple sets of arguments for a test function or class.
Assuming that you’re following the example of the previous section and have the same chat thread active, you can use the following prompt to ask ChatGPT to update the test for you:
You: Update the previous tests using parametrization to avoid repetition.
ChatGPT:
To update the previous test cases for the
fizzbuzz
function using pytest’s parametrization feature, we can consolidate them into a single test function that takes different input values and expected outputs as parameters. This approach reduces repetition and makes the test suite more concise and maintainable. Here’s how you can do it:In this updated version, the
@pytest.mark.parametrize
decorator is used to run thetest_fizzbuzz()
function multiple times with different sets of arguments (input
andexpected
). Each tuple in the list passed to the decorator represents a test case, where the first element is the input to thefizzbuzz()
function, and the second element is the expected output.This approach allows you to cover a wide range of test scenarios with a single test function, making the test code cleaner and reducing redundancy. Plus, adding new test cases is as simple as adding a new tuple to the list of parameters, enhancing the maintainability of the test suite.
Now, you have a single test function called test_fizzbuzz()
. However, with the parametrization, this function works as eight different tests—one for each pair of parameters.
Using parametrization in tests offers several advantages. It reduces code duplication, making your test suite more concise. It also improves test coverage and simplifies code maintenance. In short, parametrization is a great feature of pytest
that you’ll end up loving.
Update your test_fizzbuzz_pytest.py
file with the above code. Then import fizzbuzz()
and execute the following command to run the tests:
$ python -m pytest test_fizzbuzz_pytest.py
======================== test session starts ========================
platform darwin -- ...
rootdir: .../examples
collected 8 items
test_fizzbuzz_pytest.py ............. [100%]
========================= 8 passed in 0.03s =========================
In the output, you can see that pytest
collected eight tests in total. This means that it counts every set of parameters as an individual test.
You can also create code fixtures in pytest
to handle setup and teardown logic. To do this, you can use the @pytest.fixture
decorator.
To continue with the code examples you’ve seen so far, get back to your JSONReader
. Now, issue the following prompt into your ChatGPT window. Remember to continue in the pytest
thread:
You: Write
pytest
test cases with fixtures for the following class:ChatGPT:
To write
pytest
test cases for theJSONReader
class, including the use of fixtures for setting up necessary test files, follow the steps below. First, ensure you havepytest
installed.
- Fixture Setup: Create a fixture that sets up a temporary JSON file, which will be used by the tests. This fixture will create a file before each test runs and delete it afterward to ensure tests do not interfere with each other.
- Test Cases: Write test cases to verify that the
JSONReader
class can correctly read JSON files. This includes testing for correct content reading and handling of possible exceptions (e.g., file not found).Here is an example of how the test file might look:
In this example,
tmp_path
is a built-in pytest fixture that provides a temporary directory unique to the test function that requests it. We use it to create a temporary JSON file for testing. Thesetup_json_file
fixture creates a temporary file with some JSON content and returns the path to this file, which is then used by thetest_read_json_correctly
test function.The
test_file_not_found
function tests the behavior of theJSONReader
class when attempting to read a non-existent file, expecting aFileNotFoundError
to be raised.This setup ensures that tests are isolated and do not depend on or interfere with external files, making your tests more reliable and portable.
In this response, you have a test fixture that creates a temporary file to save the JSON sample content. In pytest
, the tmp_path
fixture provides a quick way to create temporary directories. You don’t need to explicitly remove the temporary directory because the fixture takes care of this task for you.
The response isn’t perfect. Instead of formal docstrings, the test functions start with a comment, but that’s a coding style issue. Another detail that pops up is that the response includes an unneeded import. The fixture and tests don’t use the os
module, so you can remove the import, and your test code will be ready for a try. In this case, import pytest
is necessary, since it’s used to define your fixture.
Here’s how to run the tests:
The tests run successfully! So the fixture is working correctly.
Using Other Types of Prompts for Writing TestsUp to now, you’ve included the code that you want to write tests for in every ChatGPT prompt. Sometimes, you can’t do this because you must fulfill privacy requirements or keep your company’s code safe.
In this scenario, you can create prompts that only use the signature of your classes, methods, and functions. You can also use general descriptions or specifications of your code to develop working prompts.
Note: There are many ongoing lawsuits around ChatGPT’s use of data. In any case, you should check the ChatGPT’s terms of use to make sure you’re using the tool in the correct way.
In the following section, you’ll explore examples of creating prompts with signatures and descriptions.
Including Only Signatures in Your PromptsTo illustrate how to create prompts that use the signature of your classes, methods, and functions, consider the signature of a function that checks if a number is in a given interval. Remember to start a new chat thread before running the prompt below:
You: Write
pytest
tests for a function with the following signature:ChatGPT:
To write
pytest
tests for the functionin_interval(number: int, start: int, end: int) -> bool
, we first need to define the function based on its signature. This function should returnTrue
ifnumber
is betweenstart
andend
(inclusive), andFalse
otherwise. Let’s define this function and then write severalpytest
test cases to cover various scenarios, including edge cases.First, here’s a simple implementation of the
in_interval
function:Now, let’s write the
pytest
tests. Create a new Python file for your tests, e.g.,test_in_interval.py
, and include the following code:This test file uses
pytest.mark.parametrize
to define multiple test cases for thein_interval
function. Each test case specifies anumber
,start
,end
, and theexpected
result. This approach allows you to efficiently test your function with a variety of inputs to ensure it behaves correctly under different conditions.
In this example, the function’s signature is the only context that you provide to ChatGPT. The language model is smart enough to examine the function’s name and arguments and generate a suitable implementation. This may not be the case with real-world functions.
Then, the chat generates a test function with eight sets of parameters. Each set of parameters tests the function in one specific scenario.
Note that having descriptive function and argument names is critical in this kind of prompt. Adding type hints to the function’s signature will also improve the quality of the generated docstring.
Writing Prompts That Use Code DescriptionsYou can also craft a prompt by just describing the input and output of your code or by providing a detailed description of what your code does. Again, these prompts are helpful when you’re not allowed to upload your code as part of your prompt.
To illustrate, here’s a prompt that asks ChatGPT to write tests for a function that determines whether an integer number is prime:
You: Write
pytest
tests for a function that takes an integer as an argument and determines whether it’s a prime number.ChatGPT:
To write
pytest
tests for a function that determines whether a given integer is a prime number, we first need to define the function. Here’s a simple implementation of such a function, which we’ll nameis_prime
. This function returnsTrue
if the number is a prime, andFalse
otherwise:Now, let’s write the
pytest
tests. Create a new Python file for your tests, for example,test_is_prime.py
, and include the following code:This test file uses
pytest.mark.parametrize
to specify multiple test cases for theis_prime
function. Each test case provides anumber
to test and theexpected
result (True or False). This setup allows you to comprehensively test your function against a variety of inputs, including edge cases such as the smallest prime number, composite numbers, the number one, zero, and negative numbers.
Again, the chat has generated a possible implementation for the function. After that, it generates the tests. You should provide more detailed descriptions for more complex functions, classes, or other pieces of code so that ChatGPT can generate better responses.
It’s also important to note that these tests are just a starting point. It’s up to you to modify and tweak them as needed to match your specific code. In any case, they can save you from writing a lot of boilerplate code.
ConclusionNow you know how to leverage the power of tools like ChatGPT to write unit tests for your Python code. You’ve used the LLM to generate tests using different tools, including doctest
, unittest
, and pytest
. In the process, you’ve learned how to create test cases, fixtures, test suites, and more.
In this tutorial, you’ve learned how to:
doctest
unittest
tests, fixtures, and suitespytest
tests and fixturesWith these skills, you’re ready to start creating robust and complete test batteries for your Python code and projects.
Get Your Code: Click here to download the free sample code you’ll use to write unit tests for your Python code using ChatGPT.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4