How do you generate dynamic (parameterized) unit tests in Python?

ID : 10105

viewed : 30

Tags : pythonunit-testingparameterized-unit-testpython

Top 5 Answer for How do you generate dynamic (parameterized) unit tests in Python?

vote vote

91

This is called "parametrization".

There are several tools that support this approach. E.g.:

The resulting code looks like this:

from parameterized import parameterized  class TestSequence(unittest.TestCase):     @parameterized.expand([         ["foo", "a", "a",],         ["bar", "a", "b"],         ["lee", "b", "b"],     ])     def test_sequence(self, name, a, b):         self.assertEqual(a,b) 

Which will generate the tests:

test_sequence_0_foo (__main__.TestSequence) ... ok test_sequence_1_bar (__main__.TestSequence) ... FAIL test_sequence_2_lee (__main__.TestSequence) ... ok  ====================================================================== FAIL: test_sequence_1_bar (__main__.TestSequence) ---------------------------------------------------------------------- Traceback (most recent call last):   File "/usr/local/lib/python2.7/site-packages/parameterized/parameterized.py", line 233, in <lambda>     standalone_func = lambda *a: func(*(a + p.args), **p.kwargs)   File "x.py", line 12, in test_sequence     self.assertEqual(a,b) AssertionError: 'a' != 'b' 

For historical reasons I'll leave the original answer circa 2008):

I use something like this:

import unittest  l = [["foo", "a", "a",], ["bar", "a", "b"], ["lee", "b", "b"]]  class TestSequense(unittest.TestCase):     pass  def test_generator(a, b):     def test(self):         self.assertEqual(a,b)     return test  if __name__ == '__main__':     for t in l:         test_name = 'test_%s' % t[0]         test = test_generator(t[1], t[2])         setattr(TestSequense, test_name, test)     unittest.main() 
vote vote

83

Using unittest (since 3.4)

Since Python 3.4, the standard library unittest package has the subTest context manager.

See the documentation:

Example:

from unittest import TestCase  param_list = [('a', 'a'), ('a', 'b'), ('b', 'b')]  class TestDemonstrateSubtest(TestCase):     def test_works_as_expected(self):         for p1, p2 in param_list:             with self.subTest():                 self.assertEqual(p1, p2) 

You can also specify a custom message and parameter values to subTest():

with self.subTest(msg="Checking if p1 equals p2", p1=p1, p2=p2): 

Using nose

The nose testing framework supports this.

Example (the code below is the entire contents of the file containing the test):

param_list = [('a', 'a'), ('a', 'b'), ('b', 'b')]  def test_generator():     for params in param_list:         yield check_em, params[0], params[1]  def check_em(a, b):     assert a == b 

The output of the nosetests command:

> nosetests -v testgen.test_generator('a', 'a') ... ok testgen.test_generator('a', 'b') ... FAIL testgen.test_generator('b', 'b') ... ok  ====================================================================== FAIL: testgen.test_generator('a', 'b') ---------------------------------------------------------------------- Traceback (most recent call last):   File "/usr/lib/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/case.py", line 203, in runTest     self.test(*self.arg)   File "testgen.py", line 7, in check_em     assert a == b AssertionError  ---------------------------------------------------------------------- Ran 3 tests in 0.006s  FAILED (failures=1) 
vote vote

70

This can be solved elegantly using Metaclasses:

import unittest  l = [["foo", "a", "a",], ["bar", "a", "b"], ["lee", "b", "b"]]  class TestSequenceMeta(type):     def __new__(mcs, name, bases, dict):          def gen_test(a, b):             def test(self):                 self.assertEqual(a, b)             return test          for tname, a, b in l:             test_name = "test_%s" % tname             dict[test_name] = gen_test(a,b)         return type.__new__(mcs, name, bases, dict)  class TestSequence(unittest.TestCase):     __metaclass__ = TestSequenceMeta  if __name__ == '__main__':     unittest.main() 
vote vote

60

As of Python 3.4, subtests have been introduced to unittest for this purpose. See the documentation for details. TestCase.subTest is a context manager which allows one to isolate asserts in a test so that a failure will be reported with parameter information, but it does not stop the test execution. Here's the example from the documentation:

class NumbersTest(unittest.TestCase):  def test_even(self):     """     Test that numbers between 0 and 5 are all even.     """     for i in range(0, 6):         with self.subTest(i=i):             self.assertEqual(i % 2, 0) 

The output of a test run would be:

====================================================================== FAIL: test_even (__main__.NumbersTest) (i=1) ---------------------------------------------------------------------- Traceback (most recent call last):   File "subtests.py", line 32, in test_even     self.assertEqual(i % 2, 0) AssertionError: 1 != 0  ====================================================================== FAIL: test_even (__main__.NumbersTest) (i=3) ---------------------------------------------------------------------- Traceback (most recent call last):   File "subtests.py", line 32, in test_even     self.assertEqual(i % 2, 0) AssertionError: 1 != 0  ====================================================================== FAIL: test_even (__main__.NumbersTest) (i=5) ---------------------------------------------------------------------- Traceback (most recent call last):   File "subtests.py", line 32, in test_even     self.assertEqual(i % 2, 0) AssertionError: 1 != 0 

This is also part of unittest2, so it is available for earlier versions of Python.

vote vote

59

load_tests is a little known mechanism introduced in 2.7 to dynamically create a TestSuite. With it, you can easily create parametrized tests.

For example:

import unittest  class GeneralTestCase(unittest.TestCase):     def __init__(self, methodName, param1=None, param2=None):         super(GeneralTestCase, self).__init__(methodName)          self.param1 = param1         self.param2 = param2      def runTest(self):         pass  # Test that depends on param 1 and 2.   def load_tests(loader, tests, pattern):     test_cases = unittest.TestSuite()     for p1, p2 in [(1, 2), (3, 4)]:         test_cases.addTest(GeneralTestCase('runTest', p1, p2))     return test_cases 

That code will run all the TestCases in the TestSuite returned by load_tests. No other tests are automatically run by the discovery mechanism.

Alternatively, you can also use inheritance as shown in this ticket: http://bugs.python.org/msg151444

Top 3 video Explaining How do you generate dynamic (parameterized) unit tests in Python?

Related QUESTION?