Software Testing: Category-Partition Method Example 1

Let's say we're asked to test a function which calculates the factorial (n!) of a number n, returning 0 for negative inputs and -1 when the result is out of range:

  int factorial(int n)

1. ITFs: This function does only one thing, so we identify one Independently Testable Feature: that the method correctly computes and returns the factorial of its parameter, n, or returns an appropriate error code.
2. Parameters and environment: The parameters here are pretty obvious — there's only one, n.

   What about environment? Is there anything about the containing system (operating system, computer, language, etc.) which might affect the behaviour of this function? I can think of one possibility, which is that int has a limited range: on a 32-bit computer system, C's int has a range from -2³¹ to 2³¹-1 (-2,147,483,648 to 2,147,483,647). On 64-bit computers the range is from -2⁶³ to 2⁶³-1 (-9,223,372,036,854,775,808 to 9,223,372,036,854,775,807). Those are big numbers, but factorials are big as well: 13! is too big for a 32-bit int, and 21! too big for a 64-bit int. Java defines its int to be 64-bit on all systems (note that the “specification” above is very brief, and does not specify what implementation language we're dealing with, or what “out of range” means). So let's do two things:

   1. identify the capacity of the system's int as an environment factor, and
   2. declare an assumption that the factorial function is implemented in C or something similar (the fact that it's described as a “function” rather than a “method” might be a hint that it's not an object-oriented language), so an int might be one of several different sizes depending on the platform.
3. Identify characteristics of inputs and environment: The characteristics we're concerned about here are those that will affect the behaviour of our function: is there a range of inputs for which its output will be well-defined? Un-defined? Out-of-range? Going back to the definition of factorial, we see it's only defined for positive integers, so we've got an immediate characteristic that negative numbers have undefined factorials. This corresponds to a return value of 0 in the spec above. Next, we've got out-of-range issues with large factorials. How large is too large? This will vary from platform to platform: on 16-bit systems 8! is out of range, on 32-bit systems 13! is out of range, and on 64-bit systems 21! is too big. Our characteristics for n are:
   • negative
   • in range (varies according to capacity of int)
   • out-of-range (varies according to capacity of int)

   And for our environment, we know that chips nowadays usually have an int of size 32, 64 or (mainly in embedded devices) 16 bits. There's no obvious interesting characteristic other than this.

4. Define partitions/value classes: So our partitions naturally follow: we have three environments to test in, 16-bit, 32-bit and 64-bit; these are discrete and result in different behaviour so we can reasonably call them three partitions. For n we also have three partitions: negative, in range and out-of-range:

   Value classes for environment — size of int:

   • 16-bit
   • 32-bit
   • 64-bit

   Value classes for parameters — n:

   • n < 0 (undefined: return 0)
   • 0 ≤ n ≤ max
   • n > max (out of range: return -1)

   Where max is the largest factorial we can compute in the relevant environment: 7, 12 or 20 according to the size of int.

   Note that when we get to looking at values in a continuum, boundaries between partitions become very interesting. We want to be sure that behaviour around boundaries is well tested. So we might expand our value classes for n a little:

   Value classes for parameters — n:

   • n < -1 (undefined: return 0)
   • n = -1 (undefined: return 0 — just below boundary)
   • n = 0 (just above boundary)
   • 0 < n < max
   • n = max (just below boundary)
   • n = max + 1 (out of range: return -1 — just above boundary)
   • n > max + 1 (out of range: return -1)

   Here we're now testing exactly on both sides of our boundaries, as well as further away — the downside of this extra care is that we're now looking at seven value classes for n instead of the previous three.

5. Generate test case specifications: We have three different environments and seven different value classes, so this means 3 × 7 = 21 tests, on the face of things. It's not quite so bad in practice though: we can reasonably expect that n < -1, n = -1, n = 0, 0 < n < max (for the smallest max, 7) and n > max + 1 (for the largest max, 20) will behave pretty much the same way in all environments. So we could start our test specification out with these five:

   Test case	Size of int	Value class for n	Expected result
   1	any	Value class n < -1	0 (undefined)
   2	any	n = -1	0 (undefined)
   3	any	n = 0	1
   4	any	0 < n < max (for all environments)	varies (correct answer)
   5	any	n > max + 1 (for all environments)	-1 (out of range)

   That leaves us trying to cover n = max and n = max + 1. max varies according to the size of int, so we do need 2 × 3 = 6 more cases for these:

   Test case	Size of int	Value class for n	Expected result
   6	16-bit	n = max	(correct answer)
   7	16-bit	n = max + 1	-1 (out of range)
   8	32-bit	n = max	(correct answer)
   9	32-bit	n = max + 1	-1 (out of range)
   10	64-bit	n = max	(correct answer)
   11	64-bit	n = max + 1	-1 (out of range)

6. Select test cases: From the above specification now all that remains is to select individual test cases for the value classes, and provide expected answers:

   Test case	Size of int	n	Expected result	Notes
   1	64-bit	-100	0	Value class n < -1
   2	32-bit	-1	0	Value class n = -1
   3	32-bit	0	1	Value class n = 0
   4	16-bit	5	120	Value class 0 < n < max (for all environments)
   5	64-bit	99	-1	Value class n > max + 1 (for all environments)
   6	16-bit	7	5,040	Value class n = max, max = 7
   7	16-bit	8	-1	Value class n = max + 1, max = 7
   8	32-bit	12	479,001,600	Value class n = max, max = 12
   9	32-bit	13	-1	Value class n = max + 1, max = 12
   10	64-bit	20	2,432,902,008,176,640,000	Value class n = max, max = 20
   11	64-bit	21	-1	Value class n = max + 1, max = 20

7. Execute tests: Now we've produced our test suite of 11 test cases, we need to execute it. In order to do that we need to set up three test environments, one 16-bit, one 32-bit, and one 64-bit. Then we can execute the tests for each system as apropriate.
8. Evaluate results: If we ran the tests, we would then evaluate the results: any failures should be noted, and it may be useful to write an overview if there are any obvious patterns (perhaps all 64-bit tests failed — maybe the system doesn't work in a 64-bit environment after all?).

Version 1.1, 2010/01/29 17:37:05

Home : Teaching : Courses : St : 2009-2010