Software Testing

I frequently hear people talk about unit tests when they actually mean something else. In some cases, I would think it was pedantic to argue about the specific meaning of a term like unittest, but in this case I think it often works against the Software Developer to use the term incorrectly. Below I provide a short list with descriptions to explain some useful testing concepts.

• Unit test
  • It’s important to be clear on what “unit” means
    • It means a function/method or class
    • It does not mean database, SaaS, filesystem or any other application component
  • A unit test should be runnable without any other components
  • Not all of an application can be tested with unit tests, which is why there are four other types of tests below
• Functional test
  • A functional test looks beyond the unit to include in-application interactions
  • Mocking should be used to satisfy dependencies on application components, like databases and external services
    • Use sensible abstractions, such as DAOs, with alternate implementations that are injected at runtime
  • Popular unit testing frameworks can also run functional tests
• Integration test
  • This is the first test that goes outside the application to view how it works with other applications
  • Mocking can still be used to satisfy dependencies on application components, like databases and external services
    • This can be true for both application under development and the integrated applications
  • Consider connectors and intermediate components independently
• Security test
  • This typically focuses on what happens when someone does something unexpected to your application
  • This should also include analysis of application libraries and core components that may have known vulnerabilities
  • This is phased, including
    • Analysis of the source code
    • Error states and failure modes within the application
    • Unexpected inputs to the running application
• Performance test
  • This should focus on your application
    • Test dependent systems and components independently, otherwise you may not accurately identify where to address performance problems
  • This should include various metrics, such as
    • Total time to execute
    • Memory consumed (and how it changes over time)
    • IO handles (and how they are managed over time)
    • Connections (and how they are managed over time)
    • CPU load
    • Threading overhead and system exhaustion
• Failure mode analysis (bonus)
  • What are all the systems, data, services, etc. required for my application to run?
  • For each point of possible failure, is there some way to recover?
  • How do I ensure that the user receives a sensible response, even if that response indicates that an unrecoverable error happened?
  • Are there any sensible notifications that the application can send to alert component owners to an impacting failure?

All of these types of tests, or analysis in the last case, serve a different purpose in the test plan for an application. They are all needed, but can and should be developed separately. Unit tests are a great place to start.

Hopefully it’s obvious from the above outline, but I’ll repeat myself and say that an actual database is NOT a requirement, or even appropriate in unit tests.