- Lack of organizational maturity and no governance framework for IT testing
- Non-availability of toolsets to automate these tests for some packaged applications
- Lack of skill-sets to perform these activities and interpret the results
- Over dependence on product and IT services vendors who historically treat testing as a mundane task.
- How fast can a system respond to a single user request or perform a single automated activity? Normally I call this as performance testing for simplicity (many may disagree). Identify components that can possibly attribute to the unsatisfactory performance, profile these components for possible reasons (ex. identify if the database call takes for more time, or a thread is waiting for an object lock to be released etc) and look at possible ways to improve them, at times revisiting the design if required.This is more of white box testing unlike other testing like load and stress testing that are kinds of black box testing.
- What is the average time taken to complete a specific activity (process a user request) under normal load? This should be well within the latency SLAs. If not, look at steps to improve the design.
- What is the load that the system can take without misbehaving and still give good enough performance? This helps in terms of identifying the stability of the system under peak load. Measurement of resource consumption (Memory, CPU, IO etc) and understanding the resource constraints helps us knowing ways to scale and size the system. This also helps in identifying the system parameters that should be monitored after the system goes live.
- Identifying limits of the system, sometimes called stress testing…
- Fail-over tests for the system; both local and DR based on deployment strategy, architecture and infrastructure
Talking of application performance, there is a great post on top ten performance problems taken from Zappos, Monster, Thomson and Co. I have worked on resolving performance problems of many applications and platforms that are mostly written in java. I have seen platforms like webMethods always cache the services (code) in the memory. This results in huge amount of memory being consumed after the system startup. Of course there are advantages of having the whole object tree loaded in memory to improve performance. However, this is a double edged sword. Having more objects in memory means heap size being bigger and hence the GC cycle being longer! Some may say it’s the language to blame. However, this is not exactly true, as mentioned in this interesting post on how everything that’s written is C, C++ is not always faster than Java or other languages.