By Steve Helsby
The challenge on any software project is the need for quantifiable data on resources (how much, how many and how long) to be included in project plans in advance of the activities that utilise those resources. Unlike other industries such as the construction industry, in which quantity surveyors look up information in tables and can accurately calculate costs, estimation can be more of an art than a science in software development. Estimation of testing staff numbers using ratios based on development staff numbers with a simple model appears easy and tantalising but what are the pitfalls?
One Size Fits All?
There is no universal formula which can provide an estimate for the number of testers simply based on the number of developers. A number of factors (too numerous to cover in this article) will directly impact the testing effort required on a project. Here are a few examples:-
- Software Life Cycle – in the world of Agile, close working, continuous integration and in fact relatively small teams sizes may indicate a higher ratio of testers to developers compared to a larger V model or waterfall based project. For example, although a typical Agile project may comprise 5 development staff and 2 testers , it does not mean this ratio can be applied to a V model development of 20 developers implying 8 testers required based on a 2 tester per 5 developers ratio
- Application Type – this directly impacts the testing effort. A safety critical system in a specific industry will require more extensive coverage. Avionics industry software will require more stringent coverage to meet a standards such as DO-178B (Software Considerations in Airborne Systems and Equipment Certification) than say a mobile app to book a hotel. For these critical systems in fact the developer effort involved in unit testing will be proportionally higher than a non-safety critical system and the tester effort will also be higher
- Specialist Testing – where figures for tester to developer ratios are concerned, these most likely will be based on staffing numbers involved for the duration of a project. If the software requires specialist testing such as security, usability or some form of compliance testing this is likely to be required intermittently. This additional testing effort may not be factored in to a simple tester to developer ratio
- Risk Management – Ideally risks should be identified in advance, prioritised and agreed by the business, development and test managers. The risks which will be addressed by ‘testing’ should be known and identified. Two similar projects may have very different tester to developer ratios if they differ in the level of risk being addressed. The lower the risks being addressed the greater the testing effort and hence the higher the tester to developer ratio (i.e. more testing effort needed)
- Automation – Aside from the different skill set involved in automated testing, the level of automation will have significant impact. This is particularly true if the test strategy calls for a regression averse approach to testing and whether the resultant regression tests are manually scripted and manually executed or automated. Clearly with automation, there may be a lower ratio of testers to developers (less testing effort but a more specialised tester skillset)
- Requirements – The more complex the requirements, the more testing effort required. This may also mean a more complex development effort. It does not necessarily follow that, a complex set of requirements requiring more development and therefore more testing will result in a meaningful tester to developer ratio that can be relied upon for estimation. Again, two identical projects with identical complex requirements may differ greatly if one is implemented poorly, with insufficient unit testing and with a higher number of bugs in the software compared to the same software better implemented , more thoroughly unit tested and with less bugs. The latter requiring less testing effort and resulting in a lower tester to developer ratio (less testers than the buggier software)
- Type – The type of testing will significantly alter the ratio, if say the application is more in maintenance mode, hence small changes are being implemented by developers result in a proportionally small amount of development compared to the testing effort including regression testing
- Tester tasks – any ratio will increase subject to what the testers are actually doing, there are a number of factors which will result in a higher ratio of testers to developers :-
– Customer support activities including handling support calls
– Customer site visits
– Supervising beta testing
– Assisting development team and business owners in defining acceptance criteria
– Degree of participation in User Acceptance testing
– Requirements inspections
– Writing product documentation
– Deployment or product roll-out
– Tool support
So What Is The Ratio?
Whilst ratios are always useful and may be viewed as an expedient and simple way to arrive at an estimate there is a significant risk when an organization uses another organization’s ratios and then applies these to their project without regard to differences in technology, process maturity, and skill levels.
Part of the problem with identifying a reasonably accurate ratio are the not just the reasons above but also the lack of data collected by companies on this information. Thus for many, it is not an exact figure that is sought but an assurance that the testing estimates are reasonably correct and there is a degree of confidence that the testing can be undertaken within the project time frame with the staff and budget allocated to this function.
So when looking for actual numbers (ratios), it is possible to search the web and glean information, for example according to the book ‘Microsoft Secrets’, Microsoft employs a 1-to-1 ratio of testers to developers.
A separate informal poll of participants in a conference session found the most common ratio was 1 tester to 3 developers which was based on a survey of 29 organisations :-
- Minimum ratio was 0 testers to 1 developer
- Maximum ratio was 1 tester to 30 developer
- Most common ratio was 1 tester to 3 developers
My own industry experience of large complex real time systems was closer to 1 tester per 10 developers on V model and Waterfall developments (with team sizes of 10 to 30 developers). More recently, based on training courses I have run, the ratios articulated by testers on Agile development projects across a range of companies have been 1 to 2 testers for 5 or 6 developers (with maximum team sizes of 8).
So What Do We Do?
Much of the above discussion focusses on the pitfalls of using a tester to developer ratio as a crude tool to estimate the number of testers needed. So how might the testing effort be estimated?
One way forward is start collecting data for your own organisation, this could be on projects, releases or sprints in an Agile world.
Provided your organisation is not under constant change, the software life cycle and associated processes are consistent and the software being developed per project or release remains consistent in type of application, size and complexity then a number of factors may be constant from project to project. There is value in collecting information on the quantity and absolute time or percentage of a testers time spent on:-
- Requirements (or user stories) analysed
- Test cases written manually or scripted
- Test cases executed
- Bugs found and severity of bugs
- Writing bug reports and re-testing bug fixes
This data can be reviewed (say in project and release retrospective meetings) to identify variances from estimates and estimates refined for the next release. Close engagement with the development manager is also required to clarify the consistency or not of the development from one release to another. This approach may then provide a rough estimation ratio for the testing effort within a particular organisation.
What Are The Implications For Annual Budgeting Cycles
The reality is that the testing team numbers may well have been decided before details of projects are yet available in any detail. A flexible approach may well be needed whereby the historical perspective based on collected data from the previous year is used to estimate the next financial year (with adjustments based on business view of the year ahead). In addition any peaks or specialist testing may have to be addressed by outsourcing some of the testing and should be provisioned for in the planned expense budget for the next financial year.
In essence there is no ‘silver bullet’ or ‘panacea’ for estimating testing based on ratios of testing to development staff and any organisation should question the validity of such an approach if offered such a solution. Having said that an awareness of the pitfalls (some of which I have described) and a systematic approach to collecting information may enable what I would describe as very localised tester to developer ratio to be used pertinent and optimal only to that specific organisation based on data collected within that organisation.