We know that UAT is taking us into an area
where ‘rubber meets the road’ as they say, beyond this point comes actual deployment
which may initially be on a small scale followed by a rapid ramp up across the
‘user base’. The attraction of a formal process for UAT with software proven to
comply largely to a set of formal requirements and which also integrates with a
(possibly new) business process is attractive as it provides a sense of
control. However, is this what is really required and how achievable is it?
The answer to question of how much formality is needed is one of context. I have been involved in systems where UAT is formally defined in a contract with a supplier with milestone payments based on everything from the production and agreement of formal UAT specs to actually running of tests to demonstrate the system works as agreed and agreement on the system being ‘fit for purpose’ i.e. suitable for training staff on and deploying across an Enterprise. I have equally seen systems accepted with users spending an afternoon having a quick run through the main features and giving the system the ‘thumbs up’ or ‘thumbs down’.
One system I worked on was quite complex , took 9 months to develop but was put through an informal acceptance process and rejected outright as not suitable; although implemented based on an agreed Requirements Spec it was not was wanted!
A more formal approach may well have have been better in this case. What then is the key to how much formality should be applied?
The answer lies in what may be called the ‘Context’, by this I am talking about what the app or software does (it’s purpose), how it is used and by whom. Serious consideration must be given (before development starts) to the implications of any kind of failure of the system in the live operational environment. These implications may range from no impact or a minor irritation to a user, to a financial impact on a revenue stream for a business, or worse, death or injury of a person!
Consider the context of local authority website advertising things to do over the summer holidays and the implication of the software failing with the failure of software being used by a pilot on a commercial airline. Clearly the latter demands significant formality with process, documentation, change control, and traceability which can be subjected to an audit to demonstrate compliance to specific industry standards.
Risk and the management of risk also play a major part in the formality required in UAT. If the impact of failure once the system is live is high, from a business or personal perspective, then the way forward should be a risk averse approach with a controlled process driven UAT phase. The reality is, this is only normally achievable if a risk averse mind set of the business stakeholders has been applied throughout the project lifecycle, if not , UAT will be a major gating process and the system may never be deployed without major rework or may just never reach deployment !
When talking formality, the expectation is that there are written requirements (including User Stories), which have been reviewed by the UAT team in advance. Formal Test Plans are produced with agreement on specific Acceptance Criteria (including performance aspects) which must be met. All testing is documented and there is clear change control and traceability from UAT results to requirements and business process. Adopting this approach normally requires trained testers including staff from the business who may need to undertake some formal UAT training.
From a cost perspective, it has been estimated that the 20% of the lifetime cost of a system is expended during development whilst 80% occurs post ‘go live’ (often called the ‘Maintenance Phase’ ) . Thus, even if the system is not safety critical, building some formality such as implementing elements of the Fundamental Test Process and associated artefacts such as test plans, test cases and test scripts and incident report can help reduce the seriously reduce ‘maintenance burden’ and reduce the lifetime cost of a system.
I realise that some systems may be developed with little formality and UAT can be carried out informally; this may be perfectly adequate provided the stakeholders are content with the lower level of quality this affords and an informal approach is adequate for the context. However it is realistic to expect a higher level of support once system is live plus the possibility of irritation from real users discovering the bugs (by indirectly testing) the system. If an informal approach to UAT is adopted, it is always good idea to have multiple acceptance stages. The first stage being partial acceptance, followed by a limited release to users (effectively a form of beta testing). It is important to ensure all user classes are covered and the users attempt to do their actual work with this release of the system. This approach can de-risk the system significantly. After the period of running with actual users, a business and IT review should then be held and all problems raised by users discussed (particularly from the perspective of ‘impact on businesses). This review can be the basis for a ‘go/ no go’ meeting to agree or not acceptance and subsequent roll out.
So, when it comes to formality should UAT be ‘black tie’ or ‘jeans and T shirt’, the answer is that it all depends on context and risk.
Steve Helsby, Senior Trainer