Welcome back to this series of articles looking at document automation. If you haven't seen them, please do take a look at my previous posts around the scoping and development stage of automation.

If there is one element of document automation that is overlooked the most, it is testing. In this post I'll dive into some of the tried and tested methods and pitfalls to avoid in this crucial step. 

What is testing?

At CCAS we aim to carry out 3 types of testing:

  • Quality assurance testing
  • User acceptance testing
  • Regression testing

Let's explore each of these and then look at the process.

Quality assurance testing (QAT)

QAT is concerned with ensuring that the output of the development complies with the set of requirements. If you want to be able to add the name of a party to a document, QAT makes sure that (i) the question prompting you for the name appears in the interview; and (ii) the response given appears in the document at the required location. This kind of testing is done as part of business as usual for the team when automating documents.

User acceptance testing (UAT)

UAT is concerned with ensuring that the output can work in a real-world scenario. Often when writing requirements for templates we focus on a core set of requirements. The complexity of some finance and legal documentation means that it simply isn't practical to define up-front all the different answer combinations and the resulting outputs. UAT tests that the most common user-flows and scenarios are working as expected.  

Regression testing

Regression testing comes in when we modify existing templates. One example of when this might apply is if there has been a change of regulation or a policy has been amended that needs to be reflected in a template. Regression testing ensures that any pre-existing functionality (e.g. the party name referred to in QAT above) still works as implemented. Essentially we replay the same scenarios that we tested in UAT and make sure that we get the same output (but reflecting the change we've made). 

Process

Testing can be by far and away the most time-consuming activity. How thoroughly templates are tested depends on the complexity of the template being automated and the customer's risk appetite. A simple NDA with a few inputs can be tested in a few minutes but a large finance document with hundreds of potential inputs can take several months.

At CCAS we have streamlined our testing process by developing a unique platform on which we record and process test results. We have found this to be a very effective way of capturing and processing test results and collaborating on the development of templates. 

Takeaways

When contemplating whether to automate your templates make sure that you:

  1. Don't be too detailed about your requirements. Automation should be iterative so avoid falling into the trap of trying to nail every detail down at the outset. You'll spend a lot of time upfront only to find that as you get to the UAT stage, things need to change. 
  2. Consider what your organisation's risk appetite is and therefore how rigorously you want to test your automated templates.
  3. Build-in enough time for testing!

Look out for my next article in the series where I'll be focusing on the penultimate and probably the most important stage of the automation process. Sign-off. 

Get in touch

If you've found this article helpful and would like to discuss our approach to template automation further, please get in touch by emailing sales.ccas@cliffordchance.com.

Article Series

How to automate documents - Part 1: Introduction

How to automate documents - Part 2: Preparation 

How to automate documents - Part 3: Scoping

How to automate documents - Part 4: Development