Testing, Testing – Testing, Testing

Mostly from my book: lean-ISD (1999)

Testing “prior” to general release for deployment or access – is an important step – “most” (but not “all”) of the time. More on when you might not, later.

And please note: I believe in full-destructive tests in pilot-testing my instructional content! It all depends on what you are trying to prove and/or improve. Me, I like to break it if at all possible BEFORE making the content ready for deployment and/or access.

Testing happens in two of the PACT Processes phases in MCD – Modular Curriculum Development/Acquisition – phases: Phases 4 and 5. Testing the piece-parts in Phase 4 – and Testing the whole enchilada – as they say – in Phase 5.

Tasks for MCD Phase 4 – Development/Acquisition

The tasks of Phase 4 for Modular Curriculum Development are organized into four subphases. Some changes to the tasks presented are inevitable, depending on the deployment platform for which the team is building. For example, at a microlevel, development activities for self-paced readings are different from development activities for an interactive web program. The framework of Phase 4, however, is appropriate to all media, modes of deployment, and deployment platforms.

MCD Subphase 4.3 – Developmental and Alpha Testing

During developmental and alpha testing, developers plan and conduct formal and informal testing of the piece-parts of the T&D. Not each T&D module, or lesson, or activity needs a formal developmental test―that would take too much time and too much money. The Modular Curriculum Development project manager plans the appropriate tasks for this subphase.

Alpha and Beta Testing

Most products, training or not, are tested during development. The first round of formal testing is generally called alpha testing. The second round of organized testing on the more finished product is called beta testing. Beta testing is what ISD professionals usually call pilot testing.

We feel that training developers should perform internal and informal developmental or more formal alpha tests during this phase as they see fit. For example, it’s usually worthwhile to try out exercises to ensure that instructions are complete, that learners have enough information to answer questions, that exercises are not too difficult or not too simple, and so forth.

However, some of the time the structure of the content—and the way it’s expressed—is rather arbitrary; one approach will work just as well as another. Be aware that if you ask for opinions on content and expression during a developmental test, you will surely get those opinions, along with the consequent rework (and potential schedule slippage).

Unless you feel there are substantive issues on which you would like interim feedback, it may be better to let the pilot test in Phase 5 give you the feedback you want and need. We suggest that for Phase 4 you subscribe to the realistic notion that you will deploy imperfection and then continuously improve, rather than deferring deployment for perfection. That continuous improvement is what Phase 5 is all about.


We also have an opinion on whether to conduct those infamous, time-consuming, unnecessary walk-throughs of each and every page (or screen, etc.) of the training under development. These are a developer’s nightmare.

A walk-through usually degenerates into “The Great Wordsmithing Contest of Arbitrary Choices and Developer Disempowerment.” In our experience, very few meaningful changes occur during a Phase 4 walk-through. In fact, a walk-through usually increases cycle times, increases costs, detracts value, and demeans developers through the implied micromanagement of their work. Maybe in your situation they are needed, but I like to avoid them.

Most of the Testing effort , the integrated testing of “the whole enchilada” happens in MCD Phase 5…

MCD Phase 5: Pilot Test

Overview of Pilot Testing

Description – In Phase 5 of Modular Curriculum Development, the training is delivered during a pilot test, and extensive evaluations are conducted.

Key Activities/Tasks – Project activities in this phase include preparing for the pilot delivery (conducting train-the-trainer sessions, as appropriate); conducting the pilot test; evaluating the results of the pilot test; documenting the evaluations; and developing revision recommendations for the Project Steering Team. The Project Steering Team turns the recommendations into revision specifications.

Key Outputs – The outputs of this phase include

  • The Pilot-Test Report
  • A Project Steering Team presentation

Tasks for MCD Phase 5 – Pilot Test

The tasks of Phase 5 for Modular Curriculum Development are organized into five subphases.

MCD Subphase 5.1 – Prepilot

In this subphase, the project team prepares for the pilot test. Preparations include coordinating logistics, producing materials, coordinating the personnel required for the pilot, setting up the pilot-test location, and doing final readiness checks.

MCD Subphase 5.2 – Pilot Deployment

During Subphase 5.2, pilot testing is conducted in circumstances that replicate how the T&D will be deployed once it’s ready for general release to the marketplace. Basically, the project team conducts the pilot test, coordinates the resolution of issues that arise, and conducts evaluations on the T&D being pilot-tested. Tasks in this subphase will vary depending on the chosen deployment platform and need to be adjusted accordingly by the project manager.

MCD Subphase 5.3 – Postpilot Revision Recommendations

From this subphase comes a draft of the revision recommendations of the project’s ISD professionals, based on a thorough review of the data collected during the pilot test. These revision recommendations are later reviewed and processed by the Project Steering Team.

MCD Subphase 5.4 – Pilot Phase Gate Review

In this subphase, the last formal meeting is held with the Project Steering Team for the Modular Curriculum Development effort. The Project Steering Team receives an overview of the phase along with the ISD Team’s revision recommendations. The Project Steering Team’s decisions and reactions to the recommendations evolve into a set of revision specifications for use in MCD’s sixth phase, Revision & Release.

MCD Subphase 5.5 – Post-Gate Review

In this subphase, the project manager distributes the revision specifications and obtains sign-offs on the progress completed during the phase.

When You Might Not Test

Testing is expensive and take money and time! If you are developing the 3rd batch of procedure content – and you have a good process with appropriate Developmental Testing and experienced people – perhaps you can skip the formal testing of a formal Pilot-Test.

What to Call This Testing

I’ve had clients remark that they hate Pilot Tests – and when I probed for “why” they told me that it was an excuse for deploying crap, as in “don’t worry about it not being right , it’s just a Pilot Test.” So I asked the one client that said exactly that – what they called post-development and pre-delivery efforts to assure quality. He told me and we renamed the Pilot-Test to be that (this was in the mid-1980s and I don’t recall the label he preferred and that his enterprise would better understand).

If your clients really hate the idea/concept – and just want to rush to deployment/access – then your first real deployment or access – can serve as your unnamed Pilot-Test. If there are issues (problems/ opportunites) with the content – or anything – then you’ll have to recall that content – fix it – and then re-deploy/make accessible – and then move on.

Regardless of exactly what you call it or do it – get ‘er done! And minimize any/all embarrassments and other issues with faulty content – or configuration – or other media/mode, etc.

Soon – in a Follow-Up Post: Evaluating the Pilot-Test and Debriefing the Pilot-Test Team


My book: lean-ISD is available here as a free 404-page PDF – and here as a hardbound and here as a Kindle.

It – lean-ISD – presents the PACT Processes for T&D/ Learning/ Knowledge Management – a set of flexible methodologies I’ve been using since the early 1980s.

The 5 methodology-sets of PACT – with 3 levels of ISD

Follow one of my general rules: Adopt what you can – adapt the rest.

# # #

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.