L&D: Knowledge Tests Don’t Test for Performance Capability

30 Year Ago

Performance Tests Do Test for Performance Capability

In the first consulting work efforts that my former business partners and I did creating Performance Tests back in 1987, we used our 4-Phase approach to Curriculum Architecture Design slightly modified, on 22 distinct Target Audiences, and then produced over 2200 Performance Tests in collaboration with the identified Master Performers.

Perf Tests

The Performance Tests fell into three categories:

  • Real Work
  • Simulated Real Work
  • Talk Through Troubleshooting

I created our methodology and then we tested it with several of the Target Audiences before we trained our staff and then did all of the targeted jobs.

After creating the Performance Tests and administering them the client was going to have me conduct a Curriculum Architecture Design (CAD) effort so they could build/buy any Training really needed. A good plan IMO.

It was a cold winter in Prudhoe Bay that November. Here next is a photo of one of my souvenirs from my part of the project.

2007-11-01 008

One interesting aspect of the effort – was that had been attempted twice previously and had been roundly rejected by those Target Audiences (both generated Knowledge Tests) who were each very technical in nature. And they hated paper and pencil tests – but loved our approach to Performance Testing.

Of course, they created the tests with our help – or vice versa.

This was an oil field, feeding the Alaska Pipeline – in the Arctic Circle – where sometimes the sun just don’t shine. And whales are sometimes carved up by the locals on the edge of the oil field – where the land meets the Beaufort Sea – as they have done for a long time – oil fields or not.

Oh. The Tests were part of a Pay Progression Program.

Meaning – you would only get a pay raise if you could prove additional competence – not in test taking – but in Performance Competence.

Meaning – we had everybody by their wallets – so to speak.

Meaning we had their attention. And suspicions.

They were a tough crowd. Members of oil field crews are known as roughnecks for a reason. But once they overcame their own suspicions – and they saw that they owned the whole shebang – so to speak – we had no trouble with participation/ engagement.

2007-11-01 004

All of the Performance Tests with a few exceptions were open book tests – just like the real world. That also was a big selling point. The tests were authentic in that way.

Speaking of the Real World

Here is a 15:15 minute:second video of my tour of the facilities where we lived and worked when we weren’t out in the environment.

It starts in our temporary office with my two business partners. They are looking for flights home to Chicago – through Anchorage after our cycle of a two week stay – one week home – and then repeat. I tour the living and eating areas and then my quarters. Talk about small – but I had been on a ship for most of my time in the Navy (1972-1975) – so it wasn’t small. It was cozy.

Note the temperature outdoors (on a TV monitor) when I approach one of the entrances and peak into the cold, dark night. At about the 3:15 mark.

Cold Cold Cold

The Book

Here is a book that provides very detailed steps and outputs for the Performance Tests AND the entire system we designed and developed for hand-off to the client for their use in operating and maintaining it all. 9 Systems – per the graphic on the books cover.

Slide4

This book – available as a paperback a Kindle, and a free PDF – is based on this project – and another in 1994 for 20 jobs on the Alaska Pipeline for more technical jobs (20) – and also a project for another client’s Branch Managers, Sales Staff and Technical Staff in the lower 48 (plus Hawaii) the late 1990s and early 2000s.

Employee performance-based Qualification/Certification Systems (2008)

Click on image to link to the download page.

Employee performance-based Qualification/Certification Systems is also available as a $15 Paperback book – and $7.50 as a Kindle – for more info and to order – please go – here.

Implications for Performance Support Efforts 

Creating Performance Support is much easier when you understand the Performance Competence requirements – and have the tests developed. Much easier.

A Funny Thing Happened On the Way to the Curriculum Architecture Design Effort

It was determined after the Tests were administered that it just wasn’t necessary to spec out and then develop/acquire any Training at all. Or any additional Performance Support items.

You see, as it was all aimed at improving Performance Competence – and it was tied to their pay – the motivated learners/Performers figured out how to pass each test … open book and viewable by anyone wanting to see them … by reading the available manuals, getting help from their crewmates, etc.

No Formal Training was necessary. For 2200 tests.

The tests themselves were all the Performance Support that were needed.

# # #

Advertisements

L&D: A Data Logic for ISD

The Data Feeds From Analysis to Design & Development

Note – not every data connection is displayed.

PACT Data Logic.png

Analysis

Target Audience data (not shown) enables generation of Performance data

Performance data enables generation of Knowledge/Skill data.

Performance data and Knowledge/Skill data enables generation of T&D Assessment data.

CAD – Curriculum Architecture Design

CAD efforts produce T&D/L&D Paths – or Performance Competency Development Paths.

CAD efforts DO NOT create any new content. They rationalize the needs with existing content and spec (specify) the gaps into Modules and Events in a logical, suggested sequence – for adaptation by the learner/Performer and their management (or however that is intended to occur).

cad-pots-20121

All of the Analysis data feeds CAD efforts and the generation of Module Specs (think: chapters) and Event Specs (think: books).

Modules later convert to Lessons in MCD – enabling reconfiguring them if the gaps in content are deemed worthy of funding acquisition/development efforts.

Otherwise they’ll exist as Unstructured OJT (my language/term for Informal Learning that preceded that nomenclature).

Analysis data also confirms the worthiness of Existing Content for ReUse “As Is” or “After Modification” or “Not Applicable” (for this effort – may be fine for others.

slide1-3

MCD – Modular Curriculum Development

MCD efforts produce new content. Think ADDIE – or SAM – or SAT, etc.

mcd-pots-2012

As MCD efforts build/buy Content – Instructional or Informational – of any/every Media and Mode – the Inventory must be updated. Version Control is an issue.

As MCD efforts potentially ReUse Content – “As Is” or “After Modification” – Content Variation … the engineering Target Audiences’ version might be slightly different than the sales Target Audiences’ version …  if authenticity is as important to you as it should be … another type of Version Control is another issue.

ReUse Event Map and 5-Tier Inventory

Content can vary slight or more – as appropriate – and versions between Target Audiences may not follow the same cadence.

As always it depends. And that is subject to change.

01-liml-66155-lessons_in_maki-sfull

And for Those Not Getting the CAD and MCD Efforts?

Not all Target Audiences are critical enough to warrant serving their needs for authentic Content.

That’s just the way it is.

Push-Pull and The 5 Tier Content Inventory Framework

Content So while less than ideal – depending on your policies and practices – the Content acquired/created for the most critical Target Audiences can be made available to others.

Not ideal – but perhaps real. And subject to change.

# # #

 

L&D: Climbing the Performance Analysis Learning Curve – Part 3

This Not a Step Per Se

It’s a positioning.

Of analysis – using AoPs and Performance Model charts – at a higher level than a Task, Task-set(s) of a Job or a Process. Those were covered in Part 1 and Part 2.

What if you needed to analyze an entire department or a function?

Here is my model for that … groupings of AoPs.

EPPA - Building Block View - Department

In L&D or T&D or a Knowledge Management department or function these Super AoPs exist at the Leadership level and the Core level and at the Support level.

A Super AoP may include many AoPs. Which I have grouped into the 3 sets of the model above.

Don’t confuse the grouping with a hierarchy – because I have found that Senior Managers focus more on Leadership AoPs. and Middle Managers focus on Support AoPs, and First Line Supervisors focus on the Core AoPs.

But – as always – it depends.

The Core Processes to be planned, assigned, managed and trouble-shot – might be:

  • Product Line Planning
  • Product Line ADDIE-like Development (or SAM or SAT)
  • Product Line Deployment/Access

So you would add those too – a 4th grouping if you will.

And maybe you also offer Services as well as tangible and digital products. Add those in too.

As always – it depends.

01-liml-66155-lessons_in_maki-sfull

 

Look at Your Organization Chart This Way

As a network of departments of AoPs. And not every department or function does all of the Super AoPs.

Say it with me now: as always – it depends.

org-chart-in-l-c-s-mode

AoP Branches from Enterprise to Division to Business Unit to Function to Department

Your Enterprise and focus might differ.

And againas always – it depends. 

It depends on the number of levels in your Enterprise and otherwise how it is structured.

EPPI Tier 1 View

In every level and every box – it’s all about Performance Competence … and that is not “as always – it depends.”

Slide3

The Big Picture of Enterprise Process Performance Improvement

I’ve learned – was taught by people and by experience – to start Analysis of the Process itself – before the Human enablers and the Environmental enablers.

Or of you prefer: the Environmental enablers and then the Human enablers.

Either way.

But IMO/IME start with the Process. Another not “as always – it depends.”

Slide1

Free Book PDFs

My 2007 book…

M-AoP Book Cover 2007

Click here for that book and others that I offer for free. It’s the third book down.

Books for Sale

The free book was updated in 2011…

2011 e D-Y M-AoPC

And the book on Analysis.

2011 b AofPCR

Click here for those and other related books for sale.

# # #

 

 

L&D: Climbing the Performance Analysis Learning Curve – Part 2

Once the AoPs Are Established – Detail the Performance Requirements and Gap Analysis

Part 1 covered establishing the AoPs – Areas of Performance – here.

Performance Model Chart

TMC SM Perf Model Chart

Note that there were several chart/pages for the Performance Model for the one AoP of: Staff Recruiting, Selection and Training. Note that that AoP could have been broken down to 2 or 3 AoPs. That is not the big deal about chunking.

The AoPs for the above example.

AoPs TMC

Example Chart/Page Progression

One – Capture the Tasks and Outputs/Measures.

My experience – I’ve been doing these since 1979 – suggests that people generally think of Tasks before Outputs.

So often I ask for a Task – any Task – within the AoP and then work upstream and downstream until the Task sounds like an Output: Draft the Analysis Report.

Slide1

Two – Using the measures determine the Typical… not everything/anything under the su – unless THAT is critical to your downstream efforts/outputs.

Sometimes it is necessary to go beyond typical to potential. Then change the header of the column.

Slide2

Three – determine the probable cause – unless you need to cover every possibility. Again, change the header of that column. And finally complete the last column of type of cause. See the Job Aid that follows for more on that.

Slide3

 

Eliciting the Performance Model Data – Including the AoPs

Use this job aid…

Perf Analysis Job Aid

Practice Using Your Kid/Summer Job From Part 1 

Using this blank chart.

Blank Performance Model Chart

Some Resources

Models and Matrices- NSPI PIJ -1984 – 5 page PDF – the first publication of the performance and enabler analysis methods for ISD, from NSPI’s (ISPI’s) Performance & Instruction Journal, November 1984.

Perf Modeling & Enabler Analysis – HR-Com – 2003 – 17 page PDF – an online publication at HR.Com in 2003 covering the analysis of both Performance Competence Requirements and the Enablers – part of my ISD (PACT) and Performance Improvement (EPPI) methods.

lean-ISD (1999)

Click on image to link to the download page.

Note: the cover design for “lean-ISD” was created by the late Geary A. Rummler.

Note: Guy W. Wallace’s book “lean-ISD” – was a recipient of a 2002 Award of Excellence for Instructional Communication from the International Society for Performance Improvement.

lean-ISD is also available as a $15 paperback book – and $7.50 as a Kindle – for more and to order – please go – here.

Analysis of Performance Competence Requirements (2011) – as Paperback and Kindle

– for Analysts in performance-based Instruction and Performance Improvement efforts.

Paperback $15 and Kindle $7.50

PACT

slide61

PACT as a brand was part of my former businesses at SWI– Svenson & Wallace, Inc. – and then CADDI– Curriculum Architecture Design & Development Institute, Inc. – and then EPPIC– Enterprise Process Performance Improvement Consultancy, Inc.

# # #

 

L&D: Get Over Yourself Regarding Performance Improvement

You Are A Support Function

And an important Support Function. But…

Most – not all – folks in L&D are not the business people they might wish to be … or see themselves to be.

IMO. IME.

L&D Impacts 2 or 3 of the Set of Variables

In this model, you can impact 2 … or maybe 3 … of the variable sets of Performance or what I call Enterprise Process Performance.

Your language and models may differ.

Slide3

L&D can affect 3 of the variables … “if” … you are capable of designing a new client Process to meet the balanced Requirements of all of the actual Stakeholders. And then can really impact the Awareness/ Knowledge and Skills of the Critical Performers in the Enterprise – and – maybe – provide/give access to some of the right Information/ Data needed.

But then … that’s the job of all of the other management in place – to take care of their own.

So 2 – maybe 3. The rest of the variables needed – per my model above – are the jobs of all of the other managers in place to put into place. To enable the Processes that they are in place to manage.

Or why have them?

Variable Set 1: A/K/S

Impacting/enabling the Awareness/ Knowledge/ Skills are what L&D should be all about. As directed by the client and stakeholders. And THAT is still shared with the other managers.

If you can’t do that very well all of the time – focus on that first.

Variable Set 2: Data/Information

L&D can also impact the Data/ Information – not by making it available – but by determining and sharing what and how the current Master Performers do it.

Sharing their Best Practices if you will.

And if the needed data and information isn’t available – the best you can do – from a L&D perspective – is to bring that to the attention of the client and stakeholders to make that happen.

They can then request and/or direct and fund IT … or whomever is appropriate. That’s not the job of L&D.

Variable Set 3: The Process Itself

Designing a Process – robust to all of the things that can go other than right – is tricky business.

That requires deep knowledge of the downstream requirements of the downstream Customer(s) and their Stakeholders – and of the Process and Tasks within that Process and the requirements of all of the Stakeholders.

L&D staff won’t know that – most likely.

A Performance Improvement function’s staff would know “how to” IMO.

Slide5

But L&D staff might be able to facilitate those in the know – to help them design a Process.

The Group Process

I’ve been using a group process and facilitating the same since 1979.

Here are some old articles about that.

CAD – Training Mag – 1984 – 6 page PDF – the first publication about Curriculum Architecture Design via a Group Process – published in Training Magazine in September 1984. Original manuscript (30 pages) – How to Build a Training Structure That Won’t Keep Burning Down.

Models and Matrices- NSPI PIJ -1984 – 5 page PDF – the first publication of the performance and enabler analysis methods for ISD, from NSPI’s (ISPI’s) Performance & Instruction Journal, November 1984.

Teaming for T&D GWW 1999 – 5 page PDF – on my story of inadvertently creating a team – out of frustration with too many revision cycles for a video script I was writing –  for training development back in 1979 – and liking the approach for using a Group Process to shorten cycle times and improve the quality of the output.

Modeling Mastery Performance and Systematically Deriving the Enablers for Performance Improvement – by Guy W. Wallace, CPT – Chapter 11 of the Handbook of Human Performance Technology – 3rd Edition – 2006.  This methodology was first published in this 1984 article in ISPI’s (then NSPI’s) PIJ in November 1984.

Past Blog Posts

My 12 Rules/Guidelines for Facilitation

The 12 Rules/Guidelines for PACT facilitators that I covered in this Blog series – was “sourced from” my writings from back in 1999 in my book: lean-ISD – in Appendices C.

And that was sourced from my field experiences in having conducted hundreds of Group Process analysis and design meetings.

The 12 Rules/Guidelines for PACT Facilitators are – and the links to the prior 12 Blog Postings are:

1. Go Slow to Go Fast.

2. Be Declarative.

3. Write Stuff and Post It.

4. Be Redundant by Design.

5. Use the Four Key Communications Behavior Types.

6. Review and Preview.

7. Write It Down and Then Discuss It.

8. Use Humor.

9. Control the Process and the Participants.

10. Be Legible on the Flip Chart.

11. Beware of Group-Think.

12. Assign Parking Lot Valets.

This Blog Post series embellished the original content already published in lean-ISD.

Free Book PDF: lean-ISD (1999)

Click on image to link to the download page.

Note: the cover design for “lean-ISD” was created by the late Geary A. Rummler.

Note: Guy W. Wallace’s book “lean-ISD” – was a recipient of a 2002 Award of Excellence for Instructional Communication from the International Society for Performance Improvement.

lean-ISD is also available as a $15 paperback book – and $7.50 as a Kindle – for more and to order – please go – here.

Everybody Wants to Rule the World

# # #

 

L&D: IME – Most “Training” Isn’t

It’s Education on Topics. Tasks? Maybe Micro.

IME – In My Experience
Slide1

But it isn’t Training – as it hasn’t/ doesn’t enable Performance to Requirements.

Again – In My Experience.

Slide1

My Experience

In the 75 Curriculum Architecture Design (CAD) efforts that I’ve conducted since 1982, and in the 50+ Modular Curriculum Development efforts (my ADDIE-like methods) I’ve found that most Learning Content/ Training Content was about Topics. Sometimes Tasks – but mostly Topics.

The last of my 4 types of formal Analysis is an Assessment of existing Content for ReUse in a new Path or Paths through the current Inventory and potential future Inventory of Content – and various levels of formal Instruction and Information. What have I found?

Topic after Topic … with no or weak links to any kind of a Process Model as a Scaffold to hang each Topic on, to place the Topic in a mental model of overall Task performance.

Even if Tasks were covered they were what I would call Micro-Tasks, and they too were not placed in some bigger picture. Some model/scaffold of the Macro Performance.

An example of a Macro Performance Model/ Process Model: Ill Bell Labor Relations at the Supervisor level:

Slide1

This model/framework was presented early in the Introduction – and thus set the stage for all learning content that followed, including the process details. the internal organization, the union, the current agreement based on current law and the current contract, and then the process tools and techniques.

It was a big poster on the wall plus the binder insert cover … it was, after all, 1991.

slide27.png

These graphics above were from the Analysis (the top graphic) and Design Documents (both graphics) from this 1991 effort.

Existing T&D Analysis

Again, the ETA is my 4th and last type of formal Analysis conducted during the Analysis Phase – as in reality analysis occurs in all phases of my 6 Phase effort.

My 4 Types of Analysis in the PACT Processes are:

  • Target Audience
  • Performance
  • Enabling Knowledge/Skills
  • Existing T&D Assessment

In CAD…

CAD Key Outputs

Clients often wanted me to start with thewir existying content – which I would gently push back on by stating that I wouldn;t know what I was looking for in that content until I had minimally done the analysis on Performance – and – the Enabling K/Ss. They almost always saw that logic.

They, of course, were worried that we’d throw the proverbial baby out with the bathwater. This wasn’t their first rodeo.

My bias has always been – if it looks close enough – use it – and then evaluate that decision with during the Pilot Test and the evaluations of ongoing Deployments. Prove it out versus prove it in (initially).

I firmly believe in ReUse “As Is” or “After Modification” – but if it just doesn’t fit – despite the title suggesting that it should fit – we deem it: “Not Appropriate.”

The ETA Process

PACT ETA Process

Documenting the Assessment

I use a form to document the assessment – which feeds a database. The following is the 2002 version of said form.

Check the final assessment in the bottom row of side 1 of 2.

Slide1

That assessment of the ReUse potential is: “As Is” (AI), or as source “After Modification” (AM), or do not use as it is deemed Not Appropriate (NA).

Adopt-Adapt

Given the Performance data and the enabling K/Ss data.

pm-ks-used-in-design

If you don’t know the Outputs and their Measures and the Tasks and their Measures – plus the enabling Knowledge/Skills related to those Outputs and Tasks – you are likely to miss The Mark.

The Mark of Stakeholder Requirements – for both Outputs and Process Tasks.

Slide5

Now – Switching Gears and Direction

I bring this up after reading Mirjam Neelen’s excellent August 29, 2017 post about…

THE QUESTIONABLE RELATION BETWEEN INDIVIDUAL AND ORGANIZATIONAL LEARNING

– and that brought to mind my prior reactions to other similar studies, articles and posts.

Training – Is Too Often a Misnomer

It may be called Training – but it is not.

Or it is just poor Training.

Which just seems all too prevalent. In my experience.

It might have been better labeled Education – where we often don’t know the terminal application exactly of the learner/Performer. Topics one might need to know. Tasks that one might need to know. Might need – because we were unsure of the terminal performance objective and process specifics. But then – that’s not Training.

But maybe they did need Active Listening. Sounds logical, doesn’t it?

I’ve written before about my experience where the ETA process in a CAD effort uncovered 28 versions of Active Listening that one client had in their inventory. Talk about Opportunity Rich – or Overload in the extreme.

And … they all were two hour modules. All were generic, canned programs not built for the type of Active listening needed and then reflected in the authentic INFOs, DEMOs and APPOs (Application Exercises).

Kinda like taking Training for somebody else’s job.

And that’s what my experience in ETA – Existing T&D Assessments has taught me.

Wisdom from the Late Geary A. Rummler

One Slide…

1981 MTEC G A Rummler Session

You can call in Training – even Performance Based Training.

But THAT does not make it so.

Look at the titles of your Learning Content.

Are they more Topic sounding than Task sounding?

I bet so.

Are INFOs, DEMOs and APPOs (Application Exercises) authentic enough?

The Video…

The slide was from this 1981 presentation at MTEC – Motorola’s Training & Education Center. The video is 45 minutes in length. Well worth the watch IMO.

Unless your Content – T&D, Learning or Knowledge Management – is authentic enough – it won’t likely transfer. And won’t have the impact – or ROI – that it might have.

If you are not going for Measured Results – the question becomes – why bother?

Measured Results

Resources

lean-ISD (1999)

Click on image to link to the download page for the FREE PDF:

Note: the cover design for “lean-ISD” was created by the late Geary A. Rummler.

Note: Guy W. Wallace’s book “lean-ISD” – was a recipient of a 2002 Award of Excellence for Instructional Communication from the International Society for Performance Improvement.

lean-ISD is also available as a $15 paperback book – and $7.50 as a Kindle – for more and to order – please go – here.

Book 6 Pack For Sale

The PACT 6-Pack – from 2011 –

for Instructional Systems Design/Development and Performance Improvement

Purchase through Amazon’s CreateSpace online E-store or Amazon.com for the Kindle versions.

These 6 titles include:

The Curriculum Manager’s Handbook (2011) – as Paperback and Kindle

– for Curriculum Managers, L&D Managers and Chief Learning Officers – to define your job and function in Performance Competence terms and identify targets for improvements.

Paperback $15 and Kindle $7.50

Analysis of Performance Competence Requirements (2011) – as Paperback and Kindle

– for Analysts in performance-based Instruction and Performance Improvement efforts.

Paperback $15 and Kindle $7.50

Performance-based Curriculum Architecture Design (2011) – as Paperback and Kindle

– for the Project Planner and Manager, plus the Designer of a Curriculum Architecture Design effort for produce performance-based Learning Paths and Planning Guides.

Paperback $15 and Kindle $7.50

Performance-based Modular Curriculum Development (2011) – as Paperback and Kindle

– for the Project Planner and Manager, plus the Designer of a Curriculum Architecture Design effort to produce performance-based, modular Instructional Events of any mode and media.

Paperback $15 and Kindle $7.50

Developing Your Management Areas of Performance Competence (2011) – as Paperback and Kindle

– a model for individual use or Enterprise-wide use to define a Manager’s Performance Competence requirements, assess for improvement areas, and then develop a Management Development Plan.

Paperback $15 and Kindle $7.50

From Training to Performance Improvement Consulting (2011) – as Paperback and Kindle

– a guide for a leadership team to take their Training/ Learning/ Knowledge Management organization and Stakeholders on a 2-Step Journey from Training to Performance Improvement Consulting.

Paperback $15 and Kindle $7.50

# # #