Beware the SME Interview/Observation: Love your SME – But Don’t Trust Them 1 on 1.

In response to this month’s BIG QUESTION at Learning Ciruits Blog: on
Working with Subject Matter Experts

Recommendation: Collaborate with your Customer in Teams If Possible. Avoid One Off”s if you can.

Don’t trust them one on one? No. Don’t. For they know not what they do.


I’ll answer that later – but a hint: it’s from the findings of recent research. But first, two stories to start positioning “where I am coming from.” In case that provides you with an advanced organizer for my perspective. Then keep your own – or change it.

And remember: as always, it depends.

Story 1
Back in 1981 I was at the predecessor to Motorola University – at MTEC – Motorola’s Training & Education Center. It was there that I was given the Corporate SME for a couple of projects – and I/we got severely “burned” in the Pilot-Test session. We were so “behind the times with our Content.” The SME – I found out later (too late) – had the job we were focused on, and the job they were the SME for, 7 years earlier. So our content was about 7 years out-of-date. BURNED.

Back to Rework City.

LEARNED right then for next time: Ask for people who were truly “experts” in current performance in the current context. And because I was already aware of the research about working with SME’s I also ask for groups of them, to be facilitated as a group, to come consensus on the data I was seeking. More on that too later.

Story 2
In 1984 I was a consultant working for Alcoa – my third assignment with this organization – FRP – Flat Rolled Products – and my assignment was to help them develop a 2 week course to overview the FRP Process for the plant engineers from around the world. There were to be 12 instructors from the Labs. In the 1st Project Steering Team meeting I asked “about how much content are we trying to boil down to the 80 hours?” The answer, after a little debate/discussion, turned out to be about the SWAG of: 800 hours. CONCERNED.

LEARNED previously: Get agreement from the key players and stakeholders on how to prioritize content. I left that meeting to go to my client’s office where I drew a Matrix on his flip chart easel (1984 remember?) and suggested how I might matrix the “Impact” of all of our potential content against both the key Measures and the “Process Flow” of the FRP Process – and then use that to prioritize and allocate time (a political battle that I wanted to avoid) to each of my 12 instructors from the Labs.

We completed the Impact Matrix (it later was on the cover of the big binders used for the PG and FG) in less than an hour – as he really thought through the H’s – Highs, M’s -Mediums, and L’s – Lows. And don’t forget the 0 – or – zero. But it was quick because it was simple.

My client was so impressed with this simple Matrix that he went to the head of the Labs the very next day and conducted a quick review of all labs projects to see where exactly they fit in this matrix.

Projects were stopped and new ones started at the Labs as a result.

A big win – and helped me deal with all of those super star researchers at the labs – who all thought their current project focus/foci was most/more important than others. They can’t all be #1. And “my” allocations of time for them depended on the consensus of that matrix.

More on that particular story here.

Having Been Burned I Hope to Have Learned
More is learned in rough waters than during smooth sailing.

So I don’t mind a rougher/tougher approach to dealing with SME – or as I prefer: Master Performers. Rougher for me than for them. Yeah – it might be mostly semantics this SME vs the MP – but as a colleague once quoted an anonymous source: it’s not just semantics, it’s always semantics.

So I go for facilitating a group of MP (Master Performers), as well as SME, and perhaps management of the target audience, and even novice performers with some experience on-the-job.

So what’s the difference (in my mind) between SMEs and MPs?

Master Performers – MPs were doing the job to a level of mastery performance the day before they started working with me.

Subject Matter Experts – SMEs have expertise in some aspect
of the performance, such as the policies/procedures, the tools (and even those coming down the pike that the MPs perhaps do not understand in detail).

I use templates on flip charts to capture data, and post that on the wall so that the data doesn’t get buried and forgotten. I arrange the chairs/tables for maximum visibility of those charts on the wall, and the flip chart easel that I am using to capture the data for that moment.

With MPs (and the others as needed by this project’s situational context) I establish AoPs and then complete Performance Model charts on each AoP. Then we use the PM charts to systematically derive the enabling K/S for an ISD effort, and/or other enablers if a broader HPT/Performance Improvement effort.

The AoPs
The Areas of Performance segment the entire scope of the project into buckets. Also known as Major Duties, Key Results Areas, etc.


The Performance Model Charts
The Performance Model charts capture the ideal performance as articulated by the Master Performers. They tell you what tasks they perform per output, and how those outputs are measured. RIGHT THERE ARE YOUR TERMINAL LEARNING OBJECTIVES.

The Knowledge/Skill Matrices
The K/S Matrices capture the K/S that enable the performance on the Performance Model charts. These are your enabling learning objectives.

Who says so? The Master Performers do, in consensus? Who else would you trust?

But as always, it depends.

Why In Teams and Not In One-on-Ones?

Practicality (Reason #2)
Let’s say you are NOT rapidly chasing the content for another 1-to-2-hour module. If that’s NOT the case then this reason does not apply. The other still one does.

Let’s say you need to revamp/refresh the entire 150 Event curriculum architecture (the T&D/Learning Path/Menu) for a key job were the thousands of performers’ variance from best/acceptable practices and the scary (to management and other stakeholders) impacts to Rewards potential and the Risk potential (two sides of the same coin). Mistakes by this group of performers could cost the Enterprise hundreds of millions of dollars annually.

You cannot take the time to spend three years to interview and then observe/watch them oversee/orchestrate the entire NPD Process – the New Product Development Process. With the dozens of other players on their NPD teams. If you interviewed a dozen of the best, one on one, and then compiled all of their inputs (answers to your questions) you might find conflicting overlaps, gaps, and a need for reconciliation, somehow, someway. So that’s not practical for very complex performance.

And if you cannot predict the cycle time and costs for conducting your analysis of say, BRAND MANAGEMENT at an auto firm, then you should not get that work. I’d conduct the analysis in a 4-to-5 day meeting (done it before) and then do several days assessing existing content for it’s re-use potential.

An Aside – Story 3
I had found myself on the 7th iteration of a Video Script back in 1979/80 – early in my career – and I had to put a stop to that nonsense and waste of everybody’s time, and I brought in all 5 SMEs to one room where I facilitated them through a systematic process to address the script and all of the ancillary issues that had crept in to our development process. (I hate Rework City)

It was painful, but much quicker. It was worth the trade-off. To me anyway. You may think differently. And that’s OK.

Interviewing and even observing one or many people is OK – but it takes longer and therefore also costs more. And reconciliation of “conflicts data” is often very difficult to get resolved – unless I have those in conflict in the same room/space/call (F2F is but one way to do this – and perhaps the best in most situations – but that’s another Blog Post).

If I can get the data I need for Design in one meeting – be it 1 day or 2 or 3 or 4 or 5 days in length…or more – then all the better. It is after all about getting the data needed for design. My design process is quirte structured and involves in “processing” the analysis data collected. The Design Team is a sub-set of the Analysis Team with no new players (unless thouroughly briefed).

Then Design data is produced to generate the content in Development. Etc. Etc.

One step at a time, and in la inear fashion – if we can – because that’s the shortest time frame – and time IS MONEY – so the shorter the cycle time the less the costs!

Yes I know – I prefer a linear approach to ISD especially when using some sort of ADDIE-like model as the planning and management framework. I prefer it to help make the effort more predictable in terms of time burden and schedules on busy people, the costs to be incurred end-to-end, and what exactly will be produced (a new system/organization/etc. “stood up” as they say in the military) – and then some specifics on what might be done next (Ongoing Operations – and then, Continuous Improvement anyone?).

Accuracy/Completeness (Reason #1)
But the #1 reason we should not go with inputs from a single SME or MP is that experts tend to miss up to 70% of what a novice needs to actually perform.

To paraphrase my friend Dr. Richard E. Clark of USC in this video clip: experts are on “auto pilot” their performance is automatic and they don’t know what they do.

Check out about 5:20 in this video. This is were he states that “up to 70%” figure.

I’d heard early in my career about this – about SMEs or anyone – not being able to describe fully all of the steps, the behavioral steps and the cognitive steps, adequately (from Harold Stolovitch back in the mid 1980s at ISPI-then NSPI) – along with the stories of the unreliability of eye witnesses in court and YIKES!!! – and so I always looked to avoid it – that one-on-one approach to analysis – not always possible to avoid – and so you do the best you can with the assignment and the resources and the constraints at hand.

So interviewing one, and/or observing one is problematic. That’s why many are using Cognitive Task Analysis methods. I don’t know them well, and probably do need to dig deeper into it.

The methods I use are earlier (I believe) than Cognitive Task Analysis – and my PACT Process for MCD systematizes/mechanizes the analysis and design steps, and empowers developers to then create powerful “active” learning (training) content based on the design Specs – focused squarely on the terminal learning objectives – which were captured in the Performance Model charts during analysis: perform these tasks to produce this outputs to stakeholder requirements – for both process and product/output.

Performance Competence for me is: the ability to perform tasks, to produce outputs, to stakeholder requirements. If one person cannot give me but about 30% of how they really do it and think about it as they do it – I’ve got to use another approach.

The Group Process
I was a co-author for a Training Magazine article published in September of 1984 on this group-process approach. It’s very much like getting a team together to map a process, address a problem, etc. Teams are collaborative and produce more than one person alone could – they are just harder to work with. Two heads are better than one – they just take longer to produce a consensus output – if that was the goal /so perhaps not.

I like to do my business with the analysis team, the design team, and the development teams this way. Faciliate them in groups – of the right people – and have them generate data in and or use that to extend the data. The teams seem to always like this by the middle of our time together if not earlier. But some groups are tough. Rough waters and all that. It’s great to facilitate them to a concensus – which at time just might mean there are more than one way to get something done.

I let redundancy like that in the analysis or design phases get sorted out in the downstream reviews – and let the Project Steering Team make those business decisions. Those are not ISD decisions.

Also I prefer to hold my Project Steering Team meetings as a group F2F – especially if they don’t know each other well. If they do know each other then I trust “distance reviews” fairly well (even though you cannot look into their eyes for signs of comprehension).

Not everybody needs to be in the same room for their meetings. They aren’t processing data as much as reviewing and sanctioning it – and what we tell them we’re going to try to do with it – if the data processing team in the next step allows.

I tell clients who are hesitant: it’s benchmarking best practices for this job – or the team’s jobs collectively in a process or several/many processes.

Other sources for this approach – how I work with Master Performers and SMEs, etc., – besides this Blog and the EPPIC web site – include 4 of my 5 books. 3 of the 4 books are available as free PDFs online at the EPPIC web site.

Plus 12 Audio Podcasts you can download, #8 is most relevant with this Blog Posting.

So if your approach to content development involves interviews and/or observations with an SME, or even a MP – don’t trust them one on one.

It’s not their fault. they’ve probably been operating on auto-pilot for a long time. And that’s when bringing in Novice Performers can be helpful.

Collaborate with your clients, SMEs and other business partners – BUT OWN your processes for producing your outputs! And to get more accuracy when focused on terminal performance – use groups of MPs/SMEs and others appropriate to your situation.

# # #


Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s