When You Are Confronted with a Survey as Valuable as “Bueller Bueller Bueller”

This is how I felt this morning…

I dutifully – in a catch up mode after e-shipping off a set of client deliverables this mid-morning – completed several surveys sitting in my “Read Next” folder in my email system. Some were several weeks old. Some of the senders had sent a couple of reminders.

I didn’t think any of the surveys were very good – and when I complained to a peer – she commented about one that we shared in our email system – and shared the feedback that THE organization sending out this survey had received a couple of years back on this annual effort – from a long-time stakeholder – who knows what they are talking about…let me share…

The problem is with the words used in the scale. This typifies the problems that arise when questionnaire makers insist in making up scales, particularly Likert-Like scales rather than staying with convention and using the Likert Scale itself or a thoroughly researched Likert-Type scale. I suspect that if one ran a scale reliability test on these data, there would be problems.

I have been teaching a course in survey research methods for 20+ years and it continues to surprise me how people make up questionnaire items, accompanying scales, and so on totally ignoring the long established and well researched “rules of the road” and create mostly worthless questionnaires. Acting in ignorance, they get results and believe they are meaningful. It is akin to teaching school – we all went to school so we are experts on teaching. We all have seen and filled our scores of questionnaires, so of course we can make one up and use it. The items and scales make perfect sense to us, so they will to everyone.

As your discussions indicate, the scale in this study is not valid or reliable, and the questionnaire results are not really very useful.

 Well, that was a complete waste of MY time – probably for all 3 surveys.

I dislike when THAT happens.

And the fact that this survey will lead to invalid results – that may or may not be acted upon – is more than just a buzz kill.

It diminishes the organization, and its employees/ members/ associates/ customers – and other stakeholders.

Hopefully – as in the past – we’ll all see evidence that this too shall have no effect – which I guess – is the good news.

# # #

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.