I think it was Don Norman, now a noted usability expert, who made a useful observation about this...
To catch nmost of the problems, in most cases you only need to observe 5 inexperienced people trying to use the equipment/software/etc. If you observe those 5 people sensibly, there's not much need for large-scale and very formal studies.
Yes and no. It depends on the scope of your product and the quality of the usability panel you employ. 5 is perhaps ok for a specialised product or an internal system.
The problem you have is the inevitable bell curve. A panel of 5 doesn’t necessarily have a high enough distribution of dumbasses and people who know what they’re doing. The middle bit is mostly inconclusive as they are just zombie robots.
Yes and no
I think for most (not all) situations, the first 5 naive users will give the feedback on the items that require most significant changes. Beyond that it is into diminishing returns - the 80:20 effect and all that.
It does presume choosing representative users and giving them representative tasks, not just anyone doing an "unboxing and first impression". There are too many videos of the latter, and they are all ignorable as thinly disguised press releases. "Representative" should include both new customers and existing customers, inexperienced and inexperienced customers.
I wonder what would have been observed with the circular keypad? Clearly it is usable in the sense that it can be used to enter the correct values, but desirability/speed is a separate issue.
If you are after a true mass market product with knuckle dragging customers, then you have more to lose and so it is worth spending more money and time on surveys. But then I'm sure MS did that with Win8!
The problem is this: That 80% is the difference between a product that mostly works and a product that is a joy in the hand. The difference between a Yugo and a Rav4, and again between the Rav4 and a Testarossa.
There's "It works. Sortof. Ship it." Then there's "It works right. We're ready for the big release." Then there's "OMG, this thing is almost PERFECT; it's the best thing since electric lights and running water! I can't WAIT to bring this to the people!"
Used to be we strove for "OMG!", we expected and usually got "It works right." and occasionally ran short and had to settle for "It works. Sortof."
Now we feel lucky if we get "It works. Sortof.", "It works right." is just for silly perfectionists and "OMG!" only EVER happens by accident.
There's an episode of South Park called "Raising the Bar"; it is highly critical, even downright mean towards obesity as it is happening in America. However... it is spot-on in regards the general nosedive that everything has taken in terms of what is considered "minimum acceptable".
The main reason is the shift from a manufacturing economy to a consumer economy... but that's a reason, not an EXCUSE.
If you've never made ANYTHING, how can you have pride in your work? How can you even know what a minimum standard IS?
mnem
When did basic competency at one's job become "a goal"? When did a dumb blank stare become an acceptable defense for global incompetency?