Concepts of Evaluation Methods

From openSUSE

Back to Main Page

Evaluation Methods Part 1: Expert reviews vs. standardized questionnaires

Teaching Units

Topics

Expert ratings

Heuristic analysis 

Heuristic evaluation is a form of usability inspection where usability specialists judge whether each element of a user interface follows a list of established usability heuristics. Expert evaluation is similar, but does not use specific heuristics.
Usually two to three analysts evaluate the system with reference to established guidelines or principles, noting down their observations and often ranking them in order of severity. The analysts are usually experts in human factors or HCI, but others, less experienced have also been shown to report valid problems. (Source: Usability Net - Heuristic Analysis)
Find more information in articles and papers provided by J. Nielsen here.

Expert evaluation 

Expert evaluation is similar to a heuristic analysis, but it does not use specific heuristics in the background. TBC...

Cognitive walkthrough 

This method will analyse the usability with the help of a "walkthrough" that a fictive user will make during the use of the software. Can the user solve his problems? Can he forward to the next step of interaction without problems? Can he achieve his goals? And is the user heavily loaded, e.g. can he keep his goals in mind during the use of the software?

Standardized questionnaire

A standardized questionnaire can defend the criticisms of subjectivity. The surveys can be based on theoretical or empirical background.

ISONORM 

An article about validity and realiability of this questionnaire (pdf - German text)
The questionnarire in German language for print out (pdf - German text)

Ergusto (Ergonomic Customizing für SAP-Systeme) 

It's great, Prof. Pümper published a presentation file with pre- and post values ... Ergusto Presentation (pdf - German text)

Isometrics 

"The result of IsoMetrics1 was a questionnaire (IsoMetrics) operationalising the design principles of ISO 9241 Part 10. There are two versions of IsoMetrics, both based on the same items: IsoMetricss (short) supports summative evaluation of software systems, whereas IsoMetricsl (long) is best suited for formative evaluation purposes. The current version of IsoMetrics comprises 75 items operationalising the seven design principles of ISO 9241 Part 10. The IsoMetrics design provides information that can be used within an iterative software development. In summary, these are

* Scores of the usability dimension to measure the progress of development.

* Concrete information about malfunctions and their user-perceived attributes.

* Mean weight of any user-perceived attribute, given a class of system malfunctions." (Günther Gediga)

Find information about this tool, the short and long version ind German and English here.

EVADIS 

See the power of a multi-dimensional analysis-method that includes task-analysis, user-profiles, organizational human-human-interaction, ... an why such a method is very good, but not practical in use.

Resources

Links

• Usability Net
• ISONORM and ERGUSTO questionnaire

Articles and Books

• Heinsen, S., Vogt, P. (Hrsg.) (2003). Usability praktisch umsetzen. Hanser Verlag.
  • Kap. 8 (S. 98 ff.): Task-Analyse
  • Kap. 9 (S. 116 ff.): Expertenevaluation
  • Kap. 12 (S. 172 ff.): Fragebogen zur Evaluation
• Questionaire for User Interaction Satisfaction
• Kieras, D. (1997). A Guide to GOMS Model Usability Evaluation using NGOMLS. In: M. Helander, T.K. Landauer, P. Prabhu (eds.). Handbook of Human-Computer Interaction. Elsevier Science. B.V., pp. 733 766.

Back to Main Page