Understanding HCI

Evaluation Methods

What is Evaluation?

Evaluation should have a context

  1. What do you want to evaluate and why?
  2. When and how do you do evaluation ?
  3. Evaluation is often concerned with the usability of the application or system

Why Evaluate?

  1. To try to maximise harmony of PACT elements.
  2. Is the design good enough?
  3. To compare different designs
  4. To test conformance to standards or to agreed measures
  5. To involve people in design
  6. To design iteratively because 'designers, no matter how good they are cannot get it right the first few times'

Evaluation can be:

  1. Free-form or task-based
  2. User-based or expert-based
  3. Structured or unstructured

Factors Affecting Validity

  • Evaluation Method
    • Suitability of Method
    • Test Environment
    • Completeness of prototype
  • Users
    • Previous experience
    • Motivation and interest
    • Sample size
  • Evaluators
    • How experience?
    • Are they biased?

Ethical considerations in testing and evaluation

  1. Be informative
  2. Be reassuring
  3. Be considerate
  4. Respect privacy

The DECIDE framework can be used to structure your evaluation

  1. Determine the overall goals that the evaluation addresses
  2. Explore the specific questions to be answered
  3. Choose the evaluation paradigm and techniques to answer the questions
  4. Identify practical issues that must be addressed, such as selecting participants
  5. Decide how to deal with ethical issues
  6. Evaluate, interpret and present the data

Examples

Examples of Evaluating Systems

  1. paper prototyping
  2. Participatory Heuristic Evaluation
  3. Thinking Aloud Protocol
  4. Co-discovery Learning / Constructive Interaction
  5. Retrospective Testing
  6. Performance Measurement
  7. Question Asking Protocol
  8. Remote Testing
  9. Pluralistic Walkthrough
  10. Cognitive Walkthrough