Evaluation

Don’t fall in love with your prototype

Evaluation Theory Script

    Hi, this is Simon and welcome to evaluation theory!


    Now, finally we get to the evaluation you already heard so much about. When we have a prototype or a product, especially when it comes with a new design, we need to evaluate it. In most cases we think of the product as a kind of solution for one or several specific problems. In the evaluation phase we can find out if the product can help the users solving their problem, answering the question if the product is useful. Moreover, we want to know if the users like using the product or, in other words, if the product shows a good usability.


    We have already talked about it in the sections before, but here you can see again when we should evaluate our prototype, which is after the first implementation of an idea. The evaluation marks the end of a development cycle, but not necessarily the end of the development. We can use the insights from the evaluation to create new requirements and ideas or to enhance the design of the prototype. After those modifications, we should evaluate again to see if the changes have the desired effect. In general, we want to evaluate all the different constructs we have talked about, for example usability and utility. We also want to test if the prototype meets the quantitative and qualitative design criteria. One thing we should keep in mind is that not all prototypes are suitable for every evaluation method, so the method depends on the kind and the maturity stage of the prototype.


    DECIDE is a framework which helps guide an evaluation process. As you can see here, the initial letters of decide are also the initial letters of several rules applying to an evaluation process. During this process we have to ask ourselves which goals we have by evaluating. So, do we rather want to know if the design is appealing or if we understood the user's needs correctly? Exploring questions helps us operationalize these goals. The next question would be, which evaluation paradigm we choose to answer the questions, meaning we have to decide on which tools we want to use Practical issues could be resources we need or users which act as test persons. All these practical issues can have an impact on the conduct of the evaluation. Ethical issues on the other hand could be private data protection, which should be already considered while planning an evaluation. When we have evaluated and finally generated data, we have to analyze it and interpret it. The first two letters of DECIDE, the determination of the goals and exploring the questions, is dependent on the requirements of the project. Here you can see some example questions helping operationalizing the goal of the evaluation.


    Choosing the evaluation method depends on the stage of a project or the stage of the prototype. Naturally, the maturity of the prototype plays a role when deciding how to test it, since it is hard to conduct, for example, a user study containing interactions with the system if we do not have a concrete interaction design yet. On this slide, you can see the software release cycle map. It looks like a linear process but as you can see when looking at alpha and beta here we also have different prototype stages before we get into the release period.


    When we decide which questions we want to answer by means of an evaluation, we should know the difference between quantitative and qualitative data. As the name already says, quantitative data give information on quantities, which means they are measurable. Quantitative data consist of numbers, for example measures of distances in meters or measures of time in seconds.


    Qualitative data are not measurable as they contain information about qualities. We can gather qualitative data by interviewing test persons, for example. To give you an idea: when the number of pages of a book is quantitative, a content summary would be qualitative.


    Formative evaluations should be conducted during the development phase, doing this we can verify against specifications. The summative evaluation on the contrary is a validation against the requirements. By the way, this is also how we divide the forms of evaluation you do in this course. Formative: the tasks you upload in github and summative: which is the final graded presentation. Here you can see the differentiation between formative and summative evaluation. The first one is conducted when the prototype is still in an early stage and we are still in the developing process, which means the goal of the evaluation is to gain insight into how to improve the prototype. A summative evaluation is conducted in a late project stage with an at least nearly finalized prototype. In this evaluation, we validate if the system meets the criteria defined in the requirement phase.


    Thanks for the attention and see you next time.

The goal of a designer is to listen, observe, understand, sympathize, empathize, synthesize, and glean insights that enable him or her to make the invisible visible.

Overview Evaluation tasks

We will take a closer look at evaluation for several tasks. Feel free to already discover more about the methods: