User study on 360 videos
The third home exam is concerned with the evaluation of user studies using categorical data, more specifically the analysis of categorical information that has been collected using the Pair Comparison method. To overcome the problem of collecting data for a user study, everybody must collect some data and share it with the rest of the course participants.
We are using a 360 video that we downloaded from YouTube, showing a drive through Sao Paolo. We have started with a 2K video, extracted 1 minute and created several resolutions. We have created versions of the 360 video that are rotated by a 90, 180 and 270 degrees starting angle.
Your preparation task for the assignment is that each of you finds at least 5 participants who are willing to watch all combinations of the videos in a pair comparison study, and rate which video is the better in each pair. There are 6 different videos, leading to a total 15 combinations. This means that each participant must watch 15 pairs of videos. You must show these to you viewers in random order to ensure that exhaustion affects of pair similarly.
Thanks to Magnus, we have a web server where the study can be conducted. This server can be accessed at http://panoplay.duckdns.org.
The study should be conducted in a browser on a computer of any kind. You can also ask your participants to conduct the study again from a mobile phone.
This preparation task should be completed by Tuesday, November 15 (midnight). The results will be available from the initial page of the test - from where they can be downloaded as a CSV file. This file will enable everybody to conduct a statistical analysis of the data.
We use Mattermost to discuss problems and solution.
It is desirable that we collect a relevant amount of information that can expose statistical differences between our assessors: ideal are users who assessed the quality on a mobile phone versus a computer screen, male/female users, children/grown-ups, and many other options.
From the structure of the logfile, it is obvious that it is impossible to conduct an ANOVA study of the data directly based on the binary ratings collected from the pair comparisons. Every line of the logfile samples a user's decision between two video qualities. This data has to be transformed to ensure that every alternative (column) of the ANOVA evaluation refers to a sample (or group of sample) that you want to compare with all the other samples (or groups of samples). It is important to transform the data in such a way that the statistical meaning of the pair comparison is retained.
The task for the first round is to create a presentation (about 10-20 slides) that explains your user study. The presentation should explain the setup and the dataset that has been collected including some basic statistics about the data that has been collected. You should explain the question(s) that you intend to answer with your analysis of the data, the transformation of the logged data that you have performed (including an explanation of what that means), and the results of your ANOVA study/studies. Finally, you should interpret the results.
The presentation is supposed to be given to the class in November 22. It is expected that you listen to and comment on the presentations given by others.
There is an opportunity to improve the presentation slides after the presentation. The deadline for the improved version is December 6, using Devilry.
FAQ for the home exam
Links to the videos
- high quality, backward facing: video
- medium quality, backward facing: video
- low quality, backward facing: video
- high quality, sideways facing: video
- medium quality, sideways facing: video
- low quality, sideways facing: video