E4: Statistical analysis using non-parametric tests

Objective

• Run a Friedman test with Wilcoxon signed-rank post-hoc analysis
• Replicate stats published in the paper referenced below for NASA-TLX Subjective preferences for “STUDY 1: FORCEEDGE ON DESKTOP”.

Axel Antoine, Sylvain Malacria, and Géry Casiez. 2017. ForceEdge: Controlling Autoscroll on Both Desktop and Mobile Computers Using the Force. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ‘17). ACM, New York, NY, USA, 3281-3292. DOI: https://doi.org/10.1145/3025453.3025605

The goal is to replicate the results presented on page 3286, section NASA-TLX Subjective preferences.

NASA-TLX is a questionnaire designed to measure the perceived workload, by asking participants to rate their mental demand, physical demand, temporal demand, performance, effort and frustration on a 20 points scale that can be considered as an ordinal variable.

General steps to run Friedman test and post-hoc analysis

1. Get the data from the first experiment, corresponding to NASA-TLX answers. The CSV file provides columns with the participant number, task (Select, Move), technique (ForceEdge, Baseline), question (mental, physical, temporal, performance, effort, frustration) and the corresponding score between 1 and 20.
2. Load the data in R
3. Filter data to keep only one question (i.e. frustration).
4. Non-parametric tests can use only a single independent variable and we have two (task and technique), giving a total of 4 conditions. A way to deal with this problem is to create an artificial independent variable called “task_technique”. Use unite command from tidyr package to create a column that contains the concatenation of the text in the task and technique columns.
5. Next step is to visualize the data using a boxplot to get an idea of possible significant differences between the conditions, using the boxplot command. Boxplots are typically used to represent this type of data. Create a graph with task_technique on the X axis and score on the Y axis.
6. Following step is to arrange your data to run the Friedman test. You need to get 4 columns (1 for each task x technique) with 16 lines (1 for each participant). Use the spread function to get that result. Remove the extra columns (participant, variable and value) by setting them to NULL (i.e. t$participant = NULL).
7. Run now the Friedman test using friedman.test function (you will need to use the data.matrix function to first convert your data in the expected format).
8. When you get a significant effect, you can run post-hoc pairwise comparisons using pairwise.wilcox.test function. This time the data need to be arrange in the way corresponding to step 4. Use Bonferroni correction.

Running Friedman tests on each question

Run Friedman test for each question: mental, physical, temporal, performance, effort, frustration. You should have only one line to modify in your script. You should get only significant effect for performance and frustration. For these last two, run post-hoc analysis. Note there was an error in the report of the results in the paper: it is reported there is a significant difference between ForceEdge and Baseline for the two tasks, while in fact it is only for the Move task.

Submit

Follow the submission instructions on the course information page. Provide the Rmd file and its html output (using the Knit button on the interface) showing the results only for frustration question. Make good use of markdown syntax to present your results in a way that is easy to read and understand. In your solution notes (integrated in the Rmd file), describe any problems you ran into, and the main resource(s) you used (blog posts, online tutorials, stackoverflow posts, papers, textbooks, etc.). These resources should have brief descriptions of what the resource is and how it helped you.