Between- and within-subject experimental designs in Pro Lab

When you design an eye-tracking experiment, you must decide how to distribute your participants across conditions.  Usually there are two options: either you assign participants to the different experimental conditions, so that each participant experiences only one condition; or you expose all participants to all conditions and measure how their behavior changes when the experiment’s circumstances have changed. In experimental design lingo, these options are called a between-subject design (independent group) and within-subject design (often called repeated measures).

Figure 1. The test participants are randomly divided into two groups. Each group is assigned to a different condition.  

Figure 2. The whole group of test participants is tested in both conditions, and thus measured twice during the experiment. 

Usually, you make your design choice by weighing potential spurious effects caused by repeated testing vs. losing statistical power. In between-subject designs, each participant generates a single score per dependent variable and this design relies on an effort to randomize your participant assignment to each condition. On the other hand, in a within-subject design, each participant generates multiple scores for each dependent variable and this design relies more on the effort of counterbalancing the exposure to each condition.

In some cases, your choice is inherently restricted to a specific participant design. For example, if you want to investigate whether the reading performance of children with ADHD differs from that of “neurotypical” children, you must use a between-subject design, as the independent variable (diagnosis of ADHD) is intrinsically related to each child. If your objective is to determine how reading performance improves with age in a group of children with ADHD, then the choice needs to be a within-subject design, as the nature of the research question specifically requires you to follow the reading performance of each child in the group.

In short, your design choice is strongly influenced by your research question, and therefore you should consider each design according to it. In general, between-subject designs are more robust to effects caused by the repeated testing but may be affected by individual variation, whereas within-subject designs are more powerful but might suffer from repeated measures confounds (such as the participant figuring out the outcome of the experiment).

Once you have settled on your choice of participant design you will continue to work on the rest of the experiment design by creating the tasks, stimuli and defining the trial structure of your experiment. Once this is completed, you are ready to start to create your experiment in Pro Lab. In this section, we will go through some examples of how to implement different participant designs and export the relevant experimental variables (Metrics export).

Implementing between- subject designs

When implementing a design in Pro Lab it is important to plan your data analysis, more specifically, how will you export the relevant information for you to further process your data and perform your statistical analysis. This is done by:

• clearly operationalizing your independent variable and dependent variable measures.
• identifying which of the Pro Lab experimental design components are associated with your independent variable and how to extract them in the Metrics Export. 

Scenario 1: Your independent variable is related to participant characteristics (a diagnosis); your trial design, stimuli and stimuli groups, is the same for all participants.

Take the example cited above, where your aim is to investigate whether the reading performance of children with ADHD differs from that of “typical” children. Your experiment will, most likely, be composed of single or multiple reading tasks, and your stimuli will be text excerpts or individual sentences. Both groups will be exposed to the same tasks and stimuli, and the same eye tracking performance measures will be extracted during the analysis. The group identity of the participant is related to the independent variable (diagnosis of ADHD), which is intrinsic to the participant.

In this scenario, you can implement the experiment in Pro Lab in the following way:

1. add the stimuli elements to a single timeline and build the timeline using stimuli groups and actions to manage their presentation order and timeline structure.
2. create a participant variable to allocate the different participants to the respective groups, by tagging them before or after the recording.

During the analysis you will be able to export the participant variable values in a single column, that will allow you to segment the data according to the participant diagnosis (independent variable), together with the eye tracking measures and perform the necessary statistical analysis in, for example, R or SPSS (see Figure 3).

Figure 3. Shows the implementation of the experimental variables (independent and dependent variables) and the resultant Metrics export format in Tobii Pro Lab for our first experimental design scenario. The independent variable is implemented through a participant variable, and the dependent variable using the stimuli AOI and respective measures.

Scenario 2: Your independent variable is related to different tasks, stimulus or groups of stimuli (e.g. A/B test); your trial design, stimuli and stimuli groups, is different for different groups of participants.

In the second scenario, you would like to run a study to evaluate the impact of two alternative versions of an ad on a consumer viewing behavior. Particularly we want to know if that element distracts the viewers from reading a text element that contains the message. In this situation, the two ads differ only on the design of one element, so it is likely that you will want to avoid any viewing behavior carry over due to repeated exposure to the two ad versions. Consequently, you decide to expose each version to different participant groups. Your task design will most likely be a free viewing task with several unrelated filler ads plus the target ads. Your eye tracking measures will only be extracted from the viewing behavior on the target ads. Now let’s see how you can implement this design in Pro Lab, remember that you want to expose the two versions of the add to two groups of similar characteristics participants:

1. create a single timeline, add all the necessary elements to the timeline, except the target ads. And build the timeline using stimuli groups and actions to manage their presentation order and structure in the timeline.
2. create a second timeline by copying and pasting the first timeline.
3. add one version of the ad to the first timeline, and the other version to the second timeline.
4. name your timelines with the respective ad version name. This is done to help you to identify the timeline during the recording phase, and more importantly to identify your independent variable in the metrics export. The timeline name will be exported to your Metrics export table.  

During the analysis, you can use the timeline information (your independent variable), together with your AOI based eye tracking measures to perform the necessary statistical analysis in, for example, R or SPSS.

Figure 4. Shows the implementation of the experimental variables (independent and dependent variables) and the resultant Metrics export format in Tobii Pro Lab for our second experimental design scenario. The independent variable is implemented through the timeline name or id, and the dependent variable using the stimuli AOI and respective measures.

Within-subject designs

Like in the between-subject design scenarios we will discuss how you will export the information regarding your independent variable and the measures of your dependent variable.

Scenario 3: Your independent variable is related to the recording; your trial design, stimuli and groups, is the same for all participants and all sampling periods.

Let’s imagine that you want to measure the progress of a group of children with an ADHD diagnosis that are attending a 1-year long special education program. You have decided to evaluate their performance at the start, middle and end of the program (month 0, month 6 and month 12 respectively). Due to the nature of the research question, your participant design will be a within-subject design. Like in the previous example, your experiment will, most likely, be composed of single or multiple reading tasks, and your stimuli will be text excerpts. Let’s assume that due to the long time gap between the three tests, you do not expect any strong memory or practice effects on the performance. As a result, you will keep the test design constant across the three periods (including re-using the same texts) and extract the same eye tracking measures. In this scenario, your independent variable is the stage in their education operationalized in time units – month 0, month 6 and month 12. Let’s implement this design in Pro Lab:

1. add the stimuli elements to a single timeline.
2. if using more than one stimulus in each trial, or task, use stimuli groups and actions to manage their presentation order and structure in the timeline.
3. create a participant variable to allocate the different recordings to the 3 sampling periods (e.g. Month0, Month6, and Month12).
4. when you start a recording, you will need to create a new participant and name him/her in a way that it reflects both the participant id and the stage in the education when the recording occurs (e.g. Participant1_0, Participant2_0… Participant1_6, Participant2_6… Participant1_12, Participant2_12…)*. You need to do this every recording.
5. use the participant variable to tag each recording (identified by the “participant” name) according to when they were tested (Participant1_0 is tagged with Month0, Participant1_6 is tagged with Month6, Participant1_12 is tagged with Month12, Participant2_0 is tagged with Month0, Participant2_6 is tagged with Month6, Participant2_12 is tagged with Month12, etc…)

During the analysis you will be able to export the participant variables, that will allow you to segment the data according to the 3 stages in the education program (independent variable), together with the eye tracking measures and perform the necessary statistical analysis in, for example, R or SPSS.

 *Note: this step is important to be able to assign the recording to the participant and your independent variable - the test period.  

Figure 5. Shows the implementation of the experimental variables (independent and dependent variables) and the resultant Metrics export format in Tobii Pro Lab for our third experimental design scenario. The independent variable is implemented by setting and individual name to each recording and tagging the recording with two participant variables. The dependent variable using the stimuli AOI and respective measures.

Scenario 4: Your independent variable is related to the recording; your trial design, stimuli, and groups, is the same, for all participants, but different versions of the stimuli are used in the different sampling periods.

Now imagine that like in the previous scenario you are worried about practice and repeated exposure effects, and you decide to show a different text each time you re-sample a participant. Your participant design is still a within-subject design as each participant will contribute to the three independent variable scores, however, now you will have to counterbalance the presentation of the different texts, by dividing the participants into subgroups, where one group is exposed to the different stimulus in the following order "text A" at Month 0, "text B" at Month6 and "text C" at Month12. Another group "text B" at Month 0, "text A" at Month6 and "text C" at Month12". text and so on. Thus to be able to represent all these factors in your data export when you move your experiment design to Pro Lab you will need to use a different approach than the previous example, instead, you will:

1. create three different timelines and name the timelines according to the different sampling periods (Timeline text A, Timeline text B, Timeline text C).
2. add each text stimulus version to the corresponding timeline, and if necessary, use stimuli groups and actions to manage their presentation order and structure in the timeline.
3. create a participant variable to allocate the different recordings to the 3 sampling periods (e.g. Month0, Month6, and Month12).
4. when you start a recording, you will need to create a new participant and name him/her in a way that it reflects both the participant id and the stage in the education when the recording occurs (e.g. Participant1_0, Participant2_0… Participant1_6, Participant2_6… Participant1_12, Participant2_12…). You need to do this every recording.
5. use the participant variable to tag each recording (identified by the “participant” name) according to when they were tested (Participant1_0 is tagged with Month0, Participant1_6 is tagged with Month6, Participant1_12 is tagged with Month12, Participant2_0 is tagged with Month0, Participant2_6 is tagged with Month6, Participant2_12 is tagged with Month12, etc…)

During the analysis we can then export the time of interest related to the different texts, allowing you to segment the data by the three timelines, i.e. by text (control), and participant variable, i.e. 3 stages in the education program (independent variable), together with our eye tracking measures and perform the necessary statistical analysis in, for example, R or SPSS.