Lab6: Effect Size and Power

Princeton University

In today’s lab, we will be using data from [@tekin2021; Experiment 1]. In their study, participants viewed cue-target pairs (e.g., DOOR - HOUSE) during a study phase. After, groups either provided delayed JOLs (e.g., given a cue word, how likely is it on a scale of 0-100 you will recall the target on a later test), attempted to retrieve the target word (DOOR-?), or restudied the same cue-target pairs (DOOR-HOUSE). Each group then took a final test over the pairs. The aim of their study was to determine whether engaging in retrieval practice and providng delayed JOLs had similar effects on memory.

To access the data for Experiment 1, please visit their OSF page–the link is in the paper.

Effect size

As a first step, you will need to read in the data from OSF. A link can be found in their paper.

We are interested in the scores on the final test (Total Final) as a function of Condition (Condition). We will only be looking at three conditions: Restudy, Overt retrieval (retrieval practice), and Cue-Only JOL. Please select and filter the appropriate columns from the dataset.

Visualize the differences between the three groups

Next I want you to run a lm model. Use dummy coding and fit two models to get the pairwise comparisons between all the three conditions (you will need to use the relevel function and re-run the lm model to get the third comparison).

Take one of your models from above and use emmeans and pairs function to get the pairwise comparisons. This a much easier approach and one that I much prefer you use in your work.

Calculate Cohen’s d by hand for each of the pairwise comparisons

Use the MOTE package to get 95% CIs around each d value

Write-up the results from the pairwise comparisons in APA style. The differences between each of the pairwise comparisons between each group and all relevant information (t, p, 95% CIs, size) . Make sure you correct for pairwise comparisons and state which correction you use

Power

WebPower

• Using the WebPower package calculate the the number of participants per group we need to have 90% power in our model to detect a difference.

Superpower

• Book going over Superpower https://aaroncaldwell.us/SuperpowerBook/

• Reviewer 2 asked you to calculate the power of @tekin2021 Experiment 1 after you ran it. Set up a study design using the ANOVA_design function from Superpower. Use the same means, SD, and n (use 40 per group as SuperPowercannot do unequal sample sizes for one-way designs) from their Experiment 1 study (excluding the one condition). Run a power analysis on this data.

What is our power to detect the overall effect of Condition? What about the pairwise comparisons between the groups?

• To detect the effect of Condition we have only 64.22% power (Yikes). To detect the cue-JOL vs. overt difference we have 9.36% power. To detect the Cue-JOL vs. Restudy difference we have 43.98% power. Finally, to detect the overt and restudy difference we have power of 73%.

What kind of power analysis would this be?

• Post-hoc

What do you think of this study design

• Not very well powered study.

• We now want to run a replication study. Let’s plan a study a study where we want to collect 100 Ps per group.

  Change ANOVA_design to reflect just this difference. Set N =

What would are power be using the current study parameters to detect overall effect? How about each pairwise comparison?

• Let’s say I want to power the study to be able to detect a .05 point difference on the final test between the Cue-Only JOLs and Overt Retrieval groups.

Change ANOVA_design to reflect just this difference. Set N = 40. What is our power to detect a .05 point difference with 40 per group?

Plot the power curve just for this difference. What sample size is needed to achieve 90% power?

• You need 205 participants to have 90% power to detect a .05 difference between the two conditions.