Interactions 2

Princeton University

Jason Geller, PH.D.(he/him)

2023-12-04

Final Paper

  • Due final exams week

  • Will post link to Github repository by Wednesday

    • Similar to our labs

    • Push everything to the repository

  • I want a reproducible APA formatted paper

    • This means:

      • I can reproduce your entire pdf/doc from the qmd/rmd in your repository
  • Questions should be posted via issues from the repository

Follow Along

Packages

library(pacman)
p_load("tidyverse", "easystats", "broom", "kableExtra", "emmeans", "Superpower")

Moderation/Interaction

  • The effect of X on Y depends on some third variable (moderator)

    • This tells us about the conditions that facilitate, enhance, or inhibit the effect

      • Large vs. small

      • Present vs. absent

      • Positive vs. negative

Categorical x Categorical Interaction

2 x 2 Between Factorial Dataset

  • LaPaglia, Miller, and Protexter (2022)

    • Looked at the impact of instructor fluency and gender on test performance (Quiz)

    • N = 72 (49 females, 23 males)

    • 2 (Fluency: fluent, disfluent) x 2 (Gender: male, female) design


    

Categorical x Categorical Interaction

  • Factorial design

    • Commonly used to refer to experiments where more than one factor is manipulated

    • 2-way (most common), 3-way, 4-way (don’t), 5-way (please don’t)

2-way (Factorial) ANOVA

  • In the example above we have two factors:

    • Factor A (e.g., Gender) with 2 levels (e.g., male vs. female)
    • Factor B (e.g., Fluency) with 2 levels (e.g., Disfluency vs. Fluency)
  • Fully crossed design

    • Every level of factor A is tested with every level of factor B
    • Total # groups (cells) is a x b
  • We will see how to formulate in terms of model comparisons:

    • Main effect of Gender
    • Main effect of Fluency
    • Interaction effect Gender x Fluency

Main effects vs. Simple effects

  • What is a main effect?

    • Marginal mean differences

      • They tell us what impact a particular factor has, ignoring the impact of the other factor

Main effects vs. Simple effects

  • Regardless of Gender condition, does Fluency of instructor affect performance?
Gender Disfluent Fluent Marginal
F 6.69 7.85 7.27
M 5.50 8.00 6.75
Marginal 6.10 7.92 7.01

Main effects vs. simple effects

  • Regardless of Fluency condition, does Gender of instructor affect performance?
Gender Disfluent Fluent Marginal
F 6.69 7.85 7.27
M 5.50 8.00 6.75
Marginal 6.10 7.92 7.01

Main effects vs. simple effects

  • Simple effects

    • Conditional mean effects

      • Differences between conditions/cells
Gender Disfluent Fluent Marginal
F 6.69 7.85 7.27
M 5.50 8.00 6.75
Marginal 6.10 7.92 7.01

Fitting the model: Interpretation

\[\hat{Y}= b_0 + b_1 X + b_2 Z + b_3 X*Z\]

where ( X ) is a dummy coded categorical predictor and ( Z ) is a dummy coded categorical predictor

  • Testing the simple (conditional effects), not main effects

    • \(b_0\): the intercept, or the predicted outcome when X and Z=0
    • \(b_1\): The mean difference in the intercept for a one unit change in \(X\) the predicted change in Y at Z = 0
    • \(b_2\): The mean difference in the intercept for a one unit change in \(Z\) the predicted change in Y at X = 0
    • \(b_3\): The interaction of \(X\) and \(Z\), The offset or mean difference in \(Z\) for a one-unit increase in \(X\) (or vice versa)

Fitting the model


    

    

Fitting the model


    

    

Fitting the model


    

    

Fitting the model


    

    

Fitting the model

Term Interpretation Value
intercept female disfluent instrutor 6.69
male-female female minus male, when disfluent 5.50-6.69=-1.19
dis fluent-fluent fluent minus disfluent, when female 7.85-6.69=1.16
male -female:flue nt-disfluent (male minus female when disfluent)-(male minus female when fluent)

(6.69−5.50)

−(7.85-8.00)=1.35

Linear modeling approach: Treatment/Dummy coding

  • Testing the simple (conditional effects), not main effects
Gender Fluency D1 D2 Interaction
Female=0 Disfluent=0 0 0 0
Female=0 Fluent=1 0 1 0
Male=1 Disfluent=0 1 0 0
Male=1 Fluent=1 1 1 1

\[\hat{Y}= b_0 + b_1({0}) + b_2({0}) + b_3({0})\] \[\bar Y_{Female,Disfluent} = b_0\]

Linear modeling approach: Treatment/dummy coding

Gender Fluency D1 D2 Interaction
Female=0 Disfluent=0 0 0 0
Female=0 Fluent=1 0 1 0
Male=1 Disfluent=0 1 0 0
Male=1 Fluent=1 1 1 1

\[\hat{Y}= b_0 + b_1({0}) + b_2({1}) + b_3({0})\]

\[\bar Y_{Female,Fluent}= b_0 + b_2\]

Linear modeling approach: Treatment/Dummy coding

Gender Fluecny D1 D2 Interaction
Female=0 Disfluent=0 0 0 0
Female=0 Fluent=1 0 1 0
Male=1 Disfluent=0 1 0 0
Male=1 Fluent=1 1 1 1

\[\hat{Y}= b_0 + b_1({1}) + b_2({0}) + b_3({0})\] \[\bar Y_{Male,Disfluent} = b_0 + b_1 \]

Linear Modeling Approach: Treatment/Dummy coding

Gender Fluency D1 D2 Interaction
Female=0 Disfluent=0 0 0 0
Female=0 Fluent=1 0 1 0
Male=1 Disfluent=0 1 0 0
Male=1 Fluent=1 1 1 1

\[\hat{Y}= b_0 + b_1({1}) + b_2({1}) + b_3({1})\]

\[\bar Y_{Male,Fluent} = b_0+b_1 + b_2 + b_3 \]

Marginal Means: using Emmeans


    

Linear modeling approach: deviation-coding

  • Gives main effects and interaction interpretation!

\[\hat{Y}= b_0 + b_1 X + b_2 Z + b_3 X*Z\]

  • \(b_0\): is the overall mean of all cells

  • \(b_1\): is the main effect of factor A

  • \(b_2\): is the main effect of factor B

  • \(b_3\): is the interaction effect

Linear modeling approach: deviation-coding


    

Linear Modeling Approach: Deviation-Coding

Term Interpretation Value
intercept grand mean 7.01
male-female female minus male 7.27-6.75=0.52
di sfluent-fluent fluent minus disfluent 7.92-6.10=-1.83
female-male:fl uent-disfluent (female minus male when disfluent)-(female - male when fluent)

(6.69−5.50)

−(7.85-8.00)=-1.35

Visualizing Categorical x Categorical Interactions

Visualizing Categorical x Categorical Interactions

Factorial ANOVA

ANOVA approach

  • \(df\), \(MS\), and \(F\) ratios are defined in the same way as they were for one-way ANOVA. We just have more of them

\[ \begin{aligned} MS_A &= \frac{SS_A}{df_A} \\ MS_B &= \frac{SS_B}{df_B} \\ MS_{AxB} &= \frac{SS_{AxB}}{df_{AxB}} \\ MS_{within} &= \frac{SS_{within}}{df_{within}} \\ \end{aligned} \]

\[\begin{aligned} F_A &= \frac{MS_A}{MS_{within}} \\ \\ F_B &= \frac{MS_B}{MS_{within}} \\ \\ F_{AxB} &= \frac{MS_{AxB}}{MS_{within}} \\ \end{aligned}\]

Null Hypotheses

  • The two main effects and the interaction represent three independent questions we can ask about the data. We have three null hypotheses to test

  • One null hypothesis refers to the marginal row means (Gender)

Disfluent Fluent Marginal
F \(\mu_{11}\) \(\mu_{12}\) \(\mu_{1.}\)
M \(\mu_{21}\) \(\mu_{22}\) \(\mu_{2.}\)
Marginal \(\mu_{.1}\) \(\mu_{.2}\) \(\mu_{..}\)

Null Hypotheses

Disfluent Fluent Marginal
F \(\mu_{11}\) \(\mu_{12}\) \(\mu_{1.}\)
M \(\mu_{21}\) \(\mu_{22}\) \(\mu_{2.}\)
Marginal \(\mu_{.1}\) \(\mu_{.2}\) \(\mu_{..}\)

\[ \begin{aligned} H_0&: \mu_{1.} = \mu_{2.} = \dots = \mu_{R.}\\ H_1&: \text{Not true that }\mu_{1.} = \mu_{2.} = \dots = \mu_{R.} \end{aligned} \]

Null Hypotheses

  • Main effect of Fluency
Disfluent Fluent Marginal
F \(\mu_{11}\) \(\mu_{12}\) \(\mu_{1.}\)
M \(\mu_{21}\) \(\mu_{22}\) \(\mu_{2.}\)
Marginal \(\mu_{.1}\) \(\mu_{.2}\) \(\mu_{..}\)

\[ \begin{aligned} H_0&: \mu_{.1} = \mu_{.2} = \dots = \mu_{.}\\ H_1&: \text{Not true that }\mu_{1.} = \mu_{2.} = \dots = \mu_{.} \end{aligned} \]

Null Hypotheses

The interaction null hypothesis can then be stated as follows:

Disfluent Fluent Marginal
F \(\mu_{11}\) \(\mu_{12}\) \(\mu_{1.}\)
M \(\mu_{21}\) \(\mu_{22}\) \(\mu_{2.}\)
Marginal \(\mu_{.1}\) \(\mu_{.2}\) \(\mu_{..}\)

\[ \begin{aligned} (\alpha\beta)_{ab}&= \mu_{ab} - \alpha_a - \beta_b - \mu_{..} \\ H_0&: (\alpha\beta)_{11} = (\alpha\beta)_{12} = \dots = (\alpha\beta)_{AB} = 0\\ H_1&: \text{At least one }(\alpha\beta)_{ab} \neq 0 \end{aligned} \]

F-statistics

\[ \begin{aligned} SS_{\text{total}} &= \sum_{a=1}^A\sum_{b=1}^B\sum_{i=1}^{N_{ab}}(Y_{abi}-\bar{Y}_{...})^2 \\ SS_{\text{Within}} &= \sum_{a=1}^A\sum_{b=1}^B\sum_{i=1}^{N_{ab}}(Y_{abi}-\bar{Y}_{ab.})^2 \\ SS_A &= BN\sum_{a=1}^A(\bar{Y}_{a..}-\bar{Y}_{...})^2\\ SS_B &= AN\sum_{b=1}^B(\bar{Y}_{.b.}-\bar{Y}_{...})^2\\ SS_{AB} &= N\sum_{a=1}^A\sum_{b=1}^B(\bar{Y}_{ab.}-\bar{Y}_{a..}-\bar{Y}_{.b.}+\bar{Y}_{...})^2 \\ \end{aligned}\]

ANOVA

  • Use afex package to run (my favorite)
library(afex)
aov_ez(id="Subject.Number", between=c("Gender", "Fluency"), dv="Quiz", data=gen,  anova_table = list(es = "pes")) %>% 
  nice()%>% 
  kable()
Effect df MSE F pes p.value
Gender 1, 47 3.05 1.12 .023 .294
Fluency 1, 47 3.05 13.92 *** .229 <.001
Gender:Fluency 1, 47 3.05 1.89 .039 .176
term estimate std.error statistic p.value conf.low conf.high
(Intercept) 7.0096154 0.2447905 28.635161 0.0000000 6.5171604 7.5020704
Gender1 0.5192308 0.4895810 1.060561 0.2943090 -0.4656792 1.5041407
Fluency1 -1.8269231 0.4895810 -3.731605 0.0005124 -2.8118330 -0.8420131
Gender1:Fluency1 1.3461538 0.9791620 1.374802 0.1757121 -0.6236660 3.3159737

Simple effects analysis

  • An examination of one factor at the levels of another

    • Our example:

      • Difference in quiz performance for videos with Males and Females at Disfluent and Fluent

        • Get F test for these comparisons (2 one-way ANOVAs)

\[F=\frac{MS_{gender_{oneway}}}{MS_{W_{omnibus}}}\]

Note

Using error from entire model is preferable

Simple effects analysis


    
  • No Gender effect for Fluency, F(1, 47) = 0.05, p = .82

  • No Gender effect for Disfluency, F(1, 47) = 2.91, p = .09

As t-tests

lm(Quiz~Gender*Fluency, data=gen) %>%
  emmeans::emmeans(~Gender|Fluency) %>%
  pairs()%>%
  kable()
contrast Fluency estimate SE df t.ratio p.value
F - M Disfluent 1.1923077 0.6994014 47 1.7047544 0.0948430
F - M Fluent -0.1538462 0.6852707 47 -0.2245042 0.8233382

Effect Sizes: Main Effects and Interaction

-Report \(\eta_p^2\) or \(\omega_p^2\) for Main Effects and interactions

#eta_squared
lm(Quiz~Gender*Fluency, data=gen) %>%
  effectsize::eta_squared(partial=TRUE)%>%
  kable()
Parameter Eta2_partial CI CI_low CI_high
Gender 0.0191853 0.95 0.0000000 1
Fluency 0.2259805 0.95 0.0731435 1
Gender:Fluency 0.0386598 0.95 0.0000000 1

Effect Sizes for simple effects

#effectsize from easystats
#get eta
F_to_eta2(f = c(2.91, 0.05), 
          df=c(1,1), 
             df_error=47)%>% # for full model
  kable() 
Eta2_partial CI CI_low CI_high
0.0583049 0.95 0 1
0.0010627 0.95 0 1
# get d
t_to_d(t = c(c(0.169, 0.772)), 
             df_error=47) %>% # for full model
  kable()
d CI CI_low CI_high
0.0493024 0.95 -0.5228257 0.6209075
0.2252155 0.95 -0.3495522 0.7976085

Types of sums of squares

  • Type 1 - Sequential

  • Type 2 - Hierarchical

  • Type 3 - Simultaneous

    • No difference if design is balanced, but different if unbalanced
    Disfluent Fluent
    F 13 13
    M 12 13

Type I, Type II, and Type III SS

  • By default R aov calculates Type I (SS)

    • Sequential (order listed in model)

      • First assign a maximum of variation to variable A

      • In the remaining variation, assign the maximum of variation to variable B

      • In the remaining variation, assign the maximum of variation to the interaction effect

      • Assign the rest to the residual SS

  • Can change depending on order terms are placed in the model!

Main Effect of Gender \(SS_A\)

Does Gender contribute significantly over and above an intercept-only model?

Restricted/Null model: `Quiz ~ 1`
Full/Alternative model: `Quiz ~ Gender`

F-Statistic for Gender Main Effect: \(SS_A\)

term df.residual rss df sumsq statistic p.value
Quiz ~ 1 50 193.9216 NA NA NA NA
Quiz ~ Gender 49 191.1154 1 2.806184 0.7194764 0.4004403

Main Effect for Fluency: \(SS_B\)

Does Fluency contribute meaningfully to the model over and above Gender?

Restricted/Null model: `Quiz ~ Gender`
Full/Alternative model: `Quiz ~ Gender + Fluency`

F-Statistic for Fluency Main Effect: \(SS_B\)

term df.residual rss df sumsq statistic p.value
Quiz ~ Gender 49 191.1154 NA NA NA NA
Quiz ~ Gender + Fluency 48 149.2308 1 41.88462 13.47216 0.0006072

Interaction Effect of Gender*Fluency: \(SS_{AB}\)

Does the interaction between Fluency and Gender contribute meaningfully to the model over and above the main effects?

Restricted/Null model: `Quiz ~ Gender + Fluency`
Full/Alternative model: `Quiz ~ Gender + Fluency + Gender:Fluency`

F-Statistic for Gender*Fluency Effect: \(SS_{AB}\)

term df.residual rss df sumsq statistic p.value
Quiz ~ Gender + Fluency 48 149.231 NA NA NA NA
Quiz ~ Gender + Fluency + Fluency:Gender 47 143.462 1 5.769 1.89 0.176

Full ANOVA Table

term df sumsq meansq statistic p.value
Gender 1 2.806184 2.806184 0.919345 0.3425536
Fluency 1 41.884615 41.884615 13.721984 0.0005569
Gender:Fluency 1 5.769231 5.769231 1.890080 0.1757121
Residuals 47 143.461538 3.052373 NA NA

Type I, Type II, and Type III ANOVAs

  • Type 2

    • Tests main effects

      • Ignores interactions

\[SS(A | B) A\]

\[SS(B | A) B\]

Note

Use Type II if you don’t care about interaction

Type II SS

  • Main effect Gender
Reduced/Null model: `Quiz ~ Fluency`
Full/Alternative model: `Quiz ~ Fluency + Gender`
  • Main effect Fluency
Reduced/Null model: `Quiz ~ Gender`
Full/Alternative model: `Quiz ~ Gender + Fluency`

Type II SS

  • Anova function in car package can handle these cases
library(car)

Anova(lm(Quiz~Gender*Fluency, data=gen), type="II") %>%
  tidy()%>%
  kable()
term sumsq df statistic p.value
Gender 3.255385 1 1.066509 0.3070202
Fluency 41.884615 1 13.721984 0.0005569
Gender:Fluency 5.769231 1 1.890080 0.1757121
Residuals 143.461538 47 NA NA

Type I, Type II, and Type III SS

  • Type III

    • Treats main effects and interactions simultaneously

    • Fit full model and remove effect of interest

      • How much of the variance is accounted for by X after taking into consideration all the other effects
  • Preferable if unequal cell sizes and care about interaction

Type III

  • Main effect of Gender
Reduced/Null model: `Quiz ~ Fluency + Gender:Fluency`
Full/Alternative model: `Quiz ~ Gender + Fluency + Gender:Fluency`

Type III

  • Main effect of Fluency
Reduced/Null model: `Quiz ~ Gender + Gender:Fluency`
Full/Alternative model: `Quiz ~ Gender + Fluency + Gender:Fluency`

Type III

  • Interaction
Reduced/Null model: `Quiz ~ Gender + Fluency`
Full/Alternative model: `Quiz ~ Gender + Fluency + Gender:Fluency`

Type III

contrasts(gen$Gender) <- c(0.5, -0.5)# sum code gend
contrasts(gen$Fluency) <- c(0.5, -0.5)# sum code fluency
# fit linear model
lm(Quiz~Gender*Fluency, data=gen)

Call:
lm(formula = Quiz ~ Gender * Fluency, data = gen)

Coefficients:
     (Intercept)           Gender1          Fluency1  Gender1:Fluency1  
          7.0096            0.5192           -1.8269            1.3462  
Anova(lm(Quiz~Gender*Fluency, data=gen), type="III") %>%
  tidy()%>%
  kable()
term sumsq df statistic p.value
(Intercept) 2502.861852 1 819.972435 0.0000000
Gender 3.433281 1 1.124791 0.2943090
Fluency 42.503925 1 13.924878 0.0005124
Gender:Fluency 5.769231 1 1.890080 0.1757121
Residuals 143.461538 47 NA NA

Type I, Type II, and Type III SS

Sums of Squares

Effect Type I Type II Type III
A SS(A) SS(A|B) SS(A|B,AB)
B SS(B|A) SS(B|A) SS(B|A,AB)
A:B SS(AB|A,B) SS(AB|A,B) SS(AB|A,B)

ANOVA power

  • Superpower
string <- "2b*2b"
n <- 15
# about 15 per conditon
mu <- c(5.50, 8.00, 6.69, 7.85)
# Enter means in the order that matches the labels below.
sd <-c(2.39,0.71,1.93,1.57)
# SDs
labelnames <- c("Gender", "male", "female", "Fluency", "Disfluency", "Fluency") #
# the label names should be in the order of the means specified above.

ANOVA power

design_result <- ANOVA_design(design = string,
                   n = n,
                   mu = mu,
                   sd = sd,
                   labelnames = labelnames)

Run simulation

#nsims = 100 is not enough
simulation_result <- ANOVA_power(design_result, 
                                 alpha_level = .05, 
                                 nsims = 100,
                                 verbose = FALSE)

simulation_result
Power and Effect sizes for ANOVA tests
                     power effect_size
anova_Gender            31     0.04839
anova_Fluency           96     0.22146
anova_Gender:Fluency    35     0.05879

Power and Effect sizes for pairwise comparisons (t-tests)
                                                                  power
p_Gender_male_Fluency_Disfluency_Gender_male_Fluency_Fluency         94
p_Gender_male_Fluency_Disfluency_Gender_female_Fluency_Disfluency    37
p_Gender_male_Fluency_Disfluency_Gender_female_Fluency_Fluency       79
p_Gender_male_Fluency_Fluency_Gender_female_Fluency_Disfluency       56
p_Gender_male_Fluency_Fluency_Gender_female_Fluency_Fluency           8
p_Gender_female_Fluency_Disfluency_Gender_female_Fluency_Fluency     41
                                                                  effect_size
p_Gender_male_Fluency_Disfluency_Gender_male_Fluency_Fluency           1.4758
p_Gender_male_Fluency_Disfluency_Gender_female_Fluency_Disfluency      0.5789
p_Gender_male_Fluency_Disfluency_Gender_female_Fluency_Fluency         1.1991
p_Gender_male_Fluency_Fluency_Gender_female_Fluency_Disfluency        -0.8872
p_Gender_male_Fluency_Fluency_Gender_female_Fluency_Fluency           -0.1205
p_Gender_female_Fluency_Disfluency_Gender_female_Fluency_Fluency       0.6320

Power: Drawing your interactions

Sommet, N., Weissman, D. L., Cheutin, N., & Elliot, A. J. (2023). How many participants do I need to test an interaction? Conducting an appropriate power analysis and achieving sufficient power to detect an interaction.Advances in Methods and Practices in Psychological Science, 6(2), 1-21.https://doi.org/10.1177/25152459231178728

Reporting results

I would report the three effects from this model as follows:

  • Main effect 1

  • Main effect 2

  • Interaction

    • Report M and SD for main effects
    • Significance tests (F, df, p, effect size)
      • If interaction is significant, simple effects analysis
        • Interpretation of simple effects
  • Figure visualizing either the main effects (if interaction is not significant) or interaction

Reporting results

d=aov(Quiz~Gender*Fluency, data=gen)
report(d)
The ANOVA (formula: Quiz ~ Gender * Fluency) suggests that:

  - The main effect of Gender is statistically not significant and small (F(1,
47) = 0.92, p = 0.343; Eta2 (partial) = 0.02, 95% CI [0.00, 1.00])
  - The main effect of Fluency is statistically significant and large (F(1, 47) =
13.72, p < .001; Eta2 (partial) = 0.23, 95% CI [0.07, 1.00])
  - The interaction between Gender and Fluency is statistically not significant
and small (F(1, 47) = 1.89, p = 0.176; Eta2 (partial) = 0.04, 95% CI [0.00,
1.00])

Effect sizes were labelled following Field's (2013) recommendations.

Reporting results

Tip

There was a significant main effect of fluency when examining the dependent variable of number correct on the final quiz, F(1, 47) = 13.92, p < 0.001, η2p = .229. Participants scored better on the quiz when the lecture was delivered fluently (M = 7.92, SD = 1.59) than when it was delivered disfluently (M = 6.10, SD = 2.17). No other main effects or interactions were significant for quiz scores.

Thoughts on interactions

  1. Avoid them if not theoretically motivated

  2. Testing interactions require lots of data

  3. Center continuous variables and deviation code categorical predictors

  4. For non-simple designs (> 2:2) use ANOVA afex::aov_ez (uses Type III SS)