POL90: Applied Quantitative Analysis

class: center, middle, inverse, title-slide

# POL90: Applied Quantitative Analysis
## Chapter 5: Comparisons Among Several Samples
### Prof Wasow</br>Assistant Professor, Politics
### 2022-03-12

---

# Announcements

.large[
* Assignments

+ Report 1

]

.large[
* Statistical Sleuth

+ Read Chapter 5
      - *Skip* 5.6.2, Kruskal-Wallis Nonparametric Analysis of Variance
    + Supplement
      - http://appliedstats.org/chapter5.html

]

---
# Schedule

<table>
 <thead>
  <tr>
   <th style="text-align:right;"> Week </th>
   <th style="text-align:left;"> Date </th>
   <th style="text-align:left;"> Day </th>
   <th style="text-align:left;"> Title </th>
   <th style="text-align:right;"> Chapter </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:right;"> 5 </td>
   <td style="text-align:left;"> Feb 14 </td>
   <td style="text-align:left;"> Mon </td>
   <td style="text-align:left;"> A Closer Look at Assumptions </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 5 </td>
   <td style="text-align:left;"> Feb 16 </td>
   <td style="text-align:left;"> Wed </td>
   <td style="text-align:left;"> A Closer Look at Assumptions </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:left;"> Feb 21 </td>
   <td style="text-align:left;"> Mon </td>
   <td style="text-align:left;"> Alternatives to the t-Tools </td>
   <td style="text-align:right;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:left;"> Feb 23 </td>
   <td style="text-align:left;"> Wed </td>
   <td style="text-align:left;"> Comparison Among Several Samples </td>
   <td style="text-align:right;"> 5 </td>
  </tr>
  <tr>
   <td style="text-align:right;color: black !important;background-color: yellow !important;"> 7 </td>
   <td style="text-align:left;color: black !important;background-color: yellow !important;"> Feb 28 </td>
   <td style="text-align:left;color: black !important;background-color: yellow !important;"> Mon </td>
   <td style="text-align:left;color: black !important;background-color: yellow !important;"> Comparison Among Several Samples </td>
   <td style="text-align:right;color: black !important;background-color: yellow !important;"> 5 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 7 </td>
   <td style="text-align:left;"> Mar 2 </td>
   <td style="text-align:left;"> Wed </td>
   <td style="text-align:left;"> Simple Linear Regression </td>
   <td style="text-align:right;"> 7 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 8 </td>
   <td style="text-align:left;"> Mar 7 </td>
   <td style="text-align:left;"> Mon </td>
   <td style="text-align:left;"> Simple Linear Regression </td>
   <td style="text-align:right;"> 7 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 8 </td>
   <td style="text-align:left;"> Mar 9 </td>
   <td style="text-align:left;"> Wed </td>
   <td style="text-align:left;"> Regression by Calculation </td>
   <td style="text-align:right;"> 7 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 9 </td>
   <td style="text-align:left;"> Mar 14 </td>
   <td style="text-align:left;"> Mon </td>
   <td style="text-align:left;"> Spring Recess </td>
   <td style="text-align:right;"> - </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 9 </td>
   <td style="text-align:left;"> Mar 16 </td>
   <td style="text-align:left;"> Wed </td>
   <td style="text-align:left;"> Spring Recess </td>
   <td style="text-align:right;"> - </td>
  </tr>
</tbody>
</table>

---
## Assignment schedule

<table>
 <thead>
  <tr>
   <th style="text-align:right;"> Week </th>
   <th style="text-align:left;"> Date </th>
   <th style="text-align:left;"> Day </th>
   <th style="text-align:left;"> Assignment </th>
   <th style="text-align:right;"> Percent </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:left;"> Feb 25 </td>
   <td style="text-align:left;"> Fri </td>
   <td style="text-align:left;"> PS05 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;color: black !important;background-color: yellow !important;"> 7 </td>
   <td style="text-align:left;color: black !important;background-color: yellow !important;"> Mar 4 </td>
   <td style="text-align:left;color: black !important;background-color: yellow !important;"> Fri </td>
   <td style="text-align:left;color: black !important;background-color: yellow !important;"> Report1 </td>
   <td style="text-align:right;color: black !important;background-color: yellow !important;"> 6 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 8 </td>
   <td style="text-align:left;"> Mar 11 </td>
   <td style="text-align:left;"> Fri </td>
   <td style="text-align:left;"> PS06 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 9 </td>
   <td style="text-align:left;"> Mar 18 </td>
   <td style="text-align:left;"> Fri </td>
   <td style="text-align:left;"> Spring break </td>
   <td style="text-align:right;"> NA </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 10 </td>
   <td style="text-align:left;"> Mar 25 </td>
   <td style="text-align:left;"> Fri </td>
   <td style="text-align:left;"> PS07 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 11 </td>
   <td style="text-align:left;"> Apr 1 </td>
   <td style="text-align:left;"> Fri </td>
   <td style="text-align:left;"> PS08 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 12 </td>
   <td style="text-align:left;"> Apr 8 </td>
   <td style="text-align:left;"> Fri </td>
   <td style="text-align:left;"> Report2 </td>
   <td style="text-align:right;"> 8 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 13 </td>
   <td style="text-align:left;"> Apr 15 </td>
   <td style="text-align:left;"> Fri </td>
   <td style="text-align:left;"> PS09 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 14 </td>
   <td style="text-align:left;"> Apr 22 </td>
   <td style="text-align:left;"> Fri </td>
   <td style="text-align:left;"> PS10 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 15 </td>
   <td style="text-align:left;"> Apr 29 </td>
   <td style="text-align:left;"> Fri </td>
   <td style="text-align:left;"> Report3 </td>
   <td style="text-align:right;"> 10 </td>
  </tr>
</tbody>
</table>

---
class: center, middle

# Chapter 5: Multiple comparisons 
# with ANOVA: Spock Trial

---
## Case study: Spock trial
.large[
- In 1968 Dr. Benjamin Spock was tried in United States District Court of Massachusetts in Boston on charges of conspiring to violate the Selective Service Act by encouraging young men to resist being drafted into military service for Vietnam.

- The defense in that case challenged the method by which jurors were selected, claiming that women-many of whom had raised children according to popular methods developed by Dr. Spock-were underrepresented. In fact, the Spock jury had no women.
]

---
## Case study: Spock trial

.large[
- Boston area juries are selected in three stages. 
  - From the City Directory, the Clerk of the Court selects at random 300 names for potential jury duty. 
  - Before a trial, a venire of 30 or more jurors is selected from the 300 names, again-according to the law-at random. 
  - An actual jury is selected from the venire in a nonrandom process allowing each side to exclude certain jurors for a variety of reasons.

- The Spock defense pointed to the venire for their trial, which contained only one woman. That woman was released by the prosecution, making an all-male jury.
]

---
## Spock trial: look at data

.left-code[

```r
library(ggplot2)
library(forcats)

spock <- Sleuth3::case0502
spock$Judge <- fct_relevel(spock$Judge, 
            c("Spock's", "A", "B", "C", "D", "E", "F") )
ggplot(data = spock) +
  aes(x = Judge, 
      y = Percent) +
  geom_boxplot() + 
  ylab("Percent Women")
```
]

.right-plot[
<img src="week07_01_files/figure-html/spock_boxplot_plot1-1.png" width="100%" style="display: block; margin: auto;" />
]

---
## Hypotheses

- The research question we can address with this data is whether judge assignment has an effect on percent women in a venire. If it had an effect, we should observe significant difference in percents across judge venires.

* To try and answer that question, an ANOVA approach is well suited, as it allows us to compare multiple samples without collapsing the data to a pairwise comparison.

- Our null hypothesis is that the mean percent across all judges is the same:

`$$\mu_{Spock} = \mu_{A} = \mu_{B} = \mu_{C} = \mu_{D} = \mu_{E} = \mu_{F}$$`

- Our alternative hypothesis is that the mean weight across all judges is not equal (think Sesame Street: at least one of these things is not like the other)

---
class: center, middle

# Calculating Residual 
# Sum of Squares in R

---

## Calculating RSS for equal means model in `R`

```r
spock <- spock %>% 
  mutate(equal_mean           = mean(Percent),
         eq_mean_residual     = Percent - equal_mean,
         eq_mean_residual_sq  = eq_mean_residual^2)

head(spock,3)
```

```
  Percent   Judge equal_mean eq_mean_residual eq_mean_residual_sq
1     6.4 Spock's       26.6            -20.2                 407
2     8.7 Spock's       26.6            -17.9                 320
3    13.3 Spock's       26.6            -13.3                 176
```

```r
rss_eq_mean <- sum(spock$eq_mean_residual_sq)
rss_eq_mean
```

```
[1] 3792
```

---
## Visualize Residuals with Equal Means Model

---

## Calculating RSS for separate means model in `R`

```r
spock <- spock %>% 
  group_by(Judge) %>%
  mutate(sep_mean              = mean(Percent),
         sep_mean_residual     = Percent - sep_mean,
         sep_mean_residual_sq  = sep_mean_residual^2)

spock %>% select(Judge, Percent, sep_mean, sep_mean_residual, sep_mean_residual_sq) %>% head(3)
```

```
# A tibble: 3 × 5
# Groups:   Judge [1]
  Judge   Percent sep_mean sep_mean_residual sep_mean_residual_sq
  <fct>     <dbl>    <dbl>             <dbl>                <dbl>
1 Spock's     6.4     14.6             -8.22                67.6 
2 Spock's     8.7     14.6             -5.92                35.1 
3 Spock's    13.3     14.6             -1.32                 1.75
```

```r
rss_sep_mean <- sum(spock$sep_mean_residual_sq)
rss_sep_mean
```

```
[1] 1864
```

---
## Visualize Residuals with Separate Means Model

---
class: middle, center

# Question: Is there a better model?

---

## RSS as Distance from the Data

.footnote[*Statistical Sleuth* 3ed, Display 5.11]

---
## Calculating RSS for two means model in `R`

```r
spock <- spock %>%
  mutate(spock_binary = case_when(
                          Judge == "Spock's" ~ "Spock's",
                          Judge != "Spock's" ~ "Not Spock's")
  ) %>% 
  group_by(spock_binary) %>%
    mutate(two_mean        = mean(Percent),
           two_mean_res    = Percent - two_mean,
           two_mean_res_sq = two_mean_res ^ 2) %>%
  ungroup()

spock %>% select(Judge, Percent, two_mean, two_mean_res, two_mean_res_sq) %>% head(3)
```

```
# A tibble: 3 × 5
  Judge   Percent two_mean two_mean_res two_mean_res_sq
  <fct>     <dbl>    <dbl>        <dbl>           <dbl>
1 Spock's     6.4     14.6        -8.22           67.6 
2 Spock's     8.7     14.6        -5.92           35.1 
3 Spock's    13.3     14.6        -1.32            1.75
```

```r
sum(spock$two_mean_res_sq)
```

```
[1] 2191
```

---
## Visualizing Residuals with Two Means Model

---
class: middle, center
# Linking *F*-statistic and ANOVA table

---

## Linking *F*-statistic and ANOVA table
<br><br><br>
<img src="images/f_stat_equation_anova_table.png" width="1284" style="display: block; margin: auto;" />

---
## Linking *F*-statistic and ANOVA table
<br><br><br>
<img src="images/f_stat_equation_anova_table_A.png" width="1284" style="display: block; margin: auto;" />

---
## Linking *F*-statistic and ANOVA table
<br><br><br>
<img src="images/f_stat_equation_anova_table_B.png" width="1284" style="display: block; margin: auto;" />

---
## Linking *F*-statistic and ANOVA table
<br><br><br>
<img src="images/f_stat_equation_anova_table_C.png" width="1284" style="display: block; margin: auto;" />

---
## Linking *F*-statistic and ANOVA table
<br><br><br>
<img src="images/f_stat_equation_anova_table1.png" width="1284" style="display: block; margin: auto;" />

---
## Linking *F*-statistic and ANOVA table
<br><br><br>
<img src="images/f_stat_equation_anova_table2.png" width="1284" style="display: block; margin: auto;" />

---
## Linking *F*-statistic and ANOVA table
<br><br><br>
<img src="images/f_stat_equation_anova_table3.png" width="1284" style="display: block; margin: auto;" />

---
## Linking *F*-statistic and ANOVA table
<br><br><br>
<img src="images/f_stat_equation_anova_table4.png" width="1284" style="display: block; margin: auto;" />

---
## Linking *F*-statistic and ANOVA table
<br><br><br>
<img src="images/f_stat_equation_anova_table6.png" width="1284" style="display: block; margin: auto;" />

---
## Linking *F*-statistic and ANOVA table
<br><br><br>
<img src="images/f_stat_equation_anova_table8.png" width="1284" style="display: block; margin: auto;" />

---
## Linking *F*-statistic and ANOVA table
<br><br><br>
<img src="images/f_stat_equation_anova_table9.png" width="1284" style="display: block; margin: auto;" />

---
## Linking *F*-statistic and ANOVA table
<br><br><br>
<img src="images/f_stat_equation_anova_table10.png" width="1284" style="display: block; margin: auto;" />

---
## Three ratios: Numerator ratio
<br><br><br>
<img src="images/f_stat_equation_anova_table11.png" width="1284" style="display: block; margin: auto;" />

---
## Three ratios: Numerator ratio
<br><br><br>
<img src="images/f_stat_equation_anova_table13.png" width="1284" style="display: block; margin: auto;" />

---
## Three ratios: Denominator ratio
<br><br><br>
<img src="images/f_stat_equation_anova_table12a.png" width="1284" style="display: block; margin: auto;" />

---
## Three ratios: Denominator ratio
<br><br><br>
<img src="images/f_stat_equation_anova_table14.png" width="1284" style="display: block; margin: auto;" />

---
## Three ratios: Overall ratio
<br><br><br>
<img src="images/f_stat_equation_anova_table15.png" width="1284" style="display: block; margin: auto;" />

---
## Three ratios: Overall ratio
<br><br><br>
<img src="images/f_stat_equation_anova_table16.png" width="1284" style="display: block; margin: auto;" />

---

## Calculating *p*-value

We can also use that *F* distribution to calculate the *p*-value of our *F*-statistic:

```r
# NOTE: setting lower.tail = FALSE gives us right tail
pf(6.72, 6, 39, lower.tail = FALSE) 
```

```
[1] 6.08e-05
```

```r
# repeat with high "penalty" for scientific notation
options(scipen = 10)
pf(6.72, 6, 39, lower.tail = FALSE) 
```

```
[1] 0.0000608
```

The *p*-value es extremely low, therefore we can reject the null hypothesis that all the means are equal.

---
## Visualizing *F* (6, 39) & *F*-stat = 6.72

```r
visualize::visualize.f(stat = 6.72, df1 = 6, df2 = 39, section = "upper")
```

---

## Robustness check

To check the robustness of our manual calculation, we can use the built-in aov() function:

```r
aov(Percent ~ Judge, data = spock) %>% 
summary()
```

```
            Df Sum Sq Mean Sq F value   Pr(>F)    
Judge        6   1927     321    6.72 0.000061 ***
Residuals   39   1864      48                     
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
```

---
class: middle, center
# Calculating the *F*-statistic

---

## ANOVA Table: Equal means vs two means

* Start with RSS and df for reduced model

.vertical-center[
| Source of Variation          | Sum of Squares | d.f. | Mean Square | *F*-statistic | *p*-value |
|------------------------------|----------------|------|-------------|-------------|---------|
| `$RSS_{reduced} - RSS_{full}$` | ( )            | ( )  | ( )         | ( )         | ( )     |
| (Two Means)                  | ( )            | ( )  | ( )         |             |         |
| (Equal Means)                | 3,791.53       | 45   |             |             |         |
]

---

## ANOVA Table: Equal means vs two means

* Incorporate RSS and df for full model

.vertical-center[
| Source of Variation          | Sum of Squares | d.f. | Mean Square | *F*-statistic | *p*-value |
|------------------------------|----------------|------|-------------|-------------|---------|
| `$RSS_{reduced} - RSS_{full}$` | ( )            | ( )  | ( )         | ( )         | ( )     |
| (Two Means)                  | (2190.90)      | (44) | ()     |             |         |
| (Equal Means)                | 3,791.53       | 45   |             |             |         |
]

---

## ANOVA Table: Equal means vs two means

* Divide `$RSS_{full}$` by `$df_{full}$` to calculate variance of `$RSS_{full}$` or `$s_p^2$`

.vertical-center[
| Source of Variation          | Sum of Squares | d.f. | Mean Square | *F*-statistic | *p*-value |
|------------------------------|----------------|------|-------------|-------------|---------|
| `$RSS_{reduced} - RSS_{full}$` | ()      | ()  | ()   | ( )         | ( )     |
| (Two Means)                  | (2190.90)      | (44) | (49.79)     |             |         |
| (Equal Means)                | 3,791.53       | 45   |             |             |         |
]
---

## ANOVA Table: Equal means vs two means

* Calculate `$RSS_{reduced} - RSS_{full}$` and `$df_{reduced}$` - `$df_{full}$`

.vertical-center[
| Source of Variation          | Sum of Squares | d.f. | Mean Square | *F*-statistic | *p*-value |
|------------------------------|----------------|------|-------------|-------------|---------|
| `$RSS_{reduced} - RSS_{full}$` | (1600.63)      | (1)  | ()   | ( )         | ( )     |
| (Two Means)                  | (2190.90)      | (44) | (49.79)     |             |         |
| (Equal Means)                | 3,791.53       | 45   |             |             |         |
]

---

## ANOVA Table: Equal means vs two means

* Calculate Mean Square by dividing ESS by ( `$df_{reduced} - df_{full}$` )

.vertical-center[
| Source of Variation          | Sum of Squares | d.f. | Mean Square | *F*-statistic | *p*-value |
|------------------------------|----------------|------|-------------|-------------|---------|
| `$RSS_{reduced} - RSS_{full}$` | (1600.63)      | (1)  | (1600.63)   | ( )         | ( )     |
| (Two Means)                  | (2190.90)      | (44) | (49.79)     |             |         |
| (Equal Means)                | 3,791.53       | 45   |             |             |         |
]

---

## ANOVA Table: Equal means vs two means

* Calculate *F*-statistic by dividing Mean Square terms

.vertical-center[
| Source of Variation          | Sum of Squares | d.f. | Mean Square | *F*-statistic | *p*-value |
|------------------------------|----------------|------|-------------|-------------|---------|
| `$RSS_{reduced} - RSS_{full}$` | (1600.63)      | (1)  | (1600.63)   | (32.15)         | ( )     |
| (Two Means)                  | (2190.90)      | (44) | (49.79)     |             |         |
| (Equal Means)                | 3,791.53       | 45   |             |             |         |
]
---

## ANOVA Table: Equal means vs two means

* Calculate `$p$`-value. In `R`: `pf`( *F*-statistic, `$df_{reduced}$` , `$df_{full}$`, lower.tail = FALSE)

.vertical-center[
| Source of Variation          | Sum of Squares | d.f. | Mean Square | *F*-statistic | *p*-value    |
|------------------------------|----------------|------|-------------|-------------|------------|
| `$RSS_{reduced} - RSS_{full}$` | (1600.63)      | (1)  | (1600.63)   | (32.15)     | (0.000001) |
| (Two Means)                  | (2190.90)      | (44) | (49.79)     |             |            |
| (Equal Means)                | 3,791.53       | 45   |             |             |            |
]

---
## Visualizing *F* (1, 44) & *F*-stat = 32.15

```r
visualize::visualize.f(stat = 32.15, df1 = 1, df2 = 44, section = "upper")
```

---
## Sidenote: Hypothetical *F*(3, 44) & *F*-stat = 32.15

```r
visualize::visualize.f(stat = 32.15, df1 = 3, df2 = 44, section = "upper")
```

---
## Sidenote: Hypothetical *F*(5, 44) & *F*-stat = 32.15

```r
visualize::visualize.f(stat = 32.15, df1 = 5, df2 = 44, section = "upper")
```

---
class: middle, center

# Two means vs seven means

---