POL90: Statistics

class: center, middle, inverse, title-slide

# POL90: Statistics
## Alternatives to <em>t</em>-Tests
### Prof Wasow <br/> Assistant Professor, Politics <br/> Pomona College
### 2022-02-16

---

# Announcements

.large[
* Assignments

+ PS04 due <mark>Friday</mark>
    
    + Report 1

]

.large[
* Statistical Sleuth
    + Important to read along

+ Skim Chapter 4
]

---
# Schedule

<table>
 <thead>
  <tr>
   <th style="text-align:right;"> Week </th>
   <th style="text-align:left;"> Date </th>
   <th style="text-align:left;"> Day </th>
   <th style="text-align:left;"> Title </th>
   <th style="text-align:right;"> Chapter </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:left;"> Feb 2 </td>
   <td style="text-align:left;"> Wed </td>
   <td style="text-align:left;"> Inference Using t-Distributions </td>
   <td style="text-align:right;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:left;"> Feb 7 </td>
   <td style="text-align:left;"> Mon </td>
   <td style="text-align:left;"> Inference Using t-Distributions </td>
   <td style="text-align:right;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:left;"> Feb 9 </td>
   <td style="text-align:left;"> Wed </td>
   <td style="text-align:left;"> Confidence Intervals </td>
   <td style="text-align:right;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 5 </td>
   <td style="text-align:left;"> Feb 14 </td>
   <td style="text-align:left;"> Mon </td>
   <td style="text-align:left;"> A Closer Look at Assumptions </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;color: black !important;background-color: yellow !important;"> 5 </td>
   <td style="text-align:left;color: black !important;background-color: yellow !important;"> Feb 16 </td>
   <td style="text-align:left;color: black !important;background-color: yellow !important;"> Wed </td>
   <td style="text-align:left;color: black !important;background-color: yellow !important;"> A Closer Look at Assumptions </td>
   <td style="text-align:right;color: black !important;background-color: yellow !important;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:left;"> Feb 21 </td>
   <td style="text-align:left;"> Mon </td>
   <td style="text-align:left;"> Alternatives to the t-Tools </td>
   <td style="text-align:right;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:left;"> Feb 23 </td>
   <td style="text-align:left;"> Wed </td>
   <td style="text-align:left;"> Alternatives to the t-Tools </td>
   <td style="text-align:right;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 7 </td>
   <td style="text-align:left;"> Feb 28 </td>
   <td style="text-align:left;"> Mon </td>
   <td style="text-align:left;"> Comparison Among Several Samples </td>
   <td style="text-align:right;"> 5 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 7 </td>
   <td style="text-align:left;"> Mar 2 </td>
   <td style="text-align:left;"> Wed </td>
   <td style="text-align:left;"> Comparison Among Several Samples </td>
   <td style="text-align:right;"> 5 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 8 </td>
   <td style="text-align:left;"> Mar 7 </td>
   <td style="text-align:left;"> Mon </td>
   <td style="text-align:left;"> Simple Linear Regression </td>
   <td style="text-align:right;"> 7 </td>
  </tr>
</tbody>
</table>

---
## Assignment schedule

<table>
 <thead>
  <tr>
   <th style="text-align:right;"> Week </th>
   <th style="text-align:left;"> Date </th>
   <th style="text-align:left;"> Day </th>
   <th style="text-align:left;"> Assignment </th>
   <th style="text-align:right;"> Percent </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:left;"> Feb 11 </td>
   <td style="text-align:left;"> Fri </td>
   <td style="text-align:left;"> PS03 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;color: black !important;background-color: yellow !important;"> 5 </td>
   <td style="text-align:left;color: black !important;background-color: yellow !important;"> Feb 18 </td>
   <td style="text-align:left;color: black !important;background-color: yellow !important;"> Fri </td>
   <td style="text-align:left;color: black !important;background-color: yellow !important;"> PS04 </td>
   <td style="text-align:right;color: black !important;background-color: yellow !important;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:left;"> Feb 25 </td>
   <td style="text-align:left;"> Fri </td>
   <td style="text-align:left;"> PS05 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;color: black !important;background-color: yellow !important;"> 7 </td>
   <td style="text-align:left;color: black !important;background-color: yellow !important;"> Mar 4 </td>
   <td style="text-align:left;color: black !important;background-color: yellow !important;"> Fri </td>
   <td style="text-align:left;color: black !important;background-color: yellow !important;"> Report1 </td>
   <td style="text-align:right;color: black !important;background-color: yellow !important;"> 6 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 8 </td>
   <td style="text-align:left;"> Mar 11 </td>
   <td style="text-align:left;"> Fri </td>
   <td style="text-align:left;"> PS06 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 9 </td>
   <td style="text-align:left;"> Mar 18 </td>
   <td style="text-align:left;"> Fri </td>
   <td style="text-align:left;"> Spring break </td>
   <td style="text-align:right;"> NA </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 10 </td>
   <td style="text-align:left;"> Mar 25 </td>
   <td style="text-align:left;"> Fri </td>
   <td style="text-align:left;"> PS07 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 11 </td>
   <td style="text-align:left;"> Apr 1 </td>
   <td style="text-align:left;"> Fri </td>
   <td style="text-align:left;"> PS08 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 12 </td>
   <td style="text-align:left;"> Apr 8 </td>
   <td style="text-align:left;"> Fri </td>
   <td style="text-align:left;"> Report2 </td>
   <td style="text-align:right;"> 8 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 13 </td>
   <td style="text-align:left;"> Apr 15 </td>
   <td style="text-align:left;"> Fri </td>
   <td style="text-align:left;"> PS09 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
</tbody>
</table>

---
class: center, middle, inverse

# Randomization vs Bootstrapping

---
## Statistical inferences permitted by study designs

<img src="images/randomization_selection_assignment.jpg" width="90%" style="display: block; margin: auto;" />
.pull-right[
.footnote[Source: *Statistical Sleuth*, Display 1.5]
]

---
class: top
## Random sampling from a population
.center[![](images/statistics1e_figun_03_p162.jpg)]

---
## Random assignment to two populations
<br/><br/>
.center[![](images/ss_display_1_6.png)]
.footnote[Source: *Statistical Sleuth*, Display 1.6]

---
## Sampling & Bootstrapping vs Randomization

---
## Random sampling vs randomization test

.vertical-center[
<table>
 <thead>
  <tr>
   <th style="text-align:left;"> Procedure </th>
   <th style="text-align:left;"> Randomization </th>
   <th style="text-align:left;"> Replacement </th>
   <th style="text-align:left;"> Example </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Sampling </td>
   <td style="text-align:left;"> Random sample </td>
   <td style="text-align:left;"> No </td>
   <td style="text-align:left;"> Starting with 30 observations, draw 20 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Bootstrapping </td>
   <td style="text-align:left;"> Random sample </td>
   <td style="text-align:left;"> Yes </td>
   <td style="text-align:left;"> Starting with 30 observations, draw 40 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Randomization test </td>
   <td style="text-align:left;"> Random assignment </td>
   <td style="text-align:left;"> No </td>
   <td style="text-align:left;"> Starting with two groups, shuffle group assignment </td>
  </tr>
</tbody>
</table>
]

---
class: middle, center, inverse

# Chapter 4: Alternatives to *t*-Tests

---
class: middle, center
background-color: #000000

---
## Case Study: Challenger Disaster
.center[![](images/oring.jpg)]

---
## Launch the Shuttle?
.center[![](images/challenger_table2.png)]

---
## Launch the Shuttle?
.center[![](images/challenger_table1.png)]

---

## Launch the Shuttle?

---
## Launch the Shuttle?
<img src="images/O_Ring_Commission_Chart.png" width="80%" style="display: block; margin: auto;" />

---

class: middle, center
background-color: #000000
.center[![](images/md_tuftee_portrait_640.jpg)]

---
## Better ways to examine O-ring data?

<table class="table table-striped" style="font-size: 9px; width: auto !important; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;"> flight_code </th>
   <th style="text-align:right;"> launch_temp </th>
   <th style="text-align:right;"> erosion </th>
   <th style="text-align:right;"> blow_by </th>
   <th style="text-align:right;"> damage_index </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;font-weight: bold;background-color: #E0FFFF !important;"> 51-C </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 53 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 3 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 2 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 11 </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;background-color: #E0FFFF !important;"> 41-B </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 57 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 1 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 0 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;background-color: #E0FFFF !important;"> 61-C </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 58 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 1 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 0 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;background-color: #E0FFFF !important;"> 41-C </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 63 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 1 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 0 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 1 </td>
   <td style="text-align:right;"> 66 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 6 </td>
   <td style="text-align:right;"> 67 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 51-A </td>
   <td style="text-align:right;"> 67 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 51-D </td>
   <td style="text-align:right;"> 67 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 5 </td>
   <td style="text-align:right;"> 68 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 3 </td>
   <td style="text-align:right;"> 69 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;background-color: #E0FFFF !important;"> 2 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 70 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 1 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 0 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 9 </td>
   <td style="text-align:right;"> 70 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;background-color: #E0FFFF !important;"> 41-D </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 70 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 1 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 0 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 51-G </td>
   <td style="text-align:right;"> 70 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 7 </td>
   <td style="text-align:right;"> 72 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 8 </td>
   <td style="text-align:right;"> 73 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 51-B </td>
   <td style="text-align:right;"> 75 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 61-A </td>
   <td style="text-align:right;"> 75 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 51-[ </td>
   <td style="text-align:right;"> 76 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 41-G </td>
   <td style="text-align:right;"> 78 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 51-J </td>
   <td style="text-align:right;"> 79 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 51-F </td>
   <td style="text-align:right;"> 81 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
</tbody>
</table>

---
## Better ways to examine O-ring data?
.large[
.vertical-center[
- Plot (all) the data

- Linear regression

- Permutation test

- Randomization test

- Wilcox Rank-sum test

]
]

---
## Plot (all) the data
.center[
<img src="images/plot1.png" width="85%" style="display: block; margin: auto;" />
]
---
## Some observations have the same values
<table class="table table-striped" style="font-size: 10px; width: auto !important; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;"> flight_code </th>
   <th style="text-align:right;"> launch_temp </th>
   <th style="text-align:right;"> erosion </th>
   <th style="text-align:right;"> blow_by </th>
   <th style="text-align:right;"> damage_index </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> 51-C </td>
   <td style="text-align:right;"> 53 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 11 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 41-B </td>
   <td style="text-align:right;"> 57 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 61-C </td>
   <td style="text-align:right;"> 58 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 41-C </td>
   <td style="text-align:right;"> 63 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 1 </td>
   <td style="text-align:right;"> 66 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;background-color: #E0FFFF !important;"> 6 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 67 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 0 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 0 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;background-color: #E0FFFF !important;"> 51-A </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 67 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 0 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 0 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;background-color: #E0FFFF !important;"> 51-D </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 67 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 0 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 0 </td>
   <td style="text-align:right;font-weight: bold;background-color: #E0FFFF !important;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 5 </td>
   <td style="text-align:right;"> 68 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 3 </td>
   <td style="text-align:right;"> 69 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;background-color: lightpink !important;"> 2 </td>
   <td style="text-align:right;font-weight: bold;background-color: lightpink !important;"> 70 </td>
   <td style="text-align:right;font-weight: bold;background-color: lightpink !important;"> 1 </td>
   <td style="text-align:right;font-weight: bold;background-color: lightpink !important;"> 0 </td>
   <td style="text-align:right;font-weight: bold;background-color: lightpink !important;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;background-color: lightblue !important;"> 9 </td>
   <td style="text-align:right;font-weight: bold;background-color: lightblue !important;"> 70 </td>
   <td style="text-align:right;font-weight: bold;background-color: lightblue !important;"> 0 </td>
   <td style="text-align:right;font-weight: bold;background-color: lightblue !important;"> 0 </td>
   <td style="text-align:right;font-weight: bold;background-color: lightblue !important;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;background-color: lightpink !important;"> 41-D </td>
   <td style="text-align:right;font-weight: bold;background-color: lightpink !important;"> 70 </td>
   <td style="text-align:right;font-weight: bold;background-color: lightpink !important;"> 1 </td>
   <td style="text-align:right;font-weight: bold;background-color: lightpink !important;"> 0 </td>
   <td style="text-align:right;font-weight: bold;background-color: lightpink !important;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;background-color: lightblue !important;"> 51-G </td>
   <td style="text-align:right;font-weight: bold;background-color: lightblue !important;"> 70 </td>
   <td style="text-align:right;font-weight: bold;background-color: lightblue !important;"> 0 </td>
   <td style="text-align:right;font-weight: bold;background-color: lightblue !important;"> 0 </td>
   <td style="text-align:right;font-weight: bold;background-color: lightblue !important;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 7 </td>
   <td style="text-align:right;"> 72 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 8 </td>
   <td style="text-align:right;"> 73 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 51-B </td>
   <td style="text-align:right;"> 75 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 61-A </td>
   <td style="text-align:right;"> 75 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 51-[ </td>
   <td style="text-align:right;"> 76 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 41-G </td>
   <td style="text-align:right;"> 78 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 51-J </td>
   <td style="text-align:right;"> 79 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 51-F </td>
   <td style="text-align:right;"> 81 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
</tbody>
</table>
---
## Plot (all) the data
.center[
<img src="images/plot2.png" width="85%" style="display: block; margin: auto;" />
]
---
## Run a regression to confirm

```r
lm(damage_index ~ launch_temp, data = shuttle) %>% 
  summary()
```

```

Call:
lm(formula = damage_index ~ launch_temp, data = shuttle)

Residuals:
    Min      1Q  Median      3Q     Max 
-2.2992 -1.5056 -0.5435  0.8145  5.5260

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 18.41724    4.62111   3.985 0.000728 ***
*launch_temp -0.24421    0.06638  -3.679 0.001488 ** 
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 2.153 on 20 degrees of freedom
Multiple R-squared:  0.4036,	Adjusted R-squared:  0.3738 
F-statistic: 13.54 on 1 and 20 DF,  p-value: 0.001488
```

---
## Tables to graphs
.large[
.vertical-center[
- Regressions are really useful for telling us whether relationships are statistically significant

- Data may violate assumptions

- <em>Substantive</em> significance takes a little more work

- Graphing our results is a nice way to evaluate the effects that we are estimating <em>and</em> to communicate them to broad audiences.
]
]
---
## Let's go back to our plot
.center[
<img src="images/plot2.png" width="85%" style="display: block; margin: auto;" />
]

---
## Add predicted values
.center[
<img src="images/plot3.png" width="85%" style="display: block; margin: auto;" />
]
---
## Don't forget uncertainty!
.center[
<img src="images/plot4.png" width="85%" style="display: block; margin: auto;" />
]
---
## Should we launch the rocket?
.center[
<img src="images/plot5.png" width="85%" style="display: block; margin: auto;" />
]
---
## Should we launch the rocket?

.center[
<img src="images/plot6.png" width="85%" style="display: block; margin: auto;" />
]

---
class: middle, center
background-color: #000000
<img src="images/shuttle_at_launch_ice_small.jpg" width="63%" style="display: block; margin: auto;" />

---
## Permutation test (as defined in textbook)

.large[

- Linear regression and `$t$`-tools might not be the best choice here given the data

- With small data, we can calculate *every* possible outcome

- For example, with five coins, what's probability of all heads?

]

```r
2^5
```

```
[1] 32
```

```r
1/32
```

```
[1] 0.03125
```

```r
(1/2)^5
```

```
[1] 0.03125
```

---
## Permutation test (as defined in textbook)

.large[

- Permutation tests does not rely on distributional assumptions, so let's see if we can use that to confirm the relationship between temperature and O-ring failure

- Not going to cover in depth, see *Statistical Sleuth* 4.3
]

---
## Summary of t-statistics from all 10,626 rearrangements
.center[
![](images/ss_display_4_10.png)
]
.footnote[Source: Statistical Sleuth, Display 4.10]

---
## How else could we run statistical tests?

.large[
.vertical-center[
- Randomization test

- Binomial approximation

- Rank or Rank-sum test

]
]
---
## Load Shuttle Data

```r
shuttle <- Sleuth3::case0401 %>% clean_names()
head(shuttle, 20)
```

```
   incidents launch
1          1   Cool
2          1   Cool
3          1   Cool
4          3   Cool
5          0   Warm
6          0   Warm
7          0   Warm
8          0   Warm
9          0   Warm
10         0   Warm
11         0   Warm
12         0   Warm
13         0   Warm
14         0   Warm
15         0   Warm
16         0   Warm
17         0   Warm
18         0   Warm
19         0   Warm
20         0   Warm
```

---
## Calculate Difference in Means with `infer`

```r
library(infer)

diff_observed <- shuttle %>% 
  
  # what relationship are we testing?
  specify(incidents ~ launch) %>%
  
  # what is our test statistic?
  calculate(stat = "diff in means",
  
            # in what order should we subtract?
            order = c("Cool", "Warm"))

diff_observed
```

```
Response: incidents (numeric)
Explanatory: launch (factor)
# A tibble: 1 × 1
   stat
  <dbl>
1   1.3
```

---
## Calculate Difference in Means with Base R

```r
cool <- shuttle[shuttle$launch == "Cool", ]

warm <- shuttle[shuttle$launch == "Warm", ]

mean(cool$incidents) - mean(warm$incidents)
```

```
[1] 1.3
```

---
## Randomization test: Create a Null Distribution

```r
shuttle_null_distribution <- shuttle %>%

# what relationship are we testing?
  specify(incidents ~ launch) %>%
  
  # what is our null hypothesis?
  hypothesize(null = "independence") %>%
  
  # how many randomizations/permutations?
  generate(reps = 10000, type = "permute") %>%
  
  # what test statistic? what order to subtract?
  infer::calculate(stat  = "diff in means", 
                   order = c("Cool", "Warm"))

head(shuttle_null_distribution, 2)
```

```
Response: incidents (numeric)
Explanatory: launch (factor)
Null Hypothesis: independence
# A tibble: 2 × 2
  replicate  stat
      <int> <dbl>
1         1  -0.5
2         2  -0.2
```

---
## Look at the simulated null distribution

```r
head(shuttle_null_distribution, 15)
```

```
Response: incidents (numeric)
Explanatory: launch (factor)
Null Hypothesis: independence
# A tibble: 15 × 2
   replicate  stat
       <int> <dbl>
 1         1  -0.5
 2         2  -0.2
 3         3  -0.2
 4         4   0.1
 5         5  -0.2
 6         6   0.1
 7         7  -0.2
 8         8  -0.2
 9         9   0.4
10        10  -0.2
11        11   1  
12        12   0.1
13        13   0.1
14        14  -0.2
15        15  -0.2
```

---
## Randomization test

.left-code[

```r
infer::visualize(
  shuttle_null_distribution) +
  shade_p_value(
    obs_stat  = diff_observed, 
    direction = "greater")
```

]
.right-plot[
<img src="week05_02_files/figure-html/shuttle_rand_vis_code1-1.png" width="100%" style="display: block; margin: auto;" />
]

---
## Randomization test: One-sided p-value

```r
head(shuttle_null_distribution)
```

```
Response: incidents (numeric)
Explanatory: launch (factor)
Null Hypothesis: independence
# A tibble: 6 × 2
  replicate  stat
      <int> <dbl>
1         1  -0.5
2         2  -0.2
3         3  -0.2
4         4   0.1
5         5  -0.2
6         6   0.1
```

```r
sum(shuttle_null_distribution$stat >= diff_observed$stat)
```

```
[1] 108
```

```r
sum(shuttle_null_distribution$stat >= diff_observed$stat) / 10000
```

```
[1] 0.0108
```

---
## Wilcox Rank-sum test

```r
wilcox.test(incidents ~ launch, data = shuttle)
```

```

Wilcoxon rank sum test with continuity correction

data:  incidents by launch
*W = 74, p-value = 0.001144
alternative hypothesis: true location shift is not equal to 0
```

---
class: middle, center, inverse

# Whatever the Test, 
# Communicate Clearly

---
## Communicate this in an understandable way

.vertical-center[
.large[
- Given our data, the probability of the observed relationship between launch temperature and O-ring failures, if in fact there is no relationship, is less than about 1 percent.
]
]

---
## What are features of good data communication?

.large[
* Clarity

- Always remember the key relationship that you are seeking to demonstrate
  
  - Clearly labeled figures with informative titles and legends
 
  - Be concise, avoid unnecessary jargon
]

---
## What are features of good data communication?

.large[
* More than "significance"

- Substance, magnitude, sign, original units (un-transformed)
  
  - Account for uncertainty in the analysis: prefer confidence intervals over standard errors
  
  - Tables can be informative, but (good) visualizations of key relationship can be worth 1,000 stars

- Report scope of inference: 
      - correlation or causal story? how generalizable?
]

---
## Things to Remember

.vertical-center[
.large[
- The influence of your arguments will be proportional to how well you communicate them

- Focus on clear and concise representations of the relationship of interest

- Tables are useful, but visual representations can also convey a lot of information

- Be sure to communicate uncertainty

- Focus on substantive effects
]
]

---
class: middle, center

# Questions