# Presentations

## Updates

- 10 minutes each instead of 15.
- Friday presentations can go over a bit.
- Remember: everyone must speak
- Rehearsal: OH Thursday, Sunday afternoon 3 - 4:30


## Morning Schedule

```{r, echo=FALSE, eval=FALSE}
set.seed(87)
sample(letters[1:9], replace = FALSE)
# group h must go monday
sample(letters[10:18], replace = FALSE)
# j  must go first on mon and M can't go first.
```


**Friday**

- Akchheta, Celine, Sara
- Alison, Emily, Melissa
- Shanda, Shelbie, Sylvia

**Monday**

- Cleo, Grace, Yujia
- Dana, Jung, Rose
- Caiyun, Milani, Prarthana
- Amanda, Buyan, Michelle
- Jacqui, Melissa, Whitney
- Allie, Emily, Sibonelo


## Afternoon Schedule

**Friday**

- Kelu, Siyao, Siying
- Arda, David, Thalia
- Claire, Mahima, Van

**Monday**

- Annakate, Emma, Sam
- Amna, Anne, Nikita
- Fei, Steph, Tu
- Danni, Kate, Thao
- Anne, Liz, Rebecca
- Bharati, Ma'ame, Meher


## Exam

## Exam parameters

- Anything in the notes (slides + boardwork) or HW is fair game.
- MC + short answer
- Emphasis on full holistic analysis from research question to conclusion in
context.
- One 3.5" by 5" notecard allowed.
- Practice problems on Friday.
- Recommended homework (not collected): Chapter 7, exercise 3.


# Model Selection II

## Searching the model space {.build}

<div class="columns-2">

<img src="http://static.guim.co.uk/sys-images/Guardian/Pix/pictures/2012/8/3/1343993747897/NASA-image-of-the-rover-C-008.jpg
" height="350px" width="450px" />

For a given data set with many potential predictors, there exists many possible
models.

How should we systematically evaluate those models?

</div>


## A diagram


## Model search strategies

1. All best subsets
2. Backwards elimination
3. Forward selection

- Note that each method may choose a different model!


## Example: Bridge Building

<img src="http://brotherpeacemaker.files.wordpress.com/2010/08/bridge-building.jpg" height="500px" width="700px" />


## The data

```{r}
bridge <- read.table("http://www.stat.tamu.edu/~sheather/book/docs/datasets/bridge.txt", header=TRUE)
head(bridge)
```


## All best subsets

```{r}
logDArea <- log(bridge$DArea)
logCCost <- log(bridge$CCost)
logDwgs <- log(bridge$Dwgs)
logLength <- log(bridge$Length)
logSpans <- log(bridge$Spans)
X <- cbind(logDArea,logCCost,logDwgs,logLength,logSpans)
library(leaps)
b <- regsubsets(as.matrix(X), log(bridge$Time))
summary(b)$outmat
```


## Build best models

```{r}
summary(b)$outmat
m1 <- lm(log(bridge$Time) ~ logDwgs)
m2 <- lm(log(bridge$Time) ~ logDwgs + logSpans)
m3 <- lm(log(bridge$Time) ~ logDwgs + logSpans + logCCost)
m4 <- lm(log(bridge$Time) ~ logDwgs + logSpans + logCCost + logDArea)
m5 <- lm(log(bridge$Time) ~ logDwgs + logSpans + logCCost + logDArea + logLength)
models <- list(m1, m2, m3, m4, m5)
```


## Compare best models

```{r, echo=FALSE}
# load plotting function
plot_ics <- function(model_list) {
  AICs <- unlist(lapply(model_list, extractAIC))[seq(2, length(model_list) * 2, 2)]
  BICs <- unlist(lapply(model_list, extractAIC, k = log(length(models[[1]]$res))))[seq(2, length(model_list) * 2, 2)]
  plot(c(1, length(AICs)), range(c(AICs, BICs)), ylab = "IC value",
       xlab = "# of predictors", type = "n")
  points(1:length(AICs), AICs, col = "green")
  lines(1:length(AICs), AICs, lty = 3, lwd = 2, col = "green")
  points(1:length(BICs), BICs, col = "red")
  lines(1:length(BICs), BICs, lty = 3, lwd = 2, col = "red")
  legend("topright", legend = c("AIC", "BIC"), col = c("green", "red"), pch = 16, bty = "n")
}
plot_ics(models)
```


## Comparing best models

```{r}
summary(m2)$coef
summary(m3)$coef
```


## Comparing best models {.build}

The two and the three predictor model both do well, but the two predictor model
is preferred because of the statistical significance of the predictors.

*Be aware that when looking at p-values of many many models, the burden of proof
should be much higher.  That is, we need very low p-values to be convinced of
statistical significance*.


## Backward elimination

```{r}
backAIC <- step(m5, direction = "backward", data = bridge)
backBIC <- step(m5, direction = "backward", data = bridge, k = log(nrow(bridge)))
```


## Forward selection

```{r}
mint <- lm(log(Time) ~ 1, data = bridge)
forwardAIC <- step(mint,
                   scope = list(lower = ~1, upper = ~log(DArea) + log(CCost) + 
                                  log(Dwgs) + log(Length) + log(Spans)), 
                   direction = "forward", data = bridge)
forwardBIC <- step(mint,
                   scope = list(lower = ~1, upper = ~log(DArea) + log(CCost) + 
                                  log(Dwgs) + log(Length) + log(Spans)),
                   direction = "forward", data = bridge,k = log(nrow(bridge)))
```


## Stepwise compared

**Backward Elimination**

Optimal model based on AIC included `log(CCost)`, `log(Dwgs)`, and `log(Spans)`.
Optimal model based on BIC included only `log(Dwgs)` and `log(Spans)`.

**Forward Selection**

Using AIC, the optimal model is the same as by backward AIC.  Using BIC, the
optimal model is the same as backward BIC.

*We have the same choice between the 2 and 3 predictor models.*


# Activity

## Activity 9

Build a full model (with no transformations) to the `Haldcement.txt` file from
the book's website to predict the Y using all of the X's.

1. Identify the optimal model(s) using all possible subsets and AIC/BIC.
2. Identify the optimal model(s) using backward elimination and AIC/BIC.
3. Identify the optimal model(s) using forward selection and AIC/BIC.
4. Do the methods agree on the optimal model?  If not, why not?
5. Recommend a final model and provide your reasoning.

