## Transformations

![tranformations](http://i1.wp.com/littlebylisten.com/wp-content/uploads/2013/08/breaking-bad.jpg)

Say you fit a linear model to data and find the residual plots look awful. One
strategy to regain a valid model is to transform your data.


## Example 1: Cleaning crews

A building maintenance company is planning to submit a bid to clean corporate offices.
How much should they bid?  They'd like to be able to cover the job with a team of
4 crews and a team of a 16 crews, but they want to be sure. To make a good 
prediction, they collected data on how many crews were required over a sample of
53 days.

```{r echo=FALSE, fig.height=4}
cleaning <- read.delim("http://www.stat.tamu.edu/~sheather/book/docs/datasets/cleaning.txt",
                     header = T)
plot(Rooms ~ Crews, data = cleaning, pch = 16, col = "darkgreen")
```


## Linear model?

```{r, echo=FALSE}
m1 <- lm(Rooms ~ Crews, data = cleaning)
summary(m1)$coef
plot(Rooms ~ Crews, data = cleaning, pch = 16, col = "darkgreen")
abline(m1, lwd = 2)
```


## Linearity and normality {.build}

```{r, echo=FALSE}
par(mfrow = c(1, 2))
plot(m1, 1:2)
```

The mean function appears to be linear and the residuals are well-approximated
by the normal distribution.


## Constant variance and influence {.build}

```{r echo = FALSE}
par(mfrow = c(1, 2))
plot(m1, c(3, 5))
```

There are no influential points, however there is dramatic increasing variance.


## PIs on an invalid model

```{r echo=FALSE}
pi <- predict(m1, data.frame(Crews = c(4, 16)), interval = "prediction")
plot(Rooms ~ Crews, data = cleaning, pch = 16, col = "darkgreen")
abline(m1, lwd = 2)
lines(c(4, 4), c(pi[1, 2], pi[1, 3]), col = "tomato", lwd = 2)
lines(c(16, 16), c(pi[2, 2], pi[2, 3]), col = "tomato", lwd = 2)
```

Prediction intervals are particularly sensitive to model assumptions, so we have
good reason to distrust this one.


## Square root transform {.build}

The square root tranform is often useful to reducing the increasing variance that
is found in many types of count data.

\[ X_t = \sqrt{X} \]

Let's transform both $X$ and $Y$.

```{r}
cleaning2 <- transform(cleaning, sqrtCrews = sqrt(Crews))
cleaning2 <- transform(cleaning2, sqrtRooms = sqrt(Rooms))
cleaning2[1:2, ]
```


## Transformed linear model?

```{r echo=FALSE}
m2 <- lm(sqrtRooms ~ sqrtCrews, data = cleaning2)
summary(m2)$coef
plot(sqrtRooms ~ sqrtCrews, data = cleaning2, pch = 16, col = "darkgreen")
abline(m2, lwd = 2)
```


## Linearity and normality {.build}

```{r, echo=FALSE}
par(mfrow = c(1, 2))
plot(m2, 1:2)
```

The mean function appears to be linear and the residuals are well-approximated
by the normal distribution.


## Constant variance and influence {.build}

```{r echo = FALSE}
par(mfrow = c(1, 2))
plot(m2, c(3, 5))
```

There are no influential points and the variance has been stabilized.


## PIs from a valid model

```{r echo=FALSE}
pi2 <- predict(m2, data.frame(sqrtCrews = c(2, 4)), interval = "prediction")
plot(sqrtRooms ~ sqrtCrews, data = cleaning2, pch = 16, col = "darkgreen")
abline(m2, lwd = 2)
lines(c(2, 2), c(pi2[1, 2], pi2[1, 3]), col = "tomato", lwd = 2)
lines(c(4, 4), c(pi2[2, 2], pi2[2, 3]), col = "tomato", lwd = 2)
```


## Comparing PIs

```{r}
pi
pi2^2
```




## Log Transformations


## Example 2: Truck prices

Can we use the age of a truck to predict what it's price should be?  Consider a 
random sample of 43 pickup trucks.

```{r echo = FALSE}
pickups <- read.csv("http://andrewpbray.github.io/data/pickup.csv")
plot(price ~ year, data = pickups, col = "steelblue", pch = 16)
```


## Consider unusual observations

The very old truck will be a high leverage point and may not be of interest to 
model. Let's only consider trucks made in the last 20 years.

```{r echo = FALSE}
plot(price ~ year, data = subset(pickups, year >= 1994), col = "steelblue", pch = 16)
m1 <- lm(price ~ year, data = subset(pickups, year >= 1994))
```


## Linear nodel?

```{r, echo=FALSE}
summary(m1)$coef
plot(price ~ year, data = subset(pickups, year >= 1994), col = "steelblue", pch = 16)
abline(m1, lwd = 2)
```


## Linearity and normality {.build}

```{r, echo=FALSE}
par(mfrow = c(1, 2))
plot(m1, 1:2)
```

The normality assumption on the errors seems fine but there seems to be a quadratic
trend in the mean function.


## Constant variance and influence {.build}

```{r echo = FALSE}
par(mfrow = c(1, 2))
plot(m1, c(3, 5))
```

One observation (44) should be investigated for its influence.  There is
evidence of increasing variance in the residuals.


## {.build}

```{r, echo=1, fig.height=3}
pickups2 <- transform(pickups, log_price = log(price))
par(mfrow = c(1, 2))
hist(pickups$price, main = "")
hist(pickups2$log_price, main = "")
```

Variables that span multiple orders of magnitude often benefit from a natural
log transformation.

\[ Y_t = log_e(Y) \]


## Log-transformed linear model

```{r, echo=FALSE}
m2 <- lm(log_price ~ year, data = subset(pickups2, year >= 1994))
summary(m2)$coef
plot(log_price ~ year, data = subset(pickups2, year >= 1994), col = "steelblue", pch = 16)
abline(m2, lwd = 2)
```


## Linearity and normality {.build}

```{r, echo=FALSE}
par(mfrow = c(1, 2))
plot(m2, 1:2)
```

The residuals from this model appear less normal, though the quadratic trend in
the mean function is now less apparent.


## Constant variance and influence {.build}

```{r echo = FALSE}
par(mfrow = c(1, 2))
plot(m2, c(3, 5))
```

There are no points flagged as influential and our variance has been stabilized.


## Model interpretation

```{r echo=FALSE}
summary(m2)$coef
```

\[ \widehat{log(price)} = -258.99 + 0.13 * year \]

For each additional year the car is newer, we would expect the log price of the
car to increase on average by `r round(summary(m2)$coef[2, 1], 2)` dollars.

Which isn't very useful . . .


## Working with logs {.build}

Two useful identities:

- \[ log(a) - log(b) = log(\frac{a}{b}) \]
- \[ e^{log(x)} = x \]


## {.build}

The slope coefficient for the log-transformed model is
`r round(summary(m2)$coef[2, 1], 2)`, meaning the *log* price difference between
cars that are one year apart is predicted to be `r round(summary(m2)$coef[2, 1], 2)`
log dollars.

\[
\begin{eqnarray}
log(price at year x + 1) - log(price at year x) &=& 0.13 \\
log(\frac{price at year x + 1}{price at year x}) &=& 0.13 \\
e^{log(\frac{price at year x + 1}{price at year x})} &=& e^{0.13} \\
\frac{price at year x + 1}{price at year x} = 1.14 \\
\end{eqnarray}
\]

For each additional year the car is newer we would expect the price of the car to
increase on average by a factor of 1.14.


## Transformations summary {.build}

- If a linear model fit to the raw data leads to questionable residual plots,
consider transformations.
- Count data and prices often benefit from transformations.
- The natural log and the square root are the most common, but you can use any 
transformation you like.
- Transformations may change model interpretations.
- Non-constant variance is a serious problem but it can often be solved by transforming
the response.
- Transformations can also fix non-linearity, as can polynomials - coming soon!


