Another important part of residual analysis is the identification of unusual observations in a data set. Because regression estimates are weighted averages with weights that vary by observation, some observations are more important than others. This weighting is more important than many users of regression analysis realize. In fact, the example below demonstrates that a single observation can have a dramatic effect in a large data set.
There are two directions in which a data point can be unusual, the horizontal and vertical directions. By “unusual,” we mean that an observation under consideration seems to be far from the majority of the data set. An observation that is unusual in the vertical direction is called an outlier. An observation that is unusual in the horizontal directional is called a high leverage point. An observation may be both an outlier and a high leverage point.