# Essential Statistics for Data Science: A Case Study using Python, Part I

System requirements: Language: Python 3.5
Libraries: statsmodels, pandas, matplotlib
Data: available here

Our last post dove straight into linear regression. In this post, we'll take a step back to cover essential statistics that every data scientist should know. To demonstrate these essentials, we'll look at a hypothetical case study involving an administrator tasked with improving school performance in Tennessee.

Note, this tutorial is intended to serve solely as an educational tool and not as a scientific explanation of the causes of various school outcomes in Tennessee.

## Introduction

Meet Sally, a public school administrator. Some schools in her state of Tennessee are performing below average academically. Her superintendent, under pressure from frustrated parents and voters, approached Sally with the task of understanding why these schools are under-performing. Not an easy problem, to be sure.

To improve school performance, Sally needs to learn more about these schools and their students, just as a business needs to understand its own strengths and weaknesses and its customers.

Though Sally is eager to build an impressive explanatory model, she knows the importance of conducting preliminary research to prevent possible pitfalls or blind spots. Thus, she engages in a thorough exploratory analysis, which includes: a lit review, data collection, descriptive and inferential statistics, and data visualization.

## Lit Review

Sally has strong opinions as to why some schools are under-performing, but opinions won't do, nor will a handful of facts; she needs rigorous statistical evidence.

Sally conducts a lit review, which involves reading a variety of credible sources to familiarize herself with the topic. Most importantly, Sally keeps an open mind and embraces a scientific world view to help her resist confirmation bias (seeking solely to confirm one's own world view).

In Sally's lit review, she finds multiple compelling explanations of school performance: curriculae, income, and parental involvement. These sources will help Sally select her model and data, and will guide her interpretation of the results.

## Data Collection

The data we want isn't always available (see here and here), but Sally lucks out and finds student performance data based on test scores (`school_rating`) for every public school in middle Tennessee. The data also includes various demographic, school faculty, and income variables (see readme for more information). Satisfied with this dataset, she writes a web-scraper to retrieve the data.

But data alone can't help Sally; she needs to convert the data into useful information.

## Descriptive and Inferential Statistics

Sally opens her stats textbook and finds that there are two major types of statistics, descriptive and inferential.

Descriptive statistics identify patterns in the data, but they don't allow for making hypotheses about the data.

Within descriptive statistics, there are two measures used to describe the data: central tendency and deviation. Central tendency refers to the central position of the data (mean, median, mode) while the deviation describes how far spread out the data are from the mean. Deviation is most commonly measured with the standard deviation (see here for more on standard deviation). A small standard deviation indicates the data are close to the mean, while a large standard deviation indicates that the data are more spread out from the mean.

Inferential statistics allow us to make hypotheses (or inferences) about a sample that can be applied to the population. For Sally, this involves developing a hypothesis about her sample of middle Tennessee schools and applying it to her population of all schools in Tennessee.

For now, Sally puts aside inferential statistics and digs into descriptive statistics.

To begin learning about the sample, Sally uses pandas' `describe` method, as seen below. The column headers in bold text represent the variables Sally will be exploring. Each row header represents a descriptive statistic about the corresponding column.

``````import pandas as pd
# be sure to use the correct file path where you downloaded the data
# describe the data
df.describe()
``````
Out[10]:
school_ratingsizereduced_lunchstate_percentile_16state_percentile_15stu_teach_ratioavg_score_15avg_score_16full_time_teacherspercent_blackpercent_whitepercent_asianpercent_hispanic
count347.000000347.000000347.000000347.000000341.000000347.000000341.000000347.000000347.000000347.000000347.000000347.000000347.000000
mean2.968300699.47262250.27953958.80172958.24926715.46167157.00469257.04985644.93948121.19798361.6734872.64265111.164553
std1.690377400.59863625.48023632.54074732.7026305.72517026.69645027.96897422.05338623.56253827.2748593.10962912.030608
min0.00000053.0000002.0000000.2000000.6000004.7000001.5000000.1000002.0000000.0000001.1000000.0000000.000000
25%2.000000420.50000030.00000030.95000027.10000013.70000037.60000037.00000030.0000003.60000040.6000000.7500003.800000
50%3.000000595.00000051.00000066.40000065.80000015.00000061.80000060.70000040.00000013.50000068.7000001.6000006.400000
75%4.000000851.00000071.50000088.00000088.60000016.70000079.60000080.25000054.00000028.35000085.9500003.10000013.800000
max5.0000002314.00000098.00000099.80000099.800000111.00000099.00000098.900000140.00000097.40000099.70000021.10000065.200000

Looking at the output above, Sally's variables can be put into two classes: measurements and indicators.

Measurements are variables that can be quantified. All data in the output above are measurements. Some of these measurements, such as `state_percentile_16`, `avg_score_16` and `school_rating`, are outcomes; these outcomes cannot be used to explain one another. For example, explaining `school_rating` as a result of `state_percentile_16` (test scores) is circular logic. Therefore we need a second class of variables.

The second class, indicators, are used to explain our outcomes. Sally chooses indicators that describe the student body (for example, `reduced_lunch`) or school administration (`stu_teach_ratio`) hoping they will explain `school_rating`.

Sally sees a pattern in one of the indicators, `reduced_lunch`. `reduced_lunch` is a variable measuring the average percentage of students per school enrolled in a federal program that provides lunches for students from lower-income households. In short, `reduced_lunch` is a good proxy for household income, which Sally remembers from her lit review was correlated with school performance.

Sally isolates `reduced_lunch` and groups the data by `school_rating` using pandas' `groupby` method and then uses `describe` on the re-shaped data (see below).

``````df[['reduced_lunch', 'school_rating']]\
.groupby(['school_rating'])\
.describe()\
.unstack() # transpose the data to make more readable
``````
Out[11]:
reduced_lunch
countmeanstdmin25%50%75%max
school_rating
0.043.083.5813958.81349853.079.5086.090.0098.0
1.040.074.95000011.64419153.065.0074.584.2598.0
2.044.064.27272711.95605137.054.7562.574.0088.0
3.056.050.28571413.55086624.041.0048.563.0078.0
4.086.041.00000016.6810924.030.0041.550.0087.0
5.078.021.60256417.6512682.08.0019.029.7587.0

Below is a discussion of the metrics from the table above and what each result indicates about the relationship between `school_rating` and `reduced_lunch`:

count: the number of schools at each rating. Most of the schools in Sally's sample have a 4- or 5-star rating, but 25% of schools have a 1-star rating or below. This confirms that poor school performance isn't merely anecdotal, but a serious problem that deserves attention.

mean: the average percentage of students on `reduced_lunch` among all schools by each `school_rating`. As school performance increases, the average number of students on reduced lunch decreases. Schools with a 0-star rating have 83.6% of students on reduced lunch. And on the other end of the spectrum, 5-star schools on average have 21.6% of students on reduced lunch. We'll examine this pattern further. in the graphing section.

std: the standard deviation of the variable. Referring to the `school_rating` of 0, a standard deviation of 8.813498 indicates that 68.2% (refer to readme) of all observations are within 8.81 percentage points on either side of the average, 83.6%. Note that the standard deviation increases as `school_rating` increases, indicating that `reduced_lunch` loses explanatory power as school performance improves. As with the mean, we'll explore this idea further in the graphing section.

min: the minimum value of the variable. This represents the school with the lowest percentage of students on reduced lunch at each school rating. For 0- and 1-star schools, the minimum percentage of students on reduced lunch is 53%. The minimum for 5-star schools is 2%. The minimum value tells a similar story as the mean, but looking at it from the low end of the range of observations.

25%: the bottom quartile; represents the lowest 25% of values for the variable, `reduced_lunch`. For 0-star schools, 25% of the observations are less than 79.5%. Sally sees the same trend in the bottom quartile as the above metrics: as `school_rating` increases the bottom 25% of `reduced_lunch` decreases.

50%: the second quartile; represents the lowest 50% of values. Looking at the trend in `school_rating` and `reduced_lunch`, the same relationship is present here.

75%: the top quartile; represents the lowest 75% of values. The trend continues.

max: the maximum value for that variable. You guessed it: the trend continues!

The descriptive statistics consistently reveal that schools with more students on reduced lunch under-perform when compared to their peers. Sally is on to something.

Sally decides to look at `reduced_lunch` from another angle using a correlation matrix with pandas' `corr` method. The values in the correlation matrix table will be between -1 and 1 (see below). A value of -1 indicates the strongest possible negative correlation, meaning as one variable decreases the other increases. And a value of 1 indicates the opposite. The result below, -0.815757, indicates strong negative correlation between `reduced_lunch` and `school_rating`. There's clearly a relationship between the two variables.

``````# only view these two variables
df[['reduced_lunch', 'school_rating']].corr()
``````
Out[19]:
reduced_lunchschool_rating
reduced_lunch1.000000-0.815757
school_rating-0.8157571.000000

Sally continues to explore this relationship graphically.

## Essential Graphs for Exploring Data

### Box-and-Whisker Plot

In her stats book, Sally sees a box-and-whisker plot. A box-and-whisker plot is helpful for visualizing the distribution of the data from the mean. Understanding the distribution allows Sally to understand how far spread out her data is from the mean; the larger the spread from the mean, the less robust `reduced_lunch` is at explaining `school_rating`.

See below for an explanation of the box-and-whisker plot.

Now that Sally knows how to read the box-and-whisker plot, she graphs `reduced_lunch` to see the distributions. See below.

``````%matplotlib inline  # plot graphs in your notebook
import matplotlib.pyplot as plt
import seaborn as sns

fig = plt.figure(figsize=(14,8)) # figure size
ax = fig.add_subplot(111) # axis information
ax.set_ylabel('school_rating') # sets label for y-axis

# boxplot with only these two variables
_ = df[['reduced_lunch', 'school_rating']].boxplot(by='school_rating', figsize=(13,8), vert=False, sym='b.', ax=ax)
``````

In her box-and-whisker plots, Sally sees that the minimum and maximum `reduced_lunch` values tend to get closer to the mean as `school_rating` decreases; that is, as `school_rating` decreases so does the standard deviation in `reduced_lunch`.

What does this mean?

Starting with the top box-and-whisker plot, as `school_rating` decreases, `reduced_lunch` becomes a more powerful way to explain outcomes. This could be because as parents' incomes decrease they have fewer resources to devote to their children's education (such as, after-school programs, tutors, time spent on homework, computer camps, etc) than higher-income parents. Above a 3-star rating, more predictors are needed to explain `school_rating` due to an increasing spread in `reduced_lunch`.

Having used box-and-whisker plots to reaffirm her idea that household income and school performance are related, Sally seeks further validation.

### Scatter Plot

To further examine the relationship between `school_rating` and `reduced_lunch`, Sally graphs the two variables on a scatter plot. See below.

``````plt.figure(figsize=(14,8)) # set the size of the graph
_ = sns.regplot(data=df, x='reduced_lunch', y='school_rating')
``````

In the scatter plot above, each dot represents a school. The placement of the dot represents that school's rating (Y-axis) and the percentage of its students on reduced lunch (x-axis).

The downward trend line shows the negative correlation between `school_rating` and `reduced_lunch` (as one increases, the other decreases). The slope of the trend line indicates how much `school_rating` decreases as `reduced_lunch` increases. A steeper slope would indicate that a small change in `reduced_lunch` has a big impact on `school_rating` while a more horizontal slope would indicate that the same small change in `reduced_lunch` has a smaller impact on `school_rating`.

Sally notices that the scatter plot further supports what she saw with the box-and-whisker plot: when `reduced_lunch` increases, `school_rating` decreases. The tighter spread of the data as `school_rating` declines indicates the increasing influence of `reduced_lunch`. Now she has a hypothesis.

### Correlation Matrix

Sally is ready to test her hypothesis: a negative relationship exists between `school_rating` and `reduced_lunch` (to be covered in a follow up article). If the test is successful, she'll need to build a more robust model using additional variables. If the test fails, she'll need to re-visit her dataset to choose other variables that possibly explain `school_rating`. Either way, Sally could benefit from an efficient way of assessing relationships among her variables.

An efficient graph for assessing relationships is the correlation matrix, as seen below; its color-coded cells make it easier to interpret than the tabular correlation matrix above. Red cells indicate positive correlation; blue cells indicate negative correlation; white cells indicate no correlation. The darker the colors, the stronger the correlation (positive or negative) between those two variables.

``````# create tabular correlation matrix
corr = df.corr()
_, ax = plt.subplots(figsize=(13,10))

# graph correlation matrix
_ = sns.heatmap(corr, ax=ax,
xticklabels=corr.columns.values,
yticklabels=corr.columns.values)
``````

With the correlation matrix in mind as a future starting point for finding additional variables, Sally moves on for now and prepares to test her hypothesis.

## Summary

Sally was approached with a problem: why are some schools in middle Tennessee under-performing? To answer this question, she did the following:

1. Conducted a lit review to educate herself on the topic.
2. Gathered data from a reputable source to explore school ratings and characteristics of the student bodies and schools in middle Tennessee.
3. The data indicated a robust relationship between `school_rating` and `reduced_lunch`.
4. Explored the data visually.
5. Though satisfied with her preliminary findings, Sally is keeping her mind open to other explanations.
6. Developed a hypothesis: a negative relationship exists between `school_rating` and `reduced_lunch`.

In a follow up article, Sally will test her hypothesis. Should she find a satisfactory explanation for her sample of schools, she will attempt to apply her explanation to the population of schools in Tennessee.

#### Tim Dobbins

A graduate of Belmont University, Tim is a Nashville, TN-based software engineer and statistician at Perception Health, an industry leader in healthcare analytics, and co-founder of Sidekick, LLC, a data consulting company. Find him on Twitter and GitHub.

#### John Burke

John is a research analyst at Laffer Associates, a macroeconomic consulting firm based in Nashville, TN. He graduated from Belmont University. Find him on GitHub and LinkedIn

Guest
S sahoo

Nice explanation

Guest
Bob

Thanks. Very nice concise example

Send this to a friend