Introduction to Null Hypothesis Testing with Python

Why null hypothesis testing matters

Imagine you launch a new product feature and see a higher average conversion rate. Is the improvement real, or could it be random noise? Null hypothesis testing gives us a disciplined way to answer that question using data.

At the heart of classical statistics is the null hypothesis: a precise claim about the world that we assume is true until the data provide strong evidence otherwise. We then measure how surprising our data would be if that null hypothesis were really true.

Type I and Type II errors

Because we are working with samples rather than entire populations, two types of mistakes are always possible:

• Type I error (false positive): rejecting a true null hypothesis.
• Type II error (false negative): failing to reject a false null hypothesis.

The significance level (usually α = 0.05) controls how much Type I error risk we are willing to tolerate. Statistical power (1 − β) measures how likely we are to detect a real effect when it exists.

The logic of a hypothesis test

Regardless of the specific test we use (t-test, χ² test, etc.), most hypothesis tests follow the same logical pattern. In this tutorial we will implement that pattern in Python using real data.

1. State the null and alternative hypotheses
   Translate your question into a precise statement about a population parameter (mean, proportion, etc.). For example, “The mean sepal length of Setosa is 5.0 cm” versus “It is not 5.0 cm”.
2. Choose a test statistic and significance level
   Select a test (such as a one-sample t-test) and fix a significance level, often α = 0.05. This sets the threshold for how surprising the data must be before we reject the null.
3. Check assumptions
   Most tests assume things like independent observations, approximate normality, and similar variances across groups. We will use simple normality checks later in this article.
4. Compute the test statistic and p-value in Python
   Using libraries like scipy.stats, we calculate a test statistic (e.g., t-value) and its corresponding p-value from our sample.
5. Make a decision and interpret in context
   If the p-value is below our chosen α, we reject the null hypothesis. More importantly, we interpret the result in the context of the problem, not just as “significant” or “not significant”.

The real dataset: Iris flower measurements

To keep this tutorial concrete and reproducible, we will use the classic Iris dataset: real measurements of sepal and petal dimensions for three species of iris flowers (Setosa, Versicolor, and Virginica).

We will work with a compact CSV containing just the columns we need: species name and sepal length in centimetres.

Step-by-step: One-sample t-test in Python

Our first example tests whether the average sepal length of the Setosa species is equal to 5.0 cm. This is a classic one-sample t-test.

We will set up:

• Null hypothesis H₀: The mean sepal length of Setosa is 5.0 cm.
• Alternative hypothesis H₁: The mean sepal length of Setosa is not 5.0 cm.

import pandas as pd
from scipy import stats

# 1. Load the CSV (same structure as iris_sepal.csv used on the PLEX site)
df = pd.read_csv("data/iris_sepal.csv")

# 2. Filter to Setosa only
setosa = df[df["species"] == "setosa"]["sepal_length_cm"]

print("Setosa summary:")
print(setosa.describe())

# 3. Define the null hypothesis mean
mu_0 = 5.0  # H0: true mean sepal length is 5.0 cm

# 4. Run a one-sample t-test
t_statistic, p_value = stats.ttest_1samp(setosa, popmean=mu_0)
print(f"t = {t_statistic:.3f}, p = {p_value:.4f}")

# 5. Decision rule at alpha = 0.05
alpha = 0.05
if p_value < alpha:
    print("Reject H0: evidence that Setosa mean sepal length differs from 5.0 cm.")
else:
    print("Fail to reject H0: data are compatible with a mean of 5.0 cm.")
    

Visualising the Setosa sepal length distribution

Seeing the distribution of Setosa sepal lengths helps build intuition for the test and for the concept of “typical” values.

Illustrative distribution of Setosa sepal lengths, based on the Iris dataset (counts by 0.2–0.3 cm bins).

Two-sample t-test: comparing species

Our second example compares the mean sepal length of two species: Versicolor and Virginica. This is a two-sample independent t-test.

We now test:

• Null hypothesis H₀: The mean sepal length of Versicolor equals that of Virginica.
• Alternative hypothesis H₁: The mean sepal lengths differ between the two species.

import pandas as pd
from scipy import stats

df = pd.read_csv("data/iris_sepal.csv")

versicolor = df[df["species"] == "versicolor"]["sepal_length_cm"]
virginica = df[df["species"] == "virginica"]["sepal_length_cm"]

print("Versicolor mean:", versicolor.mean())
print("Virginica mean:", virginica.mean())

t_statistic, p_value = stats.ttest_ind(versicolor, virginica, equal_var=True)
print(f"t = {t_statistic:.3f}, p = {p_value:.4f}")

alpha = 0.05
if p_value < alpha:
    print("Reject H0: evidence of a difference in mean sepal length.")
else:
    print("Fail to reject H0: no strong evidence of a difference in mean sepal length.")
    

Visual comparison of species means

A simple visual comparison of the two sample means makes the test result easier to interpret.

Illustrative comparison of mean sepal lengths for Versicolor and Virginica, based on the Iris dataset.

Checking assumptions: normality and beyond

Many parametric tests, including the t-test, assume that the sample mean is approximately normally distributed. For moderate sample sizes this often holds by the central limit theorem, but it is still good practice to check.

from scipy import stats

# Using the Setosa sample from before
shapiro_stat, shapiro_p = stats.shapiro(setosa)
print(f"Shapiro-Wilk W = {shapiro_stat:.3f}, p = {shapiro_p:.4f}")

alpha = 0.05
if shapiro_p < alpha:
    print("Reject normality: Setosa sepal lengths deviate from a normal distribution.")
else:
    print("Fail to reject normality: no strong evidence against normality.")
    

Common mistakes and better practices

Hypothesis testing is widely used and just as widely misinterpreted. Here are some common pitfalls and how to avoid them.

Complete Python script: putting it all together

The script below loads the data, runs the one-sample and two-sample t-tests, checks normality for Setosa, and prints human-readable interpretations.

#!/usr/bin/env python3
"""
Introduction to null hypothesis testing with Python.

Requires:

* pandas
* scipy
* data/iris_sepal.csv in the working directory
"""

import pandas as pd
from scipy import stats

# Load dataset
df = pd.read_csv("data/iris_sepal.csv")

setosa = df[df["species"] == "setosa"]["sepal_length_cm"]
versicolor = df[df["species"] == "versicolor"]["sepal_length_cm"]
virginica = df[df["species"] == "virginica"]["sepal_length_cm"]

print("=== Sample summaries ===")
print("Setosa:\n", setosa.describe(), "\n")
print("Versicolor:\n", versicolor.describe(), "\n")
print("Virginica:\n", virginica.describe(), "\n")

alpha = 0.05

# 1) One-sample t-test for Setosa mean sepal length
mu_0 = 5.0
t_1samp, p_1samp = stats.ttest_1samp(setosa, popmean=mu_0)
print("=== One-sample t-test: Setosa mean sepal length ===")
print(f"H0: mu_setosa = {mu_0} cm")
print(f"t = {t_1samp:.3f}, p = {p_1samp:.4f}")
if p_1samp < alpha:
    print("Result: Reject H0 (evidence that the mean differs from 5.0 cm).\n")
else:
    print("Result: Fail to reject H0 (data are compatible with a mean of 5.0 cm).\n")

# 2) Two-sample t-test: Versicolor vs Virginica
t_2samp, p_2samp = stats.ttest_ind(versicolor, virginica, equal_var=True)
print("=== Two-sample t-test: Versicolor vs Virginica sepal length ===")
print("H0: mu_versicolor = mu_virginica")
print(f"t = {t_2samp:.3f}, p = {p_2samp:.4f}")
if p_2samp < alpha:
    print("Result: Reject H0 (evidence of a difference in mean sepal length).\n")
else:
    print("Result: Fail to reject H0 (no strong evidence of a difference).\n")

# 3) Normality check for Setosa using Shapiro-Wilk
shapiro_stat, shapiro_p = stats.shapiro(setosa)
print("=== Normality check: Setosa sepal length (Shapiro-Wilk) ===")
print(f"W = {shapiro_stat:.3f}, p = {shapiro_p:.4f}")
if shapiro_p < alpha:
    print("Result: Reject normality (evidence of deviation from normality).\n")
else:
    print("Result: Fail to reject normality (no strong evidence against normality).\n")
    

Related PLEX reading

References & further reading

• Real Python — Hypothesis Testing in Python
  Practical walkthrough of hypothesis tests and p-values using SciPy and Python.
• SciPy Documentation — scipy.stats
  Official reference for the statistical tests used in this article, including ttest_1samp and ttest_ind.
• Statsmodels — Statistical Modeling in Python
  Python library for regression, time series, and advanced statistical tests.
• Wikipedia — Statistical Hypothesis Testing
  Broad overview of hypothesis tests, p-values, and test statistics.
• Nature — Scientists rise up against statistical significance
  Discussion of common misuses of p-values and statistical significance in research.
• DataCamp — Understanding p-values in Python
  Introductory tutorial on p-values with Python examples.
• UCI Machine Learning Repository — Iris Data Set
  Original source of the Iris measurements used in this tutorial.
• FRED — Federal Reserve Economic Data
  While not used directly here, FRED is a rich source of real-world data for applying hypothesis testing in economics.