pandas dataframes and data cleaning

Learning objectives

• Create and inspect a DataFrame.
• Select columns and filter rows.
• Find missing values.
• Explain every cleaning decision.

Install pandas:

python -m pip install pandas

Inspect a DataFrame

import pandas as pd

data = pd.DataFrame({
    "name": ["Ari", "Bea", "Chen"],
    "minutes": [30, None, 45],
    "completed": [True, False, True],
})

print(data.head())
print(data.isna().sum())

head() gives a quick view. isna().sum() counts missing values in each column. Do not clean a missing value until you understand why it is missing.

Filter and derive a column

complete = data[data["completed"]]
data["hours"] = data["minutes"] / 60

print(complete["name"].tolist())
print(data["hours"].round(2).tolist())

['Ari', 'Chen']
[0.5, nan, 0.75]

The missing value remains visible as nan. Replacing it with zero would claim that the student practiced for zero minutes, which may be false.

Guided practice

1. Add a project column.
2. Count completed projects.
3. Decide whether to remove, fill, or investigate the missing minutes value.
4. Write the reason for your decision in a Markdown cell.

Checkpoint

Why is “drop every row with a missing value” not always a good rule? Give one example where it could remove useful information.

See the pandas user guide for inspection, selection, and missing-data operations.