Introduction to Pandas for Physics: Data Cleaning & Missing Experimental Data - Computational Physics Formulation & Simulation

Program 1

Data Cleaning & Missing Experimental Data

Python 3.11 Ready

import pandas as pd
import numpy as np

# Simulated telemetry stream from an atmospheric physics observatory
# np.nan simulates temporary sensor dropouts or wireless connection drop
sensor_stream = {
    'time_min': [0, 5, 10, 15, 20, 25, 30, 35, 40, 45],
    'solar_irradiance_W_m2': [842.1, 846.5, np.nan, 852.0, 855.4, np.nan, np.nan, 863.1, 866.7, 869.2],
    'barometric_pressure_hPa': [1013.2, 1013.1, 1013.0, np.nan, 1012.8, 1012.7, 1012.5, 1012.4, np.nan, 1012.1],
    'sensor_flag': ['VALID', 'VALID', 'DROPOUT', 'VALID', 'VALID', 'ERROR', 'ERROR', 'VALID', 'VALID', 'VALID']
}

df_telemetry = pd.DataFrame(sensor_stream)

print("=== Raw Telemetry with Sensor Dropouts ===")
print(df_telemetry)

# 1. Null Value Diagnostics: Tally missing physical readings
print("\nMissing Measurement Count per Sensor Channel:")
print(df_telemetry.isna().sum())

# 2. Create clean working copy
df_clean = df_telemetry.copy()

# 3. Forward-Fill (ffill) for high-inertia thermodynamic state (Barometric Pressure)
df_clean['barometric_pressure_hPa'] = df_clean['barometric_pressure_hPa'].ffill()

# 4. Linear Interpolation for continuous physical fields (Solar Irradiance)
df_clean['solar_irradiance_W_m2'] = df_clean['solar_irradiance_W_m2'].interpolate(method='linear')

# 5. Type Casting and Filtering
df_clean['time_min'] = df_clean['time_min'].astype(int)
df_clean['barometric_pressure_hPa'] = df_clean['barometric_pressure_hPa'].astype(float)
df_clean['is_reconstructed'] = df_telemetry['sensor_flag'] != 'VALID'

print("\n=== Reconstructed & Cleaned Physics Dataset ===")
print(df_clean[['time_min', 'solar_irradiance_W_m2', 'barometric_pressure_hPa', 'is_reconstructed']].to_string(index=False))

# Verify all numerical gaps are resolved
remaining_nans = df_clean[['solar_irradiance_W_m2', 'barometric_pressure_hPa']].isna().sum().sum()
print(f"\nVerification: {remaining_nans} missing values remaining. Data stream continuous and valid.")

Click "Run Code" to execute script and render figures.

Mathematical Problem Formulation

Physical Model: Reconstruction of Missing Experimental Observables:

In experimental astrophysics and environmental telemetry, sensors experience intermittent dropouts where recorded data is missing ($\text{NaN}$).

1. Physical Continuity & Linear Interpolation:

For smoothly varying continuous physical fields (such as atmospheric solar irradiance $I(t)$), a missing value at time $t \in [t_0, t_1]$ is reconstructed via the linear interpolation scheme:

$$I(t) = I(t_0) + \frac{I(t_1) - I(t_0)}{t_1 - t_0} (t - t_0)$$

2. Thermodynamic Persistence (Forward-Fill):

For high-inertia thermodynamic variables such as barometric pressure $P(t)$, when sensor disconnection occurs for brief intervals ($\Delta t \ll \tau_{\text{relax}}$), the previous equilibrium value provides a reliable estimate:

$$P(t) = P(t - \Delta t)$$

Theoretical Background & Computational Mechanics

1. Why Experimental Data Cleaning is Critical:

Raw physics laboratory streams rarely arrive in pristine condition. Loose probe connections, voltage sags, ADC overflow errors, and wireless packet drops introduce NaN (Not a Number) values into data arrays.

2. Methods of Handling Missing Data in Pandas:

.isna().sum(): Rapid diagnostic tool to tally missing sensor readings per physical channel.
.dropna(): Removes records containing missing values. In physics time series, blind dropping can distort temporal continuity and step-sizes $\Delta t$.
.interpolate(method='linear'): Smoothly estimates missing samples based on physical continuity.
.ffill() / .bfill(): Forward or backward propagation of persistent states.

3. Data Type Validation:

Sensor loggers often store numbers as text strings due to formatting anomalies. Using .astype() or pd.to_numeric() ensures physical values are true 64-bit IEEE floating-point numbers ready for numerical computation.

Theoretical Principles & Computational Physics Guide

Data Cleaning & Missing Experimental Data: Computational Physics Modeling & Analysis

Dr. Alorika Chatterjee & Dr. Anirban Shaw Science & Physics Learners Python 3 / NumPy / SciPy

1. Core Principles & Theoretical Framework

Core Concept: Computational formulation and simulation of Data Cleaning & Missing Experimental Data within the study of Introduction to Pandas for Physics.

Algorithmic Translation: Translates theoretical governing equations into clean, discrete Python procedures and verified numerical solvers.

Numerical Verification: Evaluates numerical accuracy, parameter sensitivity, and error bounds for reliable computational results.

2. Mathematical Formulation & Algorithmic Pipeline

Governing computational formulation for Data Cleaning & Missing Experimental Data:

$$ \mathcal{F}_{\text{model}}(\mathbf{x}, t; \mathbf{p}) = 0 \implies \mathbf{y}_{n+1} = \mathcal{S}(\mathbf{y}_n, \Delta t, \mathbf{p}) $$

1. Define Physical Parameters & Boundaries → 2. Execute Numerical Simulation Steps → 3. Evaluate Observables & Plot Results

3. Numerical Mechanics & Implementation Logic

Parameter Initialization: Set physical constants, initial states, and coordinate boundary bounds for Data Cleaning & Missing Experimental Data.
Step Execution: Advance the computational state using optimized vector operations and scientific libraries.
Convergence Control: Monitor stability and ensure simulation metrics remain physically meaningful.

4. Relevant Computational Practice Exercises

Exercise 1

Implement Data Cleaning & Missing Experimental Data and observe how the output responds to variations in primary physical parameters.

Exercise 2

Refine the calculation grid or step size to observe convergence and reduction in numerical approximation error.

Exercise 3

Compare the simulated computational results against an analytical benchmark or limiting physical case.

Standard Literature & References:

Mark Newman, "Computational Physics", University of Michigan (2013).
Hans Petter Langtangen, "A Primer on Scientific Programming with Python", Springer (2016).

Ch. 8.4 Data Cleaning & Missing Experimental Data