Lab 4: IMU Activity and Gesture Recognition IoT

Date: July 9
Time: 1:00-5:00 PM
TA: Shanmu Wang
Hardware: ESP32 board with BMI270 IMU

Goals

By the end of this lab, each group should be able to:

• explain what an IMU measures and why accelerometer readings include gravity;
• stream 6-axis IMU readings from an ESP32 over USB serial;
• identify the physical x/y/z axes of the IMU board from accelerometer and gyroscope plots;
• implement a moving-average filter and explain the noise-latency tradeoff;
• explain why dead-reckoned trajectories from IMU double integration drift quickly;
• compare raw and smoothed trajectory estimates;
• collect short labeled IMU motion segments and train a simple live classifier.

Form a group of three and come up with a cool name: Shared Google Sheet.

Lab Code

Use the public IMU lab repository as the source of truth:

https://github.com/wshanmu/IMU_lab_students

Clone it once:

git clone https://github.com/wshanmu/IMU_lab_students.git
cd IMU_lab

Before lab, update to the latest version:

git pull

If git pull reports local changes, ask a TA before running reset, checkout, or other commands that overwrite files.

The main files used in this lab are:

File	Purpose
tools/plot_imu_serial.py	Live raw accelerometer and gyroscope plot.
tools/plot_imu_smoothed_TODO.py	Student TODO for moving-average smoothing.
tools/capture_trajectory.py	Captures a short motion and estimates a 2D XY trajectory.
tools/capture_trajectory_smoothed_TODO.py	Student TODO for adding smoothing before trajectory estimation.
tools/IMU_Classifier/python_imu_segment_demo_student.py	Student TODO for a simple 3-class IMU classifier.

The firmware has already been flashed for this lab. Students only need the Python tools unless a TA asks them to rebuild or reflash the ESP32.

System Overview

The lab device uses an ESP32 as the microcontroller and a BMI270 as the IMU sensor. The ESP32 communicates with the BMI270 over I2C, then prints each accelerometer and gyroscope sample over USB serial.

BMI270 IMU -> I2C -> ESP32 -> USB serial -> Python visualization scripts

The firmware streams lines like:

accel[g] x= 0.012 y=-0.034 z= 0.998 | gyro[dps] x= 0.10 y=-0.20 z= 0.05

In the prepared firmware, the IMU is configured for approximately 100 Hz streaming, accelerometer range +/-4 g, and gyroscope range +/-1000 dps. The Python scripts parse the serial text and plot the six channels in real time.

Firmware Note

The ESP32 firmware should already be flashed for this lab. You set up ESP-IDF in Lab 3, and this lab reuses the same VS Code extension and serial-port workflow if a board needs recovery.

If a TA asks you to reflash the IMU firmware:

1. Open the IMU_lab repository in VS Code.
2. Run ESP-IDF: Set Espressif Device Target and choose esp32.
3. Run ESP-IDF: Select Port to Use.
4. Run ESP-IDF: Build your Project.
5. Run ESP-IDF: Flash your Project.
6. Run ESP-IDF: Monitor your Device and check for accel[g] ... gyro[dps] ... output.

Useful reference: Bosch BMI270 product page.

Python Setup

Open a terminal in the lab code folder:

cd IMU_lab/tools

Activate the Conda environment from Lab 1 and install the IMU lab packages:

conda activate cosmos-ds
python -m pip install -r requirements.txt

If conda activate cosmos-ds fails, return to Lab 1 and create the course environment before continuing.

Find the ESP32 serial port.

macOS:

ls /dev/cu.* 2>/dev/null

Windows:

Open Device Manager -> Ports (COM & LPT), then look for the new COM port.

Use your actual port in the commands below. Examples:

• macOS: /dev/cu.usbserial-5B1F0080901
• Windows: COM5

Four-Hour Plan

1:00-1:20 PM - Demo and IMU Concepts

The TA will first show the live visualization and a working classifier demo.

An inertial measurement unit, or IMU, usually contains:

• Accelerometer: measures specific force along x/y/z. When the board is stationary, one or more axes still measure gravity.
• Gyroscope: measures angular velocity around x/y/z.

Useful checks:

• If the board is still, the acceleration magnitude should be close to 1 g.
• If one axis points upward, that accelerometer channel should be close to +1 g or -1 g.
• If you rotate around one board axis, the matching gyroscope channel should have the largest response.

1:20-1:45 PM - Visualize Raw 6-Axis Data

Run the live plot:

python plot_imu_serial.py --port YOUR_PORT --baud 115200

Replace YOUR_PORT with your serial port.

Do these tests:

1. Hold the board still in at least three orientations.
2. Move the board back and forth along one physical direction.
3. Rotate the board around each physical axis.
4. Watch which accelerometer or gyroscope channel changes the most.

Deliverable for this step: fill in a table like this.

Physical motion	Channel with strongest response	Evidence from plot
Board x direction
Board y direction
Board z direction
Rotation around x
Rotation around y
Rotation around z

1:45-2:25 PM - Implement Moving-Average Smoothing

Raw IMU signals are noisy because of MEMS sensor noise, hand tremor, quantization, board vibration, and small timing variation. A moving average reduces high-frequency noise by replacing each new sample with the average of the most recent N samples.

For a window of size N:

smoothed[t] = mean(raw[t-N+1], ..., raw[t])

In plot_imu_smoothed_TODO.py, implement the MovingAverageFilter.update() method. The intended data structure is:

• a deque that stores the most recent samples;
• a running_sum that avoids recomputing the full sum every update;
• one update operation that appends the newest value, removes the oldest value if the window is too long, and returns the current average.

Run your version:

python plot_imu_smoothed_TODO.py --port YOUR_PORT --baud 115200 --average-window 5

Try at least three window sizes, such as 3, 10, and 30.

Checkpoint questions:

• Which window size removes the most visible noise?
• Which window size creates the most delay?
• Why is too much smoothing bad for fast gestures?

2:25-3:15 PM - Capture a Short Trajectory

In principle, a trajectory can be estimated from acceleration:

acceleration -> integrate once -> velocity -> integrate again -> position

In practice, this is difficult. Gravity is about 9.8 m/s^2, while many hand motions create smaller linear accelerations. A small gravity-removal error or sensor bias becomes a growing velocity error after integration, then a larger position error after the second integration.

Run the prepared trajectory demo:

python capture_trajectory.py --port YOUR_PORT --baud 115200 --capture-seconds 5

Keep the board still for one second, press c, then perform a short motion while the script captures data. Try:

• a short straight-line motion;
• a small square;
• a small circle;
• a motion that starts and ends at the same place.

What the script is doing:

1. Uses the pre-capture samples as a baseline for gyro bias and gravity direction.
2. Uses gyroscope readings to update orientation during the capture.
3. Rotates acceleration into a world frame.
4. Subtracts gravity.
5. Applies small deadbands and damping to reduce obvious drift.
6. Integrates acceleration to velocity and velocity to position.
7. Plots the reconstructed XY path.

Thinking question: why does this lab visualize only the 2D XY trajectory instead of a full 3D path?

3:15-3:50 PM - Add Smoothing Before Trajectory Estimation

Copy the moving-average idea from plot_imu_smoothed_TODO.py into capture_trajectory_smoothed_TODO.py.

Run:

python capture_trajectory_smoothed_TODO.py --port YOUR_PORT --baud 115200 --capture-seconds 5 --average-window 5

Compare the raw and smoothed trajectories for the same type of motion.

Deliverable for this step:

• one screenshot or sketch of a raw trajectory;
• one screenshot or sketch of a smoothed trajectory;
• a short answer: does smoothing solve trajectory drift, or does it only reduce part of the noise?

Expected observation: smoothing can make the path less jittery, but it cannot remove constant bias, imperfect gravity subtraction, orientation error, or integration drift. A very large window can also lag behind the real motion.

3:50-4:40 PM - Train a Simple IMU Motion Classifier

Open the classifier folder:

cd IMU_Classifier
python -m pip install -r requirements-python.txt

Run the student version:

macOS/Linux:

./run_student.sh --port YOUR_PORT --classes Still,Shake,Turn

Windows PowerShell:

python python_imu_segment_demo_student.py --port YOUR_PORT --classes Still,Shake,Turn

You may choose a different three-class task set, such as:

• Still,Shake,Turn
• Tap,Circle,Flip
• UpDown,LeftRight,Rotate

Workflow:

1. Select a class.
2. Press c, Space, Enter, or the Capture 1 s button.
3. Perform the selected motion for the next second.
4. Collect at least five examples per class if time allows.
5. Press t or Train.
6. Perform new examples and observe the live prediction.

Student TODO: in python_imu_segment_demo_student.py, add summary statistics to the feature vector. Good first features are:

• standard deviation of each channel;
• min and max of each channel;
• range of each channel;
• RMS energy of each channel;
• mean absolute first difference of each channel.

    def capture_instance(
        self,
        label: str | None,
        samples: list[ImuSample],
    ) -> DataInstance:
        normalized_samples = [normalize_imu_sample(sample) for sample in samples]
        if len(normalized_samples) < self.min_segment_samples:
            raise RuntimeError(
                f"Need at least {self.min_segment_samples} IMU samples; got {len(normalized_samples)}."
            )

        data = self.np.asarray(normalized_samples, dtype="float32")
        times = data[:, 0]
        values = data[:, 1:7]
        duration_s = float(times[-1] - times[0])
        if duration_s <= 0.0:
            raise RuntimeError("Segment duration is zero.")
        if duration_s < self.segment_seconds * self.min_segment_coverage:
            raise RuntimeError(
                f"Need about {self.segment_seconds:.1f} seconds of data; got {duration_s:.2f} seconds."
            )

        rel_time = times - times[0]
        resampled = self.np.column_stack(
            [
                self.np.interp(self.target_time, rel_time, values[:, channel])
                for channel in range(values.shape[1])
            ]
        ).astype("float32")

        means = resampled.mean(axis=0)
        centered = resampled - means
        stds = centered.std(axis=0)
        mins = resampled.min(axis=0)
        maxs = resampled.max(axis=0)
        ranges = maxs - mins
        rms = self.np.sqrt(self.np.mean(centered * centered, axis=0))
        diffs = self.np.diff(resampled, axis=0)
        diff_abs_mean = self.np.mean(self.np.abs(diffs), axis=0)
        diff_std = diffs.std(axis=0)

        summary = self.np.concatenate(
            [means, stds, mins, maxs, ranges, rms, diff_abs_mean, diff_std]
        ).astype("float32")
        measurements = self.np.concatenate([centered.T.reshape(-1), summary]).astype("float32")

        return DataInstance(
            label=label,
            measurements=measurements,
            sample_count=len(normalized_samples),
            duration_s=duration_s,
            raw_samples=normalized_samples,
        )

Discussion questions:

• Which three motions are easiest to separate?
• Does the classifier fail when a different student performs the same motion?
• Does adding summary features make predictions more stable?

4:40-5:00 PM - Checkoff and Cleanup

Each group should show the TA:

1. live raw IMU data from the ESP32;
2. the completed axis-response table;
3. moving-average smoothing running with at least two window sizes;
4. one trajectory capture and one explanation of why it drifts;
5. the smoothed trajectory comparison;
6. one trained 3-class live classifier.

Final Questions

Answer these briefly in your group notes:

1. What does the accelerometer measure when the board is completely still?
2. How did you identify the physical x/y/z axes?
3. What is the tradeoff when increasing the moving-average window size?
4. Why does double integration of IMU acceleration drift so quickly?
5. Why is the XY trajectory easier to interpret than the full 3D trajectory in this lab?
6. Which features helped the classifier distinguish your motions?

Troubleshooting

Problem	Likely Cause	Fix
No serial data	Wrong port, charge-only cable, or board not powered	Try another USB cable, check Device Manager or /dev/cu.*, reconnect the board.
Port is busy	Another program has the serial port open	Close ESP-IDF monitor, Arduino serial monitor, PuTTY, or another Python script.
Plot opens but stays flat	Board is not streaming expected text format	Ask the TA to confirm the firmware and baud rate.
ModuleNotFoundError	Python packages not installed in the active environment	Activate cosmos-ds and run python -m pip install -r requirements.txt.
Qt or PyQt error	GUI backend issue	Restart the terminal, confirm PyQt6 installed, or ask a TA for the backup computer.
Classifier predictions unstable	Too few examples or motions too similar	Collect more examples and choose more distinct motions.