Counting reps without knowing the exercise

Form Coach takes a filmed set and reports on it, and before it can report on anything it has to answer the question underneath all the others: where are the reps?

The tempting design (the design you have already thought of, sitting there, three sentences in) is a lookup table. Squat means watch hip height. Bench means watch elbow angle. And so on, forever, because "and so on" is exactly what a lookup table costs: every new exercise is a code change, and the lifter doing anything slightly off-menu gets nothing at all. We have all maintained that table somewhere, in some codebase, for some product. I'm not doing it again.

So the detector is class-agnostic. It does not know what lift it is watching. It has never heard of a squat.

Measure everything a body can repeat in

Five signals per frame, chosen to survive not knowing where the camera was:

• hip_to_ankle: distance
• wrist_to_shoulder: distance
• wrist_to_hip: distance
• knee joint angle
• elbow joint angle

Distances and angles, because each lies in a different place. A chord between two landmarks goes numb near full extension: the last fifteen degrees of a knee lockout barely move hip-to-ankle at all. The joint angle is beautifully linear right there, and mute about any movement the joint doesn't bend in: a shrug has no joint angle worth the name and a perfectly legible distance signature.

Each signal is also read in both directions, because a rep is not always a shortening. A bench press starts at lockout and closes wrist-to-shoulder; an overhead press starts racked and opens it. Same wire, opposite current.

Then pick the signal the movement actually lives in

Here is where the first version was wrong, and wrong in the particular way that takes weeks to see because it looks so much like sense.

The original rule: whichever signal produced the most reps wins.

Exactly backwards. Noise generates more cycles than lifting does. Of course it does. Noise doesn't get tired.

The case that finally forced the point: a barbell squat, bar racked on the shoulders. The elbows are locked for the whole set; the elbow angle is, physically, a constant. But the elbow signal is not. Pose estimation carries five to ten degrees of jitter per joint, uncorrelated frame to frame, so the elbow channel oscillated fast and shallow, out-counted the actual squat handily, and won.

The counter reported a set of squats as sixteen reps of nothing. Sixteen. It was very sure.

Score the shape, not the count

The replacement asks each candidate a different question, not how many but how much do you look like a set, and lets the count matter only through that. Three factors, multiplied:

detectionScore = rhythm × regularity × margin

Rhythm is self-similarity: the signal correlated against a time-shifted copy of itself. Reps are periodic, so the correlation peaks at the rep period. The virtue is that it's a whole-clip fit rather than a threshold crossing: a few bad frames barely dent a correlation computed over hundreds of pairs, where two bad frames can conjure or murder an individual crossing. A clip too short to hold a second period scores a neutral 0.75 rather than zero; unavailable is not the same as bad, a distinction I keep having to make on behalf of software.

Regularity asks whether the detected reps are evenly spaced. Real sets are metronomic in a way noise never bothers to be.

Margin asks how far past the minimum threshold the excursion actually travelled, capped once it's convincing; no extra credit for showing off.

Each factor is capped so that two signals which both plainly describe the movement come out equal, instead of separated by a rounding error. Ties fall back to declaration order (hips before wrists, distances before angles) rather than to floating-point luck, because "which signal won" should not depend on the weather inside an FPU.

Two smaller fixes that mattered as much, which is always how it goes

Percentiles, not min and max. Thresholds were originally the raw extremes of the signal, which makes the entire detector hostage to its single worst frame: one limb flip, one moment of the tracker falling in love with a passer-by, and the range stretches, the hysteresis bands drift past where any real rep reached, and a clip full of reps returns zero. Thresholds now come from the 5th and 95th percentiles. Costs nothing. A handful of bad frames cannot move it. I have no idea why we ever did it the other way, except that everyone does, once.

Hysteresis, not local minima. A lifter who pauses at the bottom, or bounces, writes two local minima where one rep lives. Hysteresis, demanding the signal cross a high band and then a low band rather than merely turn around, counts it once. Crossings are read from a median-despiked copy; the turnaround is then located in the raw signal. Smoothing gets to decide whether there was a rep. It does not get to decide when. Know your place.

The general lesson, such as it is

The bug was never in the geometry. The bug was in the objective.

"Most reps wins" is a proxy that looks perfectly aligned with the goal and is in fact anti-correlated with it, because the system's failure mode produces more of the maximised quantity than its success does. Score the property you actually want (this looks like a person repeating a movement) and the per-exercise table stays unwritten, and a new lift stays a zero-line change, and you get to go home. Metaphorically.

Related

The pose pipeline underneath all this is in quantize the lifter, not the detector, which contains its own perfectly reasonable assumption dying in benchmarks.

Form Coach ships in OneRep; a free account runs it about five times a month before the allowance runs dry. The tracking app underneath (workouts, food, progress) is free and unlimited.

Compare: OneRep vs Strong