HighlightHub
Finding the highlight in hours of footage, automatically
A machine learning system that detects highlight-worthy moments in long-form video by reading audio and temporal signals, so creators stop scrubbing timelines by hand.
The problem
A streamer finishes an eight-hour broadcast. Somewhere inside it are perhaps twelve moments worth clipping. Finding them means watching the whole thing again, which is why most creators either pay an editor or simply never post short-form content at all.
The signal is there. Crowd noise spikes, the streamer’s voice jumps an octave, chat accelerates, the game state changes sharply. Humans read these cues instantly. The question was whether a model could.
The approach
HighlightHub treats a highlight as a multi-signal event rather than a visual one. Audio carries most of the information: amplitude envelopes, spectral shifts, speech energy. Temporal attributes carry the rest: how quickly the scene is changing, how that compares to the session’s own baseline.
Two design decisions did the heavy lifting:
Score relative to the session, not to a global threshold. A quiet commentator and a loud one produce completely different absolute values. Normalizing each stream against itself is what made the model transfer between creators at all.
Rank rather than classify. “Is this a highlight?” is a question with no stable answer. “Which twenty seconds of this hour are most likely to be worth clipping?” is answerable, and it is the question a creator actually has.
Where it stands
An ongoing build since late 2024, and the project that taught me the most about the distance between a model that works on a benchmark and a system someone will use every day. That gap is almost never the model.