I Built a Video Player Out of Sprite Sheets

Why Not Just Use the Video Player?

Unity has a VideoPlayer component. It plays videos. That's its entire job. But I didn't think it could give me the control I wanted — frame-accurate scrubbing on the Timeline, tight integration with the rest of the scene, reliable WebGL playback without browser quirks. So instead of testing that assumption, I decided to build my own.

In hindsight, maybe I should have just tried the VideoPlayer. Or bought a plugin. But here we are.

How It Works

I extract every frame from an mp4 using FFmpeg at 30fps. Then I pack those frames into 2048x2048 sprite sheets — each sheet holds a grid of frames (2 columns, 3 rows, 6 frames per sheet at 1024x576 per cell). The packer calculates the optimal grid layout to maximize frames per sheet while keeping each cell at least 75% of the source resolution.

Then I chunk those sheets — groups of 64 frames each get their own Addressables label. At runtime, a custom Timeline track drives a SpriteFrameRenderer. The mixer calculates which frame to show based on the playback time, loads the right chunk via Addressables, swaps the sheet texture using a MaterialPropertyBlock (zero allocation per frame), and shifts the GameObject's position so the correct cell is centered on camera.

It's a sliding window. On WebGL it keeps a max of 2 chunks in memory at once because WASM linear memory can only grow, never shrink — if you spike to 200MB during a video, you're stuck at 200MB for the rest of the session. So the mixer aggressively evicts chunks behind the playhead and preloads 1 chunk ahead.

The Size Problem

Here's the part that keeps me up at night. A 5MB mp4 file (Death1, a 5-second death animation) turns into ~20MB of Addressable bundles. A 50MB mp4 (Video2, a longer cutscene) becomes 357MB of bundles across 44 parts. The entire game's remote content directory is 176 bundles totaling 1.4GB. And there's barely any actual game content yet.

The initial build is only ~15MB (compressed wasm + data). That's fine. But the moment you hit play and the first video loads, you're downloading sprite sheet bundles from the server and caching them in browser cache. Every frame that was a few KB as a compressed video frame is now a chunk of a DXT1-compressed 2048x2048 texture. The math is brutal.

Is This Even the Right Approach?

I genuinely don't know. I built a custom FFmpeg frame extractor tool, a sprite sheet packer, an auto-chunker, an Addressables label assigner, a custom Timeline track, a custom mixer with a sliding window chunk loader, a playback controller that kicks the PlayableDirector graph to force initialization — all so I could play a video at 30fps.

There are plugins on the Asset Store that probably do this. The Unity VideoPlayer might have worked fine with some tweaking. I might be doing everything wrong. But it works. The playback is smooth, the chunks stream in, the memory stays under control on mobile browsers, and Timeline scrubbing works in the editor.

My game is over a gig and there's like 4 levels. That feels wrong. But I shipped v90 and the videos play at 30fps and that's something.