Hello, Recap360

What was involved in playing the native 360 video files? I didn’t expect it to be easy, but it was harder than I expected.

I wouldn’t have bothered, but I really needed something for quick playback & review and grew annoyed every time the native players failed me. (I record each and every bicycle ride I take, whether it’s to the grocery store or a 50 mile trip to Fairfax, and review each ride afterwards. At 70GB+ per hour, that adds up to a lot of video.)

What lives inside a .360 file

A GoPro .360 is an MP4 container with two video tracks (front/back lens), one or two audio tracks, and a GPMF metadata stream that carries the gyro samples we need for horizon lock. Equirectangular it is not — GoPro uses EAC (equiangular cubemap), which means you can’t naïvely feed it into a fisheye shader.

GoPro was kind enough to document their metadata thoroughly on Github. They describe what the data is, but fall a little short on how to tie it all together.

moov
├── trak (video, lens A — H.265, EAC)
├── trak (video, lens B — H.265, EAC)
├── trak (audio)
└── trak (meta, GPMF)
    └── ACCL, GYRO, GPS5, ...

The DJI Osmo 360 is friendlier in one way (single dual-fisheye track) and weirder in another (the IMU samples are a separate, undocumented data tracks). More on that in a future post.

What’s next

Over the next few posts we’ll cover:

• How we decode dual-track .360 on a single AVSampleBufferDisplayLayer
• The math that turns gyro quaternions into a stable camera orientation
• Why AV1 made everything both better and harder

If you’ve poked at this stuff before, we’d love to compare notes.

Thanks to those who have plummed these depths