Things Developers Believe About Video Files (Proven Wrong by User Uploads) - Derek Buitenhuis

8 min read Original article ↗

Transcript

  2. /whois derekb 1 5 October 2017 • Senior Video Encoding

    Engineer at Vimeo • Distributed chunk transcoder • On-the-fly image service (before it was cool) • HLS/DASH on-the-fly packager • Open Source Multimedia Troll^WDeveloper • FFmpeg, L-SMASH, FFMS2, x264, etc. • Daemon404 @ FreeNode • VideoLAN Non-Profit Association Member

  3. Everything’s Great Until Users Show Up 2 5 October 2017

  6. Everything’s Great Until Users Show Up 2 5 October 2017

    • Most users are well behaved (H.264 + AAC in MOV/MP4) • Others… • In the last 90 or so days: • 119 Video Codecs • 73 Audio Codecs • 39 Container • N Combinations of the above • Observation: Users with the worst files tend to complain the loudest

  7. Everything’s Great Until Users Show Up 2 5 October 2017

    • Most users are well behaved (H.264 + AAC in MOV/MP4) • Others… • In the last 90 or so days: • 119 Video Codecs • 73 Audio Codecs • 39 Container • N Combinations of the above • Observation: Users with the worst files tend to complain the loudest • Fixing the worst files has lead to a much more robust all around product

  8. The “Obvious” Ones 3 5 October 2017

  9. The “Obvious” Ones 3 5 October 2017

  10. The “Obvious” Ones 3 5 October 2017

  12. Things Change… 3.5 5 October 2017 • Pixel packing format

    • Color family • Color primaries • Transfer function • Color matrix • Resolution • Frame rate (more on this later!) • Color range • Codec • Sampling rate • Aspect ratio • Bit depth • Whether it’s interlaced or not • Whether it’s telecined or not • Field order • Chroma location • Number of audio channels • Channel layout • Audio sample format • About 10 billion more!

  15. The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 •

    That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?)

  16. The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 •

    That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?) • That even when you do have a constant frame rate, that you play at that rate • mvhd rate and edts media_rate in ISOBMFF / MOV

  17. The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 •

    That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?) • That even when you do have a constant frame rate, that you play at that rate • mvhd rate and edts media_rate in ISOBMFF / MOV • That you don’t need to do rate conversion • Example: sub-1fps slideshow videos → bad seeking, bad subtitles

  18. The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 •

    That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?) • That even when you do have a constant frame rate, that you play at that rate • mvhd rate and edts media_rate in ISOBMFF / MOV • That you don’t need to do rate conversion • Example: sub-1fps slideshow videos → bad seeking, bad subtitles • That every frame will have a timestamp

  19. The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 •

    That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?) • That even when you do have a constant frame rate, that you play at that rate • mvhd rate and edts media_rate in ISOBMFF / MOV • That you don’t need to do rate conversion • Example: sub-1fps slideshow videos → bad seeking, bad subtitles • That every frame will have a timestamp • That discontinuous timestamps are invalid • OGG, MPEG-TS

  22. 4 5 October 2017 • That you shouldn’t display negative

    timestamped samples • AllF entry in udta atom • That you don’t need sample durations • Duration of the last frame + sub-1fps slideshows • That you’ll have sample durations • Looking at you, Matroska Yes, Timestamps Are Pain, Have Some More

  23. 4 5 October 2017 • That you shouldn’t display negative

    timestamped samples • AllF entry in udta atom • That you don’t need sample durations • Duration of the last frame + sub-1fps slideshows • That you’ll have sample durations • Looking at you, Matroska • That these will match the timestamp deltas Yes, Timestamps Are Pain, Have Some More

  26. Less Obvious: That Was a Fun Day of Debugging 5

    5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh • That coded samples match what should be displayed • Edit lists, ordered chapters

  27. Less Obvious: That Was a Fun Day of Debugging 5

    5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh • That coded samples match what should be displayed • Edit lists, ordered chapters • That you only need to display what’s in the file • Audio silence gap-filling, post-stream black frames

  28. Less Obvious: That Was a Fun Day of Debugging 5

    5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh • That coded samples match what should be displayed • Edit lists, ordered chapters • That you only need to display what’s in the file • Audio silence gap-filling, post-stream black frames • That you should display every packet/sample • VP8, VP9, AV1

  29. Less Obvious: That Was a Fun Day of Debugging 5

    5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh • That coded samples match what should be displayed • Edit lists, ordered chapters • That you only need to display what’s in the file • Audio silence gap-filling, post-stream black frames • That you should display every packet/sample • VP8, VP9, AV1 • That you only need to play one track of a kind at once • Multiple default-on MOV tracks

  33. Less Obvious: That Was a Fun Week of Debugging 6

    5 October 2017 • That packets will be well interleaved • That packets will even be interleaved at all • That packets will even be vaguely in-order • That you should use the bitstream color properties • That you should use the container color properties

  34. Less Obvious: That Was a Fun Week of Debugging 6

    5 October 2017 • That packets will be well interleaved • That packets will even be interleaved at all • That packets will even be vaguely in-order • That you should use the bitstream color properties • That you should use the container color properties • That you shouldn’t use both

  35. Less Obvious: That Was a Fun Week of Debugging 6

    5 October 2017 • That packets will be well interleaved • That packets will even be interleaved at all • That packets will even be vaguely in-order • That you should use the bitstream color properties • That you should use the container color properties • That you shouldn’t use both • That the color properties will even map to some existing standard • MOV’s embedded ICC profile in the colr atom

  38. 7 5 October 2017 • That file actually has an

    end • Segment linking or dref atom loop (A → B → A → B → …) • LOOP entry in udta atom • That the video track is even a video “codec” • PowerPoint-in-MOV, FIRE, VNC commands, RAR in MKV “multimedia is basically neverending pain” — wm4, mpv-player lead

  39. 7 5 October 2017 • That file actually has an

    end • Segment linking or dref atom loop (A → B → A → B → …) • LOOP entry in udta atom • That the video track is even a video “codec” • PowerPoint-in-MOV, FIRE, VNC commands, RAR in MKV • That you can easily downmix or play a stereo track on a mono speaker • Phase cancellation is great fun “multimedia is basically neverending pain” — wm4, mpv-player lead

  42. Tools, Sources, etc. 8 5 October 2017 • ffprobe: http://git.videolan.org/?p=ffmpeg.git

    • boxdumper: https://github.com/l-smash/l-smash • bxd: https://github.com/dwbuiten/bxd • Beyond Compare is great as well • ffms2: https://github.com/FFMS/ffms2/ • dvbsnoop: http://dvbsnoop.sourceforge.net/ • JPEG/MP3 sample epic bug report: • https://github.com/mpv-player/mpv/issues/3973 • Source of IDCT Images: • https://guru.multimedia.cx/the-mpeg124-and-h26123-idct

  43. 9 5 October 2017 Thanks! Questions (or )?