Video Formats, Codecs & Container Architecture

Technical Deep Dive & Specifications

Comparison Matrix: Containers and Codecs

RFC 6381 Codecs String Syntax

When a browser evaluates multiple <source> elements, it inspects the type attribute. If you specify only the container MIME type (e.g., type="video/mp4"), the browser may need to start downloading bytes just to discover that it cannot decode the internal codec.

By providing the RFC 6381 codecs parameter, the browser can make an instantaneous decision without touching the network:

<!-- AV1 Video (Next-Gen High Compression) with Opus Audio in WebM -->
<source src="clip-av1.webm" type='video/webm; codecs="av01.0.05M.08, opus"'>

<!-- VP9 Video with Opus Audio in WebM -->
<source src="clip-vp9.webm" type='video/webm; codecs="vp9, opus"'>

<!-- H.264 Video (Constrained Baseline Profile) with AAC Audio in MP4 -->
<source src="clip-h264.mp4" type='video/mp4; codecs="avc1.42E01E, mp4a.40.2"'>

Decoding the H.264 avc1 String:

• avc1: Advanced Video Coding (H.264).
• 42: Profile indicator (Hex 0x42 = 66, Baseline Profile).
• E0: Constraint flags (compatibility bits).
• 1E: Level indicator (Hex 0x1E = Level 3.0, defining max bitrate and frame resolution).

Decoding the AV1 av01 String:

• av01: AOMedia Video 1.
• 0: Main Profile.
• 05M: Level 5.0, Main Tier (capable of 4K 60fps).
• 08: 8-bit color depth.

The MP4 Box Structure & The moov Atom Optimization

An MP4 container is organized as a hierarchical tree of binary blocks called atoms (or "boxes"):

+---------------------------------------------------------------------------------------+
|                       DEFAULT ENCODING (BROKEN STREAMING)                             |
|  +--------------+  +--------------------------------------------+  +---------------+  |
|  |  ftyp Atom   |  |            mdat Atom                       |  |   moov Atom   |  |
|  | (File Type)  |  | (Gigabytes of Raw Video & Audio Frames)    |  | (Index Table) |  |
|  +--------------+  +--------------------------------------------+  +---------------+  |
|         ^                                                                  ^          |
|    Byte Offset 0                                                      End of File     |
+---------------------------------------------------------------------------------------+

1. ftyp (File Type Box): Identifies container brand and version compatibility.
2. mdat (Media Data Box): Contains the raw compressed video and audio sample frames (99% of file size).
3. moov (Movie Header Box): The index catalog containing stream durations, frame rates, sample sizes, and byte offset pointers needed to decode frames.

The Problem:

Standard video encoders (like default FFmpeg or Adobe Premiere) write the moov atom at the very end of the file because the total duration and frame table offsets are only known after the entire video is encoded. When a browser streams this MP4, it cannot decode a single frame until it downloads the entire file from start to finish to reach the moov atom at byte offset end-of-file!

The Solution: FFmpeg +faststart

+---------------------------------------------------------------------------------------+
|                    OPTIMIZED MP4 WITH FASTSTART (INSTANT STREAMING)                   |
|  +--------------+  +---------------+  +--------------------------------------------+  |
|  |  ftyp Atom   |  |   moov Atom   |  |            mdat Atom                       |  |
|  | (File Type)  |  | (Index Table) |  | (Raw Video & Audio Sample Data)            |  |
|  +--------------+  +---------------+  +--------------------------------------------+  |
|         ^                  ^                                                          |
|    Byte Offset 0      Read in First 16KB -> Instant Video Startup!                    |
+---------------------------------------------------------------------------------------+

Relocating the moov atom to the head of the file allows the browser to read index tables in the very first HTTP request chunk, enabling instant playback start and non-blocking seeking.

The FFmpeg Transcoding Recipe:

# Relocate moov atom to beginning of file without re-encoding video streams:
ffmpeg -i input.mp4 -c copy -movflags +faststart output_faststart.mp4

# Encode production-ready WebM VP9 with Opus audio:
ffmpeg -i input.mp4 -c:v libvpx-vp9 -b:v 1500k -c:a libopus -b:a 128k output.webm

# Encode next-gen AV1 video:
ffmpeg -i input.mp4 -c:v libsvtav1 -crf 30 -c:a libopus -b:a 128k output_av1.webm

💻 Interactive Code Playground

Starter Code

Line-by-Line Code Breakdown

• Lines 38–41 (<source ... type='video/mp4; codecs="avc1.64001F, mp4a.40.2"'>):
  • The browser reads the top <source> tag first.
  • It checks if its internal GPU/software decoder supports H.264 High Profile (avc1.64001F) and AAC audio (mp4a.40.2).
  • If supported, it locks onto this stream and halts all evaluation of subsequent <source> tags.
• Lines 44–46 (<source ... type='video/webm; codecs="vp8, vorbis"'>):
  • Secondary fallback for open-source engines prioritizing WebM containers.
• Line 57 (vid.currentSrc):
  • The DOM property video.currentSrc returns the exact absolute URL of the <source> tag that was selected by the browser's media engine.

Expected Browser Render Output

+-------------------------------------------------------------+
| Multi-Format Codec Cascade                                  |
| +---------------------------------------------------------+ |
| |                                                         | |
| |                    [ VIDEO CANVAS ]                     | |
| |                                                         | |
| | [ > ] [===o=========================] 0:00 / 0:15 [🔊][⛶]| |
| +---------------------------------------------------------+ |
| Resolved Media Source: https://commondatastorage...mp4      |
| Network State: 2 (Active Stream Connected)                  |
+-------------------------------------------------------------+