How to Run Your Own Codec Bake-Off: A DTEA Methodology Guide

Written by

DTEA.ORG

in

If you’re working in the streaming world, you’ve probably seen the marketing slides. AV1 claims 30% better efficiency than HEVC. VVC promises even more. But when your boss asks, “Should we switch next quarter?” you can’t just point to a vendor’s PDF. You need your own data.

Running a "codec bake-off" is the standard way to prove which technology actually works for your specific content, your audience, and your budget. But if your methodology is flawed, your results are worthless. A slight mismatch in color space or an unfair encoder preset can lead to millions in wasted egress costs or a degraded user experience.

At the Data Transmission Efficiency Alliance (DTEA), we’re building the industry’s first independent certification system to stop the guesswork. Until your favorite codec has a DTEA seal, here is our professional guide on how to run a rigorous, fair, and actionable codec benchmark in-house.

Step 1: Curate Your "Mezzanine" Source Files

The biggest mistake engineers make is testing with a single movie trailer or a random 1080p clip. Codecs react differently to different types of content. A codec that handles a talking-head interview perfectly might fall apart during a high-motion football game or a dark, grainy horror film.

To get a real answer, you need a diverse library of "mezzanine" files (high-bitrate, near-lossless sources). Your test set should include:

  • High Motion: Sports or action sequences to test temporal compression.
  • Fine Texture: Grass, water, or film grain.
  • Graphic Content: Animation or screen recordings (gaming) with sharp edges.
  • Low Light: To see how the codec handles noise and banding in dark gradients.
  • Static Content: News anchors or interviews to test "easy" efficiency.

Aim for at least 10–15 clips, each roughly 10 to 30 seconds long. Ensure they are all in the same format: usually YUV 4:2:0 10-bit is the modern standard for HDR-ready workflows.

A collage of diverse video source types: a football player running, a close-up of a face, an animated character, and a dark forest scene, arranged in a grid pattern

Step 2: Define the Test Matrix (The Bitrate Ladder)

You aren't just testing if AV1 looks better than HEVC at 2 Mbps. You are testing their Rate-Distortion (RD) curves. This means you need to encode every clip at multiple bitrates to see where the "break-even" point is.

A typical test matrix looks like this:

  • Resolutions: 1080p, 4K (and maybe 720p for low-bandwidth scenarios).
  • Bitrate Points: Pick 4 to 6 points per resolution. For 1080p, you might test 500kbps, 1Mbps, 2Mbps, 4Mbps, and 8Mbps.
  • The Comparison: If you are testing AV1 vs. HEVC, you must run both codecs through the exact same bitrate points.

Pro Tip: Don't just use Constant Bitrate (CBR). For a bake-off, 2-pass Variable Bitrate (VBR) or Constant Quality (CRF) is often better because it allows the codec to be as efficient as possible.

Step 3: Choose Your Tools and Presets

This is where most benchmarks go off the rails. It’s easy to make a codec look bad by using a "Fast" preset while giving its competitor the "Slower" preset.

To keep it fair, you need to align the "effort" the CPU is putting in.

  • HEVC: Use x265.
  • AV1: Use SVT-AV1 (the industry standard for production) or libaom.
  • VVC: Use VTM (reference) or vvenc (production).

You must decide if you are testing for VOD (where you can afford slow, high-quality encodes) or Live (where you have a strict millisecond budget). If you're testing for VOD, use the "slow" or "veryslow" presets across the board. If you’re testing for Live, use a preset that can actually run in real-time on your hardware.

Step 4: Measuring Quality (Beyond the "Eye Test")

You can't watch 500 videos and give them a score. You need objective metrics. At DTEA, we recommend a "triangulation" approach using three key metrics:

1. VMAF (Video Multi-Method Assessment Fusion)

Developed by Netflix, VMAF is currently the king of metrics. It uses machine learning to predict how a human would rate the video. A VMAF score of 100 is identical to the source; 90-95 is considered "excellent."

  • Why use it: It’s the most accurate for streaming.

2. PSNR (Peak Signal-to-Noise Ratio)

This is the "old school" math-based approach. It measures the physical difference between pixels.

  • Why use it: It’s great for technical debugging. If VMAF looks good but PSNR is tanking, you might have a weird artifact that the ML model missed.

3. SSIM (Structural Similarity Index)

SSIM looks at textures and patterns rather than just raw pixels.

  • Why use it: It’s a good "tie-breaker" when VMAF and PSNR don't agree.

A stylized 3D graph showing two lines (RD curves) on an X-Y axis, representing Bitrate vs. Quality, with data points labeled VMAF and PSNR

Step 5: The Math of the Winner (BD-Rate)

Once you have your data points, you don't just look at the average. You calculate the BD-Rate (Bjontegaard Delta Rate).

BD-Rate tells you exactly how much bitrate you save for the exact same quality. For example, if your bake-off shows a BD-Rate of -25% for AV1 vs. HEVC, it means AV1 provides the same VMAF score while using 25% less data.

Wait! Don't forget Complexity.
A codec that saves 30% in bandwidth but costs 10x more in server electricity might not be a win. You must track:

  • Encoding Time: How long did it take to crush the file?
  • Decoding Complexity: Can a 3-year-old budget smartphone play it back without melting the battery?

Common Pitfalls to Avoid

  • The "Cherry-Pick" Trap: Don't just test the one scene where AV1 wins. Test the scenes where it struggles.
  • Color Space Mismatches: Ensure your measurement tool isn't converting your 10-bit HDR file to 8-bit SDR before calculating the score. This happens more often than you’d think.
  • Ignoring the Header: In very low bitrate tests (like 200kbps), the "container overhead" (the data used just to say "this is a video file") can skew results. Make sure you’re measuring the total bits delivered, not just the raw video stream.

Why DTEA Certification Matters

Doing this yourself is hard. It takes weeks of compute time and hundreds of engineering hours to do it right. Even then, your hardware vendor might disagree with your results.

That’s why the Data Transmission Efficiency Alliance exists. We are establishing the first independent, transparent certification system for these technologies. When a codec or a hardware encoder carries a DTEA certification, you don't have to run a 3-week bake-off. You’ll already know exactly how it performs against industry benchmarks.

We believe that data transmission should be efficient, and efficiency should be verifiable. Whether you are AWS, Netflix, or a growing startup, standardized metrics save everyone money and reduce the global carbon footprint of data centers.

A professional silver and blue holographic emblem or seal with the text 'DTEA CERTIFIED' and 'Efficiency Benchmark Standard', clean corporate design

Conclusion: Stop Guessing, Start Measuring

Switching codecs is a massive infrastructure decision. It affects your storage, your compute, and your CDN egress bills. By running a methodical bake-off: using diverse sources, aligned presets, and the "big three" metrics (VMAF, PSNR, SSIM): you can make a data-driven choice that stands up to scrutiny.

Ready to see how the industry's top encoders stack up? Check out our latest performance benchmarks at DTEA.org and join the alliance to help set the standards for the future of streaming.

More posts