Perceptual video quality & compression - PSNR measurements

Compression is a fact of life, there have only been two production VTRs that stored uncompressed video - D1 and D5; they are no longer used because they were both SD (and D1 was only eight-bit video). So, the vast majority of the material we handle is compressed and so there should be a way of quantitatively judging it. There three methods of numerically analysing how good pictures are, but for the most part and engineer or editor proclaiming "...those pictures look a bit soft" is what still passes for picture quality analysis! 

Don't regard this blog-post as definitive (which of my rambling are?!), but this comes from a discussion with a couple of industry colleagues earlier in the week about the quality of satellite contribution circuits. They'd got into a bit of a to-and-fro with the carrier over bit-rates and chroma-sampling structures ("but 4:1:1 isn't as good as 4:2:0 for the same data rate" etc.) which for my money entirely misses the point. You have to assess the picture quality of a compressed link (satellite, IP, etc.) not on encoder settings but on perceived picture quality. A few thoughts;

1. Modern codecs perform better than older codecs when considering data rate vs quality
2. Progressive pictures compress nicer than interlaced pictures
3. Statistical multiplexing always produces better results over multiplexed connections
4. Long GOP codecs outperform iFrame codecs by a factor of 5:1 typically.

The three methods used for determining video quality are;

• Peak signal-to-noise ratio - PSNR
• The "Just Noticeable Difference" or JND; a very BBC-type measurement and commonly derived by asking a crowd of observers to assess picture quality.  I like the idea of this and when I was at the BBC you'd often hear people specifying "...half a JND" as being a required spec; that was also know as a "gnats" as in "gnat's whisker"! It's problematic because it describes perception rather than the effect of exposure - an editor friend told me that he preferred to work on DigiBeta pictures over DVCam footage not because he could spot the difference between shots from the more expensive format over the cheaper format (all other things - camera, lens, lighting etc being equal), but the more compressed pictures just made him feel more tired by the end of the day. The JND takes no account of the cumulative effect of looking at compressed footage that may at that moment look just as good but more subtly takes it toll on the viewer.
• Mean Opinion Score - MOS; very similar to the JND but with a numeric score. I won't talk about this.

So, PSNR - to steal from Wikipedia;

...the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation. Because many signals have a very wide dynamic range, PSNR is usually expressed in terms of the logarithmic decibel scale.

PSNR is calculated as a rolling set of differences between source material and the compressed version and is most easily defined via the mean squared error (MSE). Given a noise-free m×n monochrome image I and its noisy approximation K, MSE is defined as:

The point is that it can be calculated from the pixels. No observer bias is involved.
Engineers love quick rules of thumb, and PSNR for video images are no different;

• For idential images the MSE is zero and hence the PSNR is infinite (more dBs = better pictures!)
• 40dBs is considered to be indistinguishable from uncompressed production quality (that's where the BBC JND lives!)
• 32dBs is considered desirable for quality broadcast link circuits
• High twenties is what you can expect for over-the-air transmission - the 10Mbit DVB-T2 pictures you watch on Freeview or Sky.

 

So, to return to my point 1 (above) - here is a graph showing data rates against codec types for the same SD pictures. Venerable old MPEG2 (from the mid-90s!) up against vanilla MPEG4 (late nineties) and AVC (AKA H.264 / MPEG4.pt10 - early noughties). A full 6dBs of quality (twice as good in layman's terms?) lie between those two codecs at 2.2Mbit/sec (all other things being equal - use of a Stat Mux etc). You could even dive in further to MPEG2 and see the difference between the implementations from twenty years ago and what folks like Main Concepts are doing in 2014). The decoders particularly are now much better at hiding macro-block edges and recovering from corrupt frames.

Point 2 (above) seems obvious, but only when comparing 1080i with 1080p pictures AT THE SAME FRAMERATE; so perhaps best to say 1080i vs 1080PsF; Interlaced pictures will always be a challenge as pixels (and hence macro-blocks) move within a frame, unlike progressive pictures. BUT, you still get better motion rendition withing interlaced frames for the same framerate. Eventually we'll have moved to 1080 50/60P and so it'll be a moot point.

This graph shows data rate for HD pictures, we expect over-the-air HD to be at ten megabits in the UK.

So, how to make these assessments if you're worried about a contribution circuit or transmission path that you're responsible for? If you work in coding and mux then you probably already have tools to assess. PSNR is such an important part of delivery specs/SLAs in broadcast (you need to keep the accountants at bay after all!) that you'll have a Tektronix PQA600 or a Rohde & Schwarz DVMS-series test set. 

However, "traditional" video quality measurement needs access to the compressed signal and the original which may not be possible; particularly in the case of my pals who are at loggerheads with their satellite provider. What you need is a test signal that you can feed over the connection and then make an assessment from the picture content as to how badly the pictures are being degraded. I've banged on about the SRI Visualiser before but it has a compression multiburst that shows you lines-of-TV-resolution against perceived bit-depth. You can then relate the two worst-case lines/bit-depth figures to the table of PSNR values.

 

Forgive the voice-over!