Home    Search    Gallery    How-To    Books    Links    Workshops    About    Contact

adorama

Fallacies of Noise Measurements
© 2006 KenRockwell.com

Please help KenRockwell..com

I get my goodies at Amazon and Adorama.
It helps me publish this site when you get yours from those links, too.

Introduction

Today no one publishes meaningful noise measurements of digital cameras.

Many of the measurements which beginners consider so important aren't even measured properly! Personally my real job for decades was in the design and measurement of digital imaging systems both for photography and television. Hollywood TV production has been digital since the 1980s and we know an awful lot about how to measure things. By comparison, many photo magazines and commercial websites are just making it up as they go along. I used to pull in over a hundred grand a year in my day job and magazines just can't afford people with this level of expertise. Luckily for you I do this site for free!

Our eyes work in more dimensions than simply brightness and x-y location. Our eyes are also sensitive to spatial frequency, chromatic, environmental and temporal issues.

To create numerical measurements that agree with perceived image quality requires complex weightings to incorporate these other dimensions. No one does this for testing digital cameras today.

Failure to incorporate all these factors create measurements that don't correlate well with perception, which in simple English means that the numbers won't agree with how good or bad a camera looks. A camera with better noise performance might have a higher measured number and vice versa.

If you're comparing relatively identical cameras from the same maker of the same model year you're probably OK comparing the numbers you see. Figures are much less likely to provide a valid comparison when comparing different makes and models and vintages. Therefore it's misleading to compare published figures for Nikons versus Canons, for instance.

People with Ph.D.s are still researching the response of the human vision system, and machines that measure image fidelity like the Tektronix PQA300 still cost $64,000. These $64,000 instruments only measure at resolutions of 720 x 483 pixels. I used to work at Tektronix with these. We had a heck of a time selling them to major TV networks and equipment manufacturers, mostly because they don't tell you anything more than your eyes do. The reason people would pay for these is because they would give the same readings from one month to the next and it saved the designers from going blind rating images all day. Websites and photo magazines can't afford these, even if they did measure digital still camera images.

Modern cameras use clever noise reduction which blurs the image in clever ways to hide noise. These systems are often clever enough to keep edges sharp. Unfortunately they blur soft details and textures along with the noise. We can't measure noise well today even if it was uniform across the frame. Today's better cameras alter the noise from pixel to pixel.

You can change the noise readings on the Nikon D200 because the D200 provides four different levels of selectable noise reduction! There's no way to measure this today.

The only way to measure noise and the effects of noise reduction are to look at the image and judge for yourself. Sorry.

An Example in Popular Photography

I love Popular Photography Magazine and have been a subscriber since the 1970s. Unfortunately, like most people with less than a decade of Ph.D. level digital experience under their belts, they still make some common mistakes. On page 101 of Pop Photo's December, 2005 issue they explain in impressive detail that they measure digital camera noise by analyzing a section of flat gray. They then use Photoshop's histogram and average the standard deviations of Y, R, G and B. Who can argue with that? Anyone actually familiar with engineering level, not technician level, image noise measurements can.

First, eyes are sensitive to colors differently. Imaging professionals have known since the CIE tests of 1931 that to add up to 100% intensity we're 60% sensitive to green, 29% sensitive to red, and only 11% sensitive to blue. That's how we respond to intensity versus color. To a laboratory meter, or PhotoShop, each would be 33%. Think for yourself: when you see digital camera noise isn't it usually green-magenta blobs? Duh, that's because we're most sensitive to green. Therefore when professionals measure noise we weight it 60% green, 29% red and 11% blue. Since Pop Photo is making the mistake of weighing all these equally they could save themselves time by just taking the Y channel, which in photoshop is simply equal portions of R, G and B. To do this properly one could use PhotoShop's Channel Mixer to weight the channels properly.

Ideally noise is measured at RMS, not standard deviation. You can't do that trivially in PhotoShop.

Our eyes are also differently sensitive to different levels of detail. Noise at very high frequencies (pixel-by-pixel) disappears at most magnifications, and noise at low frequencies (big blotches) are far more serious. Pop Photo's measurements can't tell the difference. To measure noise professionally we weight it by frequency. You can't do this trivially in Photoshop, although you certainly could run some filters to do this and then measure the noise. You also need different filters for each color since we see details differently in different colors. You'd have to do this in Lab color space since it better approximates the axes on which our eyes are best simulated.

You probably could write an action for Photoshop to measure noise properly, and someone probably has. The reasons people who know better don't bother is because measuring noise on a flat section of an image is meaningless precisely because the noise reduction (NR) software in many cameras, regardless of quality, specifically will have its greatest effect on flat areas where there is no detail to preserve! At best these measurements of noise are just telling us how far the NR is cranked inside competing cameras!

I'm not dissing on Pop Photo. I'm illustrating just one of the many reasons you shouldn't waste time on potentially misleading figures published anywhere.

The best way to measure noise is to go make some photos and look for yourself! Look carefully not just at the noise at high ISOs, but look to see that the image hasn't been smoothed over by the camera's attempt at noise reduction (NR). The NR inside cameras tends to be worse than the NR one can get in PhotoShop with good plug ins like Grain Surgery. Grain Surgery is smart enough not to smear details and reduce noise at the same time, but some cameras, like the Casio EX-S500 seem just to make everything blurry at high ISOs in an attempt to smooth over the noise.

You can't make meaningful measurements of digital camera noise, so as an artist and a pro with an engineering degree and decades of digital image processing and measurement experience I have to laugh at methods used to report on digital cameras. Even worse are the well meaning beginners who place more emphasis on looking at a test number than looking at photos!

It is possible to design a system to measure this, but no one has.

Read more at Static versus Dynamic Image Quality

Home     Gallery     How-to     Links    Workshops     About     Contact