Incorrect Grayscale Conversion

[Queuecumber]

What did you do?

I'm having a weird inconsistency in the way Pillow is treating grayscale images, and the difference is enough to throw off some metrics I need to compute. After a lot of debugging I was able to trace the inconsistency back to the grayscale conversion. I have included a comparison with OpenCV which I have verified to be the correct conversion. By correct conversion I mean one that is consistent with other programs I have tried (they all agree with OpenCV)

It is very hard to detect this difference by just looking at the grayscale ouputs but I am including them for completeness. The difference becomes extremely apparent under aggressive JPEG compression.

Pillow Output (verified wrong)

OpenCV Output (verified correct)

Difference

Here is the code used to compute these images:

im = np.asarray(Image.open(args.input).convert('L'))
im2 = cv2.cvtColor(cv2.imread(args.input), cv2.COLOR_BGR2GRAY)

diff = im - im2

cv2.imwrite('pillow_output.png', im)
cv2.imwrite('opencv_output.png', im2)
cv2.imwrite('diff.png', diff)

Things to Note

• This difference is only apparent when I do a grayscale conversion. If I leave the images as color images they are identical
• OpenCV loads the images as BGR so line 2 is not a typo
• Maybe this is related to a difference in the chosen luma transform?

What are your OS, Python and Pillow versions?

• OS: Various Linux
• Python: 3.5 and 3.6
• Pillow: 6.0.0

[Queuecumber]

Going off of https://docs.opencv.org/3.1.0/de/d25/imgproc_color_conversions.html OpenCV appears to be using the same grayscale conversion that pillow is so I really have no idea why this is happening

[Queuecumber]

And based on https://github.com/cloudflare/jpegtran/blob/master/jccolor.c#L48 libjpeg does the same

[radarhere]

Could you attach the source image that you are passing into your code?

[Queuecumber]

Sorry for the delayed reply.

First off here is more complete code that you can run to reproduce, I realized I only posted a snippet before. This thing you can run and give the image as the sole argument and it should reproduce the images I attached to the original post

from PIL import Image
import cv2
from argparse import ArgumentParser
import numpy as np

parser = ArgumentParser()
parser.add_argument('input')
args = parser.parse_args()

im = np.asarray(Image.open(args.input).convert('L'))
im2 = cv2.cvtColor(cv2.imread(args.input), cv2.COLOR_BGR2GRAY)

diff = im - im2

cv2.imwrite('pillow_output.png', im)
cv2.imwrite('opencv_output.png', im2)
cv2.imwrite('diff.png', diff)

Next, here is the image I am using. I had to convert it to PNG for github, which is probably not a problem but it case it is, it is parrots.bmp from the live1 image quality assessment database (https://live.ece.utexas.edu/research/quality/subjective.htm).

[radarhere]

I find that changing 'L' to 'I' works.

from PIL import Image
import cv2
from argparse import ArgumentParser
import numpy as np

parser = ArgumentParser()
parser.add_argument('input')
args = parser.parse_args()

im = np.asarray(Image.open(args.input).convert('I'))
im2 = cv2.cvtColor(cv2.imread(args.input), cv2.COLOR_BGR2GRAY)

diff = im - im2

cv2.imwrite('pillow_output.png', im)
cv2.imwrite('opencv_output.png', im2)
cv2.imwrite('diff.png', diff)

[Queuecumber]

'I' is supposed to be signed integer though. Maybe it's because of an increase in precision?

I'm also noticing slight discrepancies with GIMP and mogrify as well so this might not be a major issue. I have a feeling there is some overflow that is making the difference image look worse than it is

[radarhere]

It also works with F.

[Queuecumber]

I think maybe "works" needs better definition. The resulting image appears black, sure, but here's an updated script that also prints the RMSE and the difference image to console after making sure the types have maximum precision (float64).

from PIL import Image
import cv2
from argparse import ArgumentParser
import numpy as np

parser = ArgumentParser()
parser.add_argument('input')
args = parser.parse_args()

im = np.asarray(Image.open(args.input).convert('L')).astype(np.float64)
im2 = cv2.cvtColor(cv2.imread(args.input), cv2.COLOR_BGR2GRAY).astype(np.float64)

diff = im - im2

print(diff)

rmse = np.sqrt((diff**2).mean())

print(rmse)

cv2.imwrite('pillow_output.png', im)
cv2.imwrite('opencv_output.png', im2)
cv2.imwrite('diff.png', diff)

If you play with different values of the .convert argument, you'll see that none of them have zero RMSE.

Also note the values of the difference image, the seem to be either 0 or -1, 0 is good obviously, -1 would explain the bright white spots in my original difference image (underflowing to 255).

This definitely looks like a precision issue, so I guess the question is do you care about some pixels being off by a single gray level? Does uint8 make things worse since it is susceptible to underflow?

[btxgit]

Sorry to post what may be a different grayscale conversion problem than is being experienced here, but I felt like my answer could solve some minor differences between Pillow and other image libs.

Specifically, when using convert('L'), which uses the traditional 299r + 587g + 114b / 1000 technique, the value is divided by 1000 using what I believe is the wrong divide operator. By using the // operator, a rounding error is introduced. I only noticed this because of single bit errors I was seeing in perceptual hashes I made with Pillow and a go-based library. Normally, it seems like this rounding error would be insignificant, but in my case it's throwing things off but perhaps a few problems combining?

[radarhere]

The rounding problem from @btxgit should now be fixed, thanks to #4320. This should also improve the differences mentioned in the original post. Regarding removing all differences in the operations between OpenCV and Pillow, see #4320 (comment)