Deviations associated with different image processing programs.

[ajax]

First, let me say that my problem may not be with GIMP but since I'm using and trying to learn GIMP this seemed like the best forum for discussion.  As a result, of some success using GIMP to edit images obtained by scanning old and often decaying photographs I decided to upgrade my camera to something capable of producing images in raw format.  I now have a Canon EOS Rebel T6 set to produce raw images.  I'm running GIMP on Win7 and have installed UFraw Version 0.19.2 in order open the raw image files.

The problem is that there is a huge difference in the picture being displayed depending on what software is being used for viewing it.  When I say view I mean that I have not attempted to do any editing (i.e., make any changes).  I don't know how to demonstrate the problem with raw images that you forum users may not be able to process.  Fortunately, the problem which at present is entirely about appearance is preserved when different software is used to prepare .jpg files for reference purposes.  I've uploaded those .jpg files to Google Photos in order to demonstrate the problem being described.  While Google Photos does perform conversion of it's own, such as scaling, the defects I'm describing are preserved in these images.  For example,

Here (https://goo.gl/photos/sAStoedrd6KEi31cA) is an image prepared by DPP4 (i.e., Canon Digital Photo Professional 4).  Based on the subject of the photograph this one is consistent with photographer's expectation.

Here (https://goo.gl/photos/o81YQcLsxK4XzYJ46)  is the exact same image prepared by GIMP.

When I say same image it means the same raw file converted to .jpg format by the respective software.  Insofar as the raw file has to be processed by UFraw in order for GIMP to obtain an image, it is quite reasonable to conclude that UFraw may have done something to it.  In fact, the process of opening the file in GIMP results in the invocation of UFraw which does display the image prior to forwarding it to GIMP.  The view displayed in UFraw does look like what is is displayed in GIMP and the resulting .jpg created by GIMP.  You might conclude it is obvious that UFraw changed it which in this case amounts to pretty dramatic alteration, especially given that I made no attempt to change anything.

However, that is not the end of story.  I also use Google Picasa for organizing photographs on my computer.  Interestingly, Picasa is able to handle the raw files.  In fact, it looks like Picasa knows how to process them even though Windows does not.  For example,

Here (https://goo.gl/photos/ex25PVZ52eWLF9Ki7)  is the exact same image prepared by Picasa.  Picasa is also the program that is launched when I open a raw file in Windows Explorer.  While, to me, it looks like the .jpg image prepared by Picasa is similar in appearance to the defective one prepared by GIMP they are distinguishable from each other.  However, both of these images appear to have the same defects when compared to what was expected from the actual subject of the photo.  The image prepared by DDP4 is consistent with expectation.  Interestingly when the Picasa viewer is launched it momentarily displays an image that looks much more like the one prepared by DPP4 but it momentarily reverts to the appearance of the referenced image prepared by Picasa (i.e., like casting a dark shadow over the entire image).

My apologies for the verbosity but the big problem here is the extent of the inconsistency in results.  Absent some kind of explanation for what is causing these variations how can one have any confidence in the accuracy of the results obtained from using any of this, or for that matter other, software for editing graphic images.  How can this phenomenon be reconciled?  I hope there is a simple explanation that amounts to my error (i.e., lack of knowledge).  Can anyone explain?

[Ofnuts]

The secret is, AFAIK there is no absolute accuracy (after all, you can still underexpose or overexpose your picture). Gimp/Picasa rendering is an image which on the average is pretty close to 40% gray, and this is what I get with pictures from my camera that I find correctly exposed. Your picture is 64% grays and is a bit washed out: light areas lack detail, like the stern of the sailboat and the clouds in the back.

DPP may be using proprietary information in the picture metadata, while UFRaw and Picasa don't. If you want an accurate rendering, then use a color chart and shoot it in at least one picture of the series. Then correct colors in UFRaw so that the chart is accurately rendered, save the settings, and apply them on the other pictures.

However, the purpose of raw images is to let you take advantage of the full dynamics of your sensor to extract the most data from the shot. Remember that Gimp is only 8-bit, while your camera sensor is 12 or 14 bits. The raw converter works in 16-bit (or even 32-bit), so there is very little color loss when you do general color/level/contrast processing in it, unlike Gimp where Color tools will induce rapid color loss. You have to make choices and decide woith of the 14-bits are going to be squeezed into the final 8 bits. This is work (IMHO, a good 10 minutes on the simpler cases...) and is a matter of "artistic" interpretation. If you just use the defaults there is absolutely no point in using raw, you can just as well use the JPEG from the camera. By the same token since you are supposed to spend so much time in the raw conversion process you can just as well save that result. And at that point, little difference between going to Gimp directly or switching applications and explicitly loading that file. And if you do so you are no longer limited to the more Gimp-friendly applications, there are plenty of free and powerful raw converters (for instance RawTherapee and Photozone...).

[ajax]

Thanks for reply.  When it comes to digital photography I'm definitely a novice.  Some of your feedback pertains to topics for which I have little or no knowledge.  However, I am grateful for the feedback and while I still don't completely comprehend what is at play here it does point toward some topics that I need to learn a little more about.

With that said, I have now a bit more information to contribute to the discussion.  First, is that the .jpg file produced by GIMP cannot be opened by DPP4.  DPP4 says it is not a valid .jpg file.  This is troubling because I had the idea that the one big advantage of .jpg is pretty reliable universal support.  On the other hand I cannot vouch for the robustness of DPP4.  Second when I use DPP4 to convert the raw file to a standard format, either the .jpg referenced above or an 8bit .tiff, that I can open in GIMP the appearance of all of these files/images, to my somewhat non-critical eye, is pretty much the same (i.e., as opposed to the pretty drastic deviations revealed in the samples referenced above).

Everything seems to suggest that the process of passing the image data through UFraw results in conversion that affects the appearance which at least in this case is pretty drastic.  Your point about the camera sensors being capable of more than 8 bits of color depth which I assume would be reflected in the raw format is new to me but does seem to be significant.  When combined with the knowledge that GIMP operates in 8bit mode only this means some kind of conversion is indeed necessary.  I think it safe to say that my experiment for bypassing UFraw left this job to DPP4.

A bit more research has also caused me to discover that there is something called an ICC profile that varies from one device to another.  It looks like, I haven't completed research as of yet, UFraw must be customized with appropriate profile information for the specific camera in question.  I did not do any such thing but rather expected that this is something that metadata could handle.  What I'm thinking now is that this need was satisfied by the DPP4 software which knows about my camera.  Furthermore, if this is not something that can be magically handled with metadata then Picasa has to deal with the same issue.  So it now looks like that while Picasa could display a picture from the raw file it does not really, genuinely, support raw files.  It tricks one into thinking so.

There is nothing special about the photo I used for an example.  It was taken pretty hastily by an admittedly novice photographer and the idea that the exposure is less than perfect is probable.  Similarly, my interest in raw files is not because I'm such a sophisticated photographer that I need them but rather that I'm curious about the technology associated with digital graphics.  They are doing what I wanted which is causing me to learn more about the subject.

Thanks again for that.

[Ofnuts]

Yes, there are such things as "color profiles". But as far as I know (I own two Canon DSLRs) the Canon pictures use the sRGB format which is the default. In practice, you can tell your camera to produce both CR2 and JPEG. If you do so, you can compare the JPEG from the camera with the JPEG produced by from the CR2 by various applications. There is also the matter of the indicated white balance (but in your sample pictures, I think the problem is more an optimization of the global exposure).

As to DPP4 not supporting JPG from GIMP: when you export the JPG in Gimp, un-tick the "progressive JPEG" option in the advanced settings. If that fixes the problem, it means that DPP4 is not supporting something that has been part of the JPEG standard for about 15 years. For their defense, DDP4 is meant to process what comes form the camera and nothing else.

[ajax]

(08-15-2017, 05:23 PM)Ofnuts Wrote: Yes, there are such things as "color profiles". But as far as I know (I own two Canon DSLRs) the Canon pictures use the sRGB format which is the default. In practice, you can tell your camera to produce both CR2 and JPEG. If you do so, you can compare the JPEG from the camera with the JPEG produced by from the CR2 by various applications. There is also the matter of the indicated white balance (but in your sample pictures, I think the problem is more an optimization of the global exposure).

As to DPP4 not supporting JPG from GIMP: when you export the JPG in Gimp, un-tick the "progressive JPEG" option in the advanced settings. If that fixes the problem, it means that DPP4 is not supporting something that has been part of the JPEG standard for about 15 years. For their defense, DDP4 is meant to process what comes form the camera and nothing else.

I did chase the profile matter today.  The UFraw website was not much help.  Broken links for just about all of the references.  However, I was able to find a single .icm file within the DPP4 directories and this appears to be exactly what you describe.  It was called "sRGB Color Space Profile (sRGB v1.31 (Canon) - Canon, Inc.)".  I was able to add that to UFraw and it had a very minor/subtle effect on the appearance but enough to suggest that it was being used.  This suggested to me that my issue is caused by something else.

Another experiment involved using UFraw to see if I could manipulate the appearance to resemble what DPP4 was producing without doing anything.  By using the exposure control only I can get very close if not right on.  Using only my eyes to judge limits the extent of the match I would claim to have but since it is something conspicuous to my eyes which triggered this discussion I'd say we're onto something.  Interestingly, when I open the raw file with UFraw the exposure control shows -.29.  When I select the reset button it goes to zero.  This suggests to me that UFraw did make an adjustment even though I hadn't requested any change.  If I set the value to 1.0 the picture resembles what DPP4 is showing me without trying to change anything.  At this point I have no idea what kind of adjustments DPP4 might be making without my knowledge.

The idea that UFraw or any other editing software can make simple adjustments that improve the appearance is what we're after but I want to know what is being changed.  I've done enough study to know that camera's have a lot of exotic software that automatically performs some, if not a lot, of editing.  That was my prime motive for being able to produce raw files was for me to be in control of such changes, which I'm expecting to help me learn how this stuff works.  The idea of setting the camera to make both raw and jpeg files is something I also thought might help.  My camera setting have already been changed but I have yet to take any pictures.  I'm going to try and match the conditions in the example photo we've been discussing.  Hopefully tomorrow.

I'll try the suggestion regarding GIMP export to jpeg and post results.

[ajax]

I took pictures this morning of the same scene with lighting as close as, my guess, could make it to being similar.  In that about the same time of day with similar cloud formations.  This time the camera is set to record both a raw and a jpeg file.  When using DPP4 to view both files they are indistinguishable to my admittedly untrained eye.  However when opened in GIMP where the UFraw plugin is used to load the raw image the very same kind of difference that triggered this post is evident.  Whereas with GIMP the .jpg file looks the same as with DPP4 and other viewers.

My speculation is that the camera, as I expected, made some automatic corrections when preparing the .jpg file.  Then the same, or at least similar corrections, are being made by DPP4 when it opens the raw (.CR2) file.  This would mean that when using DPP4 I'm not getting to see the actual raw file.  I'm also thinking that this defeats the purpose of having raw images.  Possibly if I study the Instruction Manual for DPP4 there is some solution for this problem.  In that, either turn off or reverse these automatic corrections.  To the extent that my speculation is correct I'm not sure I'd want to waste my time doing that.  As a result, it now looks to me like UFraw may be delivering the actual raw image with a caveat about, my previous observation and, what looks like UFraw automatically making some exposure adjustment, which can be easily undone.

Finally, as suggested, turning off the "progressive" option when using GIMP to export the same .jpg file referred to above did result in DPP4 being able to open it.  Seems like another reason not to waste one's time working with DPP4.

Based on the various findings resulting from this discussion I'm also now inclined to think that Picasa is handling the raw files more correctly than DPP4.

Where am I going wrong?

[Ofnuts]

Where you are going wrong is that you assume that RAW processing is automatic. It is not. DDP4 gives the same result at the JPEG from the camera because its *default behavior* is to do so, but you typically use RAW when you think the automated processing (in-camera or with DPP4, which uses the camera settings encoded in the file metadata) is not going to cut it and you want to do it yourself.
In fact, with DPP4 you can even go the other way: you take a raw, process it to get the results you expect, and then you can tell DDP4 and the EOS Utility to upload this process "recipe" to the camera as a user setting.

Also, by default your camera settings for JPG are very neutral, because you can always post-process a neutral photo. if you know you won't do post processing on the JPG, you can add more contrast/saturation/sharpening but if this makes more pleasant pictures it also makes them harder to correct.

Personally, I take JPG+CR2, and do the culling on the JPEG. Then there are plenty of pictures for which I know I won't need "deep" post processing (family gatherings, sports events) and for which I discard the CR2 right away (unless I have problems that I can fix with the CR2, such as a bad color balance). I keep the CR2 for picture that have a difficult lighting or that I find very good (rare!)

[ajax]

I think we're saying the same thing but my lack of experience with digital photography may have me using terminology that sounds strange.  When I said "Then the same, or at least similar corrections, are being made by DPP4 when it opens the raw (.CR2) file." I'm sort of equating that to what you mean when saying "DDP4 gives the same result at the JPEG from the camera because its *default behavior* is to do so".  Your point about DPP4 using the metadata to determine what automatic processing, which I called automatic corrections, to perform is a subtlety that I hadn't considered but does make plenty of sense.  Insofar as the camera and DPP4 are intended to complement one another this also appears to be a capability that would make sense for DPP4 whereas other more generic image processing software vendors might forgo trying to do such for all the possible cameras they intend to support.

My reason for buying a camera with raw capability was so that I could be able do what you say "but you typically use RAW when you think the automated processing (in-camera or with DPP4, which uses the camera settings encoded in the file metadata) is not going to cut it and you want to do it yourself."  If I assume that Canon & the DPP4 developers have the same intention when it comes to raw why do they think it is a help to perform automatic processing on a raw file?  I suppose the answer could be "because they can so why not".  My speculation was that this, what you call "default behavior", is a reasonable starting point and I'd agree this is fine as long as I can turn it off and work with the unprocessed (i.e. raw?) image.  That is what I haven't figured out how to do with DPP4 but I am thinking that is what I come pretty close to getting with UFraw.  I've also now looked into Rawtherapee, per your tip, and it looks worth pursuing further.  I have a scanner that is capable of producing both color and grayscale in 16bit formats.  Support for this has some appeal to me and there is no way to tell when GIMP 2.10 might emerge.

Your point that I have control over how the camera does automatic processing (i.e., "In fact, with DPP4 you can even go the other way:") is something I've not realized but is something I appreciate your advising me about and will definitely want to learn how to do.  Many thanks for pointing that out.

I suppose where I'm different than lots of people is that, at present, my motive for doing it myself is to learn how this stuff works rather than the kind of pictures I end up getting.  It is entirely possible that I never end up making a better picture than what is produced using the automatic processing but I'll be happier with that outcome when I have a better understanding of the whole process and appreciation for what the automatic processing achieved.

I certainly did not intend to convey the impression that I assumed or expected raw files to be subject to automatic processing.  My reason for posting this thread was lack of certainty about what the raw file should look like and therefore whether or not revisions I might make are the only ones in affect.  My initial assumption was that DPP4 should be showing it to me.  Isn't that where I was wrong?

[Ofnuts]

(08-17-2017, 03:58 PM)ajax Wrote: That is what I haven't figured out how to do with DPP4

Maybe you should read the doc, because DPP4 is designed to allow such editing. I used it ages ago (when I as still using Windows) and doing this was a no-brainer.

[ajax]

(08-17-2017, 08:59 PM)Ofnuts Wrote:

(08-17-2017, 03:58 PM)ajax Wrote: That is what I haven't figured out how to do with DPP4

Maybe you should read the doc, because DPP4 is designed to allow such editing. I used it ages ago (when I as still using Windows) and doing this was a no-brainer.

Yes, I've now done that.  As best I can tell it doesn't do what I'd like.  I've also learned that it does something else which strikes me as pretty strange.  It updates the raw file (i.e., the one created by the camera).  Supposedly this is done in a manner that does NOT alter the image data captured by the sensor (i.e., the raw portion of the data is preserved) and only what they call "image processing conditions information" is changed.  However, when you look at the modification date in the file system directory it doesn't say what software performed the modification nor what was modified.  I do have some knowledge and experience in IT and would offer that the way GIMP does it by creating its' own file in it's own format while leaving the file created by the camera unaltered is much better.

Anyway, our discussion has strayed pretty far from GIMP.  I'd like to say that I very much appreciate your willingness to engage.  It has been a help to me.  I'll now shut up and get on with some needed research/learning.  No doubt there will be some future questions about GIMP.  Many thanks.