Weird...
So the next step is: what did you do to the mask? A simple thresholding doesn't produce invalid results.
- If I use your "Before" image, and threshold the mask to 0.06 I get essentially the same mask as in your "impossible colors" picture, but the image looks good.
- If I copy/paste that new mask to the mask of the bad picture, the picture is fixed
- If I copy the mask of the bad picture and the mask of the good picture as layers in a new picture and compare them... more weirdness ensues. The masks look identical, so putting the top one in Difference mode should give a black image.... and it doesn't. And when I use the Pointer dialog to look at the mask values, these values are completely out of range. The values that should all be 0. <= x <= 1.0 (as they are in the mask if the good picture) are in the thousands range! Also, they appear quite random, and two adjacent pixels have very different values.
- So my guess is that due to the out of range values in the mask, the code that computes the final color gets out of range colors and what is displayed is a bit random.
So the next step is: what did you do to the mask? A simple thresholding doesn't produce invalid results.