mutually exclusive layer masks

[jonnie45]

Hi, newbie to the forum here.

I wondered if anyone could advise on the following?

I am using artistic effects that apply to the whole layer rather than ones that can be used with a paint tool.

I want to style the artistic effect differently in different parts of the image - for example an artistic effect might have a stroke direction - an 'angle' -
so "on this part of the image strokes should be at 30 degrees and over here strokes should be at 60 degrees".

So the problem is how to vary effects that apply to the whole layer with regional variations by changing a variable or setting supplied in the artistic filter, I am using 'stroke angle' for an example but it could be any variable that can be changed on an artistic effect that is applied to the whole layer. 

The method I have been experimenting with is to make say 5 layers and apply the "whole layer" artistic effect to each layer in turn, each time with a different stroke angle.
So I may have

layer0 - stroke angle of 0 degrees applied to artistic effect
layer1 - stroke angle of 30 degrees applied to artistic effect
layer2 - stroke angle of 60 degrees applied to artistic effect

and so on....

I then added a layer mask to each layer.

This means I can achieve the result I want and I can see the result live in the main layer as I use say the rubber tool to remove areas of layer mask for each layer, if I rub out the layer1 mask at one location I can have a stroke of 30 degrees and then if I rub out the layer2 mask in a different place then I see the 60 degree stroke coming through.

Great it works !!! But.........


The only problem is that for this to work well I need to make sure that my layer masks are mutually exclusive from one another - if I use the rubber tool on one layer mask at position A then I must not use the rubber tool at that same position in any of the other masks otherwise I will not have the desired result that for any pixel the contribution comes from only one of my layers - in my case the layers in question are mode=dodge so I do not get the desired effect if more than one layer is allowed to contribute to any single pixel.

I cannot use modes on the layers themselves to combine them in different ways because they all have to be "dodge".

So it seems that the only way I can get the mutual exclusivity I want is to have some "magic" option that views my five layer masks as being part of a set and if a pixel in any one of these masks is transparent then the same pixels in all the other masks must be opaque - if I use the rubber tool to make a particular pixel transparent then the other layer masks must all react automatically and set that same pixel opaque.

It could be progressive, for instance perhaps the pixel at a particular position in the image might have transparency in the layer masks as follows  0.6, 0.2, 0.1, 0.1, 0.0   - ie adding up to 1.0 at all times - so if I used a rubber tool I might use multiple strokes to progressively make layer 5 dominant in a particular region but always so that the contribution from the other layers is decreased accordingly.   

I can't help thinking it must be a common wish to use an artistic effect that applies to the whole image and to wish to use layers each with the same image but a different setting for the artistic effect and then to want to paint on masks to determine which setting is used in different places on the screen?

Is there any kind of mask that can take say 5 layers as 'input' be painted on freehand with facilities to allow only one layer as output or to blend the contribution from each layer in a sensible way?

Conceptually a mask where each pixel has a value 1 to 5 to indicate layer and so can only accept one layer as input for any particular pixel? -

Or a mask system were five channels are blended according to how I paint on the mask but the result is only a weighted average of the 5 layers, perhaps a system that is progressive so that if I wish to have layer 5 dominate in a particular area then repeated strokes of a tool on the mask in that area will progressively weight more and more to layer 5 with the other other layers diminishing in their input at every stroke?

Its not an easy one to describe

Thanks in advance
Jon

[Ofnuts]

You can substract selection from others or you can obtain a selection from the maslk of one of the layers, and then use it to paint the other masks black (so all other layers are transparent where that one is opaque).

If you create a layer group, this creates a composite image that only includes the layers in the group, and then you can put another layer mode on the layer group, to apply the result of the group with that mode to the rest of the image(*). In your case, that could be just using the "Normal" mode for your layers, and leaving the Dodge mode to the group.

You can also apply a mask to the group, if necessary..

(*) somewhat like parentheses in algebra

[jonnie45]

(06-10-2021, 12:52 PM)Ofnuts Wrote: You can substract selection from others or you can obtain a selection from the maslk of one of the layers, and then use it to paint the other masks black (so all other layers are transparent where that one is opaque).

If you create a layer group, this creates a composite image that only includes the layers in the group, and then you can put another layer mode on the layer group, to apply the result of the group with that mode to the rest of the image(*). In your case, that could be just using the "Normal" mode for your layers, and leaving the Dodge mode to the group.

You can also apply a mask to the group, if necessary..

(*) somewhat like parentheses in algebra

Hi Ofnuts, thanks for the reply - yes I have got it working now using a similar method. However it would still be a great asset when mixing inputs from several different modified versions of the same image to have a method whereby when the input for a particular pixel was increased for layer 1 ( say ) then the other layers would be reduced automatically - this would allow an artist to blend the inputs more naturally.

At the moment the order is the order that the layers are arranged. This means if I burn a big hole in the first mask and then go to the last layer in the stack and want to make that one contribute more I first have to go and heal the hole in the top mask otherwise it always dominates.

The feature I would like to have here would be that if I paint on the mask for layer 3 say - perhaps with a brush of opacity 0.1 the each brush stroke on the layer 3 mask is not only increasing the contribution from layer 3 but it is also diminishing the contribution from the other layers - that way I can stay on layer 3 and at every stroke make it more and more dominant, then maybe I change my mind and go to layer 2 and make that one dominant in the same place - brush stroke by brush stroke. At the moment I have 5 masks to worry about and it gets very complicated changing your mind - you have to keep on "undoing" what you did with other masks to make the changes on the current one have any affect.

So in my mind I see a way of linking say 5 masks together so that whatever you do to one mask the other masks are automatically adjusted so that the net contribution from all the masks always adds up to 100%.

It would be like painting with semi-transparent colours - every stroke of 10% opacity green gradually changes the previous colour of say blue but the image does not get brighter.

What I want to do is to paint not with colours but with the five variations of the same image so that I can blend the contribution from each of the five images at a particular place and let for instance image 2 dominate at this place and image 3 dominate at that place.

I do have this working as I said but I have to manage all 5 masks rather than just painting on the one I am currently most interested in knowing that as it's contribution is increased  the others will be automatically decreased.

[Ofnuts]

The opacity of layers is not additive. If you want the result to be fully opaque at least one of the layers has to be fully opaque too. So if you make an opaque layer non-opaque, which one of the other layers has to become opaque?

[jonnie45]

(06-11-2021, 12:09 PM)Ofnuts Wrote: The opacity of layers is not additive. If you want the result to be fully opaque at least one of the layers has to be fully opaque too. So if you make an opaque layer non-opaque, which one of the other layers has to become opaque?

Thanks Ofnuts

I guess an implementation of what I want would first create a "Result layer".

I then create say 5 layers - in my case of the same image but with different artistic effects ( or different settings of the same artistic effect ).


Each pixel in the Result layer might start off initially being 100% of the corresponding pixel in layer 1 - we have to start somewhere so I choose layer 1. 

I then paint strokes on the mask for (say) layer 2 at a strength value of 0.2 for each stroke then instead of changing the layer that the mask belongs to the action instead would be to change pixels in the Result layer with a weighted calculation for instance for the pixel at X = 1, Y = 1, if the stroke passes over it then....

RGB Result pixel ( 1,1 ) = RGB Existing Result Layer pixel ( 1,1 ) * 0.8 + RGB Layer 2 pixel( 1,1 ) * 0.2

The result pixel has to be recalculated every stroke so that the effects are accumulative in the Result layer.
 
So every time I stroke on the mask for layer 2 the Result layer in that area progresses increasingly towards being the same as layer 2 in that area.

If I now change my mind and turn to layer 3 then I can use repeated strokes on layer 3 to now make the same area in the Result layer move increasingly towards layer 3.

This way I don't have to go back and repair masks when I change my mind in a particular area, I just proceed to overwrite that area with a weighted contribution from the layer I have selected instead.

In way there is no real requirement for mask functionality if a way could be found to allow a paint tool to pass over a layer and without altering that layer record which pixels were passed over and to calculate the effect of their weighted average on the Result layer

In a sense I want to paint but with brushes that correspond to the whole image modified by various filters and to incrementally blend those different versions of the same image by the act of painting operations - repeating strokes to get a stronger effect.

I think this could be a very useful capability - there are some very useful filters which work on the whole image and do not work directly as brushes so my way of thinking is to make up as many layers as I have filter variations in mind and then to bleed through contributions progressively. The chief advantage is that I do not have to work out which mask it is that I need to repair to change something I did earlier that I regret - I can move forwards all the time simply overwriting areas I am not happy with until it turns out right.

If anyone has pointers in terms of the best way to implement this I would be grateful- I am an experienced coder but no real experience in writing plugins or scriptfu for gimp.