It is a special mode for paint tools. As its name implies, it work as if you were painting on the back of the layer, behind the existing colors (so the paint is visible only where the initial pixels are at least partially transparent). It is strictly equivalent to adding a layer below the one your want to paint, bucket-filling it with the color, and then merging down the target layer.
See here for some examples.
See here for some examples.