Brainstorming/Color transformations in YCbCr space

Update 2013-03-20 The W3C have now settled on a standard way of doing Hue, Huminosity, Saturation and Color modes, now published as [Compositing and Blending 1.0]. The OpenRaster specification has been updated to refer to these algorithms, so the following brainstorm is of historical interest only.

See also:
 * Brainstorming/Comparison of candidate Color and Lightness modes - this and other candidate methods
 * Brainstorming/Layers

Colour remapping, or colorization, or lightness adjustment are ultimately the same thing: a method for combining two colour values using the brightness of one regardless of its colour, and colour of the other regardless of its brightness.

This page describes a scheme for doing this, explaining what it would mean to our users, gathers the technical requirements, outlines a simple algorithm for adjusting lightnesses of colours via the YCbCr colour space, and discusses various options and algorithms for resolving the resultant out-of-gamut problem when converting the result back to RGB space for display, storage, and export. The goal is to develop a carefully-worded and complete technical specification which can be shared between all programs working with .ORA files.

Use cases
MyPaint's users have a need for remapping color via either brushmodes or layers in a way similar to the way Photoshop or GIMP's "Color" layer modes operate.


 * Alice doesn't think naturally in terms of colour, at least not at first. She's really good with form, light and shade though, so she prefers to paint in black, white and intermediate shades of grey at first, playing with form and chiaroscuro and getting her composition right that way. She normally colourises such work after she has painted them in greyscale using Photoshop's "Color" layer mode, but likes the idea of Open Source software and wants to give MyPaint a try, as an alternative to other painting programs.


 * Bob uses colour quite naturally, right from beginning a composition, almost working a fluid free-form colour prototype into a finished artwork. The trouble is, he likes to change things half-way through, at all stages of the process, altering colours and brightnesses as he goes and correcting mistakes. He needs to be able to see and compare alternative strategies before picking one and using Merge down to finalise things. And given the way he works, sometimes one of the scenarios he needs is a pure greyscale view because colour is sometimes misleading and can obscure tonal similarities or differences he doesn't want there when his work is converted to greyscale.


 * Carol works in both MyPaint and the GIMP, sharing ORA files between the two programs, and works as Alice does. Even though she doesn't use colour management on her computer, it's important to her that the same transform be used in both for remapping colours, because if that's not true, the two programs will export an identical file to PNG in ways which look different.

Perceptually relevant colour spaces
RGB is not sufficient for colour remapping in this way directly; the colourspace we need should have a lightness parameter which can be varied independently of colour. The HSV colour space which MyPaint uses internally for brushes and colour choice is unhelpful too: its V component is a lightness axis in the broadest sense, but it is not perceptually relevant. For example, both blue and green primaries contribute equally to HSV's V, but the human eye doesn't respond like that).

Requirement: the algorithm must produce results which are perceptually relevant.

The GIMP uses LCHab colourspace (a variant of L*a*b*) when it's in GEGL mode. This however is rather slow because the transformation must pass through several steps. LCHab is a cylindrical space with a perceptually relevant Luminance axis, L. Gamut mapping is required, and is rather complex.

The YCC spaces - YIQ, and YUV (sometimes called YCbCr) - look sensible as well: their Luminance (Y) parameters are perceptually relevant by design, and it's computationally easy to convert RGB values to and from YCC spaces. YIQ and YCbCr are used in video processing and broadcast analogue and digital TV. A workaround to the out-of-gamut problem is required here too, but because the space is geometrically simple this can be done speedily and quite effectively.

Interoperability and the ORA format
It's vital that the colour remapping scheme we choose is supported by as many programs as possible if the scheme is to be used as a way of combining layers in the layers stack. Therefore there should be broad agreement between the Krita, MyPaint and GIMP developers before moving forward.

Colour mapping via YCrCb
The YCrCb colour space is a practical choice for computational reasons when you are assuming RGB storage and display. Conversions to and from the working space are fast, and covered in detail in the literature.

RGB and YCrCb conversion
The formulae and/or matrices for conversion are well known. The transformation below requires linear light, so conversion to linear RGB may be necessary beforehand. We should probably use the Rec. 709 coefficients


 * KB = 0.0722
 * KR = 0.2126
 * KG = 1 - KB - KR = 0.7152

because these reflect how current models of display equipment are calibrated.

Luminance assignment
The problem of applying combining the chrominance from one layer onto the luminance of a second layer is identical in effect to applying the luminance of the second to the chrominance of the first. Most specifications in the literature define the operation in terms of a SetLum operation, so let's do the same.

func SetLuminance(rgbA, rgbB): yccA = rgb2ycc(rgbA.{r, g, b}) ; cheaper to calculate just the Y     yccB = rgb2ycc(rgbB.{r, g, b}) rgbOut = ycc2rgb(yccA.Y, yccB.C1, yccB.C2)

So far so good, but it results in out-of-gamut values so we need to resort to some form of clipping or scaling. The same fix to the out-of-gamut problem can also be used for any lightening or darkening filter which works in terms of perceptual lightness.

Fixing out-of-gamut values
Simply assigning the luminance of one colour to another is intuitive and obvious, but because the YCbCr space is larger than the RGB space and does not share its shape, this can result in out-of-gamut colours. The RGB gamut placed inside an YCC space looks very roughly like a rotated cuboid standing on its black corner with its white corner directly above in the Y axis and the other 6 corners (C,M,Y,R,G,B) at various heights which correspond to their contribution to the overall perceptual brightness. Geometrically it is a parallelepiped.

Requirements

 * The fixup algorithm must preserve luminance (so simply clipping rgbOut to valid RGB values won't work). This means that hue or saturation, or both must be sacrificed.


 * Preservation of hue (chrominance vector angle from a given chrominance axis) is more important than preserving saturation (magnitude).


 * It must be fast.


 * It must not require the pre-computation of histograms for the layers to be merged. This property also makes the luminance setting algorithm useful for brush blend modes, and simplifies operation.

Options

 * Naive linear scaling along planes of constant luminance proportionate to the Y-distance from Y=1 (for yccA.Y > yssB.Y) or Y=0.
 * This results in fairly significant loss of saturation from my experimentation.


 * Yoonsung Bae et al. "Gamut-Adaptive Correction in Color Image Processing"; Proc. IEEE 17th Int. Conf. on Image Processing Sep 26-29 2010 Hong Kong; Department of Electrical Engineering, KAIST, Daejeon, Korea; via http://dx.doi.org/10.1109/ICIP.2010.5652000
 * This describes a rather speedy "GAS" technique for gamut-adaptive scaling, and a similar "GAC" technique for gamut-adaptive clipping, and doesn't suffer much from greying towards the poles of the RGB parallelepiped. It sacrifices saturation, but not hue. Looks rather nice in practice.
 * Of note, the recommendations from this paper regarding which of the GAC/GAS techniques to apply depend on whether you're performing chrominance manipulation (corresponding to our desired set-colour operation, and for which the authors recommend GAC) or luminance manipulation (corresponding to our desired set-lightness operation; here GAS is suggested).


 * Devereux VG and the BBC. "Limiting of YUV Digital Video Signals"; BBC Research Department Report December 1987, retrieved from http://downloads.bbc.co.uk/rd/pubs/reports/1987-22.pdf.
 * This describes a simple clipping scheme; it is covered by the (expired) UK patent GB2198008. To be honest, I've not been able to get this to work yet.


 * Samadani R, Guo Li. "Geometrical Methods for Lightness Adjustment in YCC Color Spaces"; HP Laboratories Palo Alto; HPL-2005-191.
 * A technique for reducing (or, for linear YCC, eliminating) clipping by traversing chrominance values along Bezier curves. Untried.