Rate-distortion optimizers and optimization techniques including joint optimization of multiple color components

Examples of encoders and video encoding are described that include optimizers and techniques for optimizing syntax elements such as transform coefficients. In some examples, multiple color components of a video signal may be jointly optimized by employing a cost calculation using a combination of distortion and/or rate metrics for multiple color components. In some examples, a color transformation may occur and the optimization may take place in a different color domain than encoding. In some examples, distortion metrics used in the cost calculations performed by optimizers are based on structural similarity index.