Learning-based image harmonization techniques are usually trained to undo synthetic random global transformations applied to a masked foreground in a single ground truth photo. This simulated data does not model many of the important appearance mismatches (illumination, object boundaries, etc.) between foreground and background in real composites, leading to models that do not generalize well and cannot model complex local changes.
We propose a new semi-supervised training strategy that addresses this problem and lets us learn complex local appearance harmonization from unpaired real composites, where foreground and background come from different images. Our model is fully parametric. It uses RGB curves to correct the global colors and tone and a shading map to model local variations. Our method outperforms previous work on established benchmarks and real composites, as shown in a user study, and processes high-resolution images interactively
We present a demo video to demonstrate the workflow of our method running on CPUs. Our model is fully parametric. It uses RGB curves to correct the global colors and tone and a shading map to model local variations. Users can incorporate posthoc edits to both the curves and the shading map, greatly enhancing users' creative controls!
Our proposed semi-supervised dualstream training strategy alternates between two training streams: a) Supervised training with artist-retouched composite image pairs (left). Artist adjustments include global color editing, shading correction, and other local edits. b) Unsupervised adversarial training with realworld composite images (right). It uses a GAN training procedure, comparing our harmonized results with a large dataset of realistic image composites.
RGB curves harmonize the global color/tone (center), while our shading map corrects the local shading in the harmonization output (right).
Representative visual comparisons between state-of-the-art harmonization results. Our results show better visual agreements with the ground truth in terms of color harmonization (rows 1,2 and 4) and shading correction (row 3).
@article{wang2023semi,
title={Semi-supervised Parametric Real-world Image Harmonization},
author={Wang, Ke and Gharbi, Micha{\"e}l and Zhang, He and Xia, Zhihao and Shechtman, Eli},
journal={arXiv preprint arXiv:2303.00157},
year={2023}
}This work was done by Ke Wang during the Adobe internship. The website template is taken from Nerfies project page.