“CariGANs" are the first Generative Adversarial Network (GAN) for unpaired photo-to-caricature translation. In simple terms, this means enabling a machine to learn to create a caricature from a real image without human intervention. Head to https://www.razorthink.com/ to learn more.

1. CARIGANs
UNPAIRED PHOTO-TO-CARICATURE TRANSLATION

2. AGENDA
2
❖ GANs
❖ CariGANs: Unpaired Photo-to-Caricature Translation
❖ How are they doing it?
❖ CariGANs’ Performance
❖ Extensions / Scope

3. GANS (GENERATIVE ADVERSARIAL NETWORKS)
3

4. Generative adversarial networks (GANs) are deep neural net architectures comprised of two nets, pitting one against the other (thus the
“adversarial”).
The difference between generative networks and discriminative networks is that the later maps features to labels. They are concerned
solely with that correlation. One way to think about generative algorithms is that they do the opposite. Instead of predicting a label given
certain features, they attempt to predict features given a certain label.
How Gans Work ?
One neural network, called the generator, generates new data instances, while the other, the discriminator, evaluates them for authenticity;
i.e. the discriminator decides whether each instance of data it reviews belongs to the actual training dataset or not.
The discriminator network is a standard convolutional network that can categorize the images fed to it, a binomial classifier labeling images
as real or fake.
The generator is an inverse convolutional network, in a sense: While a standard convolutional classifier takes an image and downsamples it
to produce a probability, the generator takes a vector of random noise and upsamples it to an image.
GAN
4

5. CARIGANS: UNPAIRED PHOTO-TO-CARICATURE
TRANSLATION
5

6. The first Generative Adversarial Network (GAN) for unpaired photo-to-caricature translation, which are called “CariGANs".
CARIGAN
6

7. 1. Given a real picture how to learn a mapping that converts it to a caricature.
1. The exaggerated shape should maintain the relative geometric location of facial components, and only emphasize the subject’s
features, distinct from others.
1. The final appearance should be faithful to visual styles of caricatures, and keep the identity with the input face, as addressed in
other face generators.
1. Moreover, the generation must be diverse and controllable. Given one input face photo, it allows for the generation of variant types
of caricatures, and even controls the results either by example caricature, or by user interactions.
1. We want to learn a mapping Φ : X → Y that can transfer an input x ∈ X to a sample y = Φ(x), y ∈ Y .
WHAT ARE THEY TRYING TO ACHIEVE ?
7

8. 1. Earlier efforts were mostly supervised learning. Given a real image and a pairing with the caricature image. The network learns how
to convert it to the caricature image .
1. As is commonly known, most photo and caricature examples are unfortunately unpaired in the world. Building such a dataset with
thousands of image pairs (i.e., a face photo and its associate caricature drawn by artists) would be too expensive and tedious.
PREVIOUS EFFORTS
8

9. Let X and Y be the face photo domain and the caricature domain respectively, where no pairing exists between the two domains.
1. For the photo domain X , we randomly sample 10, 000 face images from the CelebA database {x i } i=1, ..., N , x i ∈ X which covers
diverse gender, races, ages, expressions, poses and etc.
1. To obtain the caricature domain Y , we collect 8, 451 hand-drawn portrait caricatures from the Internet with different drawing
styles (e.g., cartoon, pencil-drawing) and various exaggerated facial features, {y i } i=1, ..., M , y i ∈ Y .
DATA COLLECTION
9

10. HOW ARE THEY DOING IT ?
10

11. The pipeline is split into two GANs:
1. CariGeoGAN : which only models the geometry-to-geometry transformation from face photos to caricatures.
2. CariStyGAN, which transfers the style appearance from caricatures to face photos without any geometry deformation.
3. Two GANs are separately trained for each task, which makes the learning more robust.
4. To build the relation between unpaired image pairs, both CariGeoGAN and CariStyGAN use cycle-consistency network structures,
which are widely used in cross-domain or unsupervised image translation.
5. Finally, the exaggerated shape (obtained from CariGeoGAN ) serves to exaggerate the stylized face (obtained from CariStyGAN )
via image warping. The warping is done by a differentiable spline interpolation module.
6. In CariGeoGAN, we use the PCA representation of facial landmarks instead of landmarks themselves as the input and output of
GAN. This representation implicitly enforces the constraint of face shape prior in the network.
7. In the second stage, we use CariStyGAN to learn the appearance-to-appearance translation from photo to caricature while
preserving its geometry. Here, we need to synthesize an intermediate result y ′ ∈ Y ′ , which is assumed to be as close as caricature
domain Y in appearance and as similar as photo domain X in shape.
HOW ARE THEY DOING IT ?
11

12. HOW ARE THEY DOING IT ?
12

13. GEOMETRY DEFORMATION
13
1. Face shape can be represented by 2D face landmarks either for real face photos X ,or for caricatures Y .
1. We manually label 63 facial landmarks for each image in both X and Y .

14. 1. To centralize all facial shapes, all images in both X and Y are aligned to the mean face via three landmarks (center of two eyes and
center of the mouth) using affine transformation.
1. In addition, all images are cropped to the face region, and resized to 256 × 256 resolution for normalizing the scale
1. Note that the dataset of real faces is also used to finetune a landmark detector, which supports automatic landmark detection in
the inference stage.
1. For our CariGeoGAN, to further reduce dimensions of its input and output, we further apply principal component analysis (PCA) on
the landmarks of all samples in X and Y . We take the top 32 principal components to recover 99.03% of total variants. Then the 63
landmarks of each sample are represented by a vector of 32 PCA coefficients. This representation helps constrain the face structure
during mapping learning.
GEOMETRY DEFORMATION(CONTD)
14

15. 1. CariGeoGAN is inspired by the network architecture of CycleGAN.
1. The forward generator learns the mapping from real images to caricature images while the backward generator learns the exact
inverse of the forward generator.
1. As it is a Cycle Gan. Both the backward and forward generators share weights in both the paths.
1. The discriminators learn to distinguish real samples and generated samples.
CARIGEOGAN
15

16. CARIGEOGAN(SUMMARIZED)
16

17. CARIGEOGAN (LOSSES)
17
We define three types of loss in the CariGeoGAN.
1. The first is the adversarial loss, which is widely used in GANs. Specifically, they adopted the adversarial loss of LSGAN to
encourage generating landmarks indistinguishable from the hand-drawn caricature landmarks sampled from domain Y . Similarly
another loss is defined for the generating portrait photo landmarks that cannot be distinguished from real photos.
1. The second is the bidirectional cycle-consistency loss, which is also used in CycleGAN to constrain the cycle consistency between
the forward mapping and the reverse mapping. The idea is that when we apply reverse mapping to the synthesized image we
should get back to the original input image.

18. CARIGEOGAN (LOSSES)
18
3. Cycle-consistency loss further helps constrain the mapping solution from the input to the output. However, it is still weak to
guarantee that the predicted deformation can capture the distinct facial features and then exaggerate them. The third is a new
characteristic loss, which penalizes the cosine differences between input landmark and the predicted one after subtracting its
corresponding means.
● The characteristic loss in the reverse direction is defined similarly.
● The underlying idea is that the differences from a face to the mean face represent its most distinctive features and thus should be
kept after exaggeration. For example, if a face has a larger nose compared to a normal face, this distinctiveness will be preserved or
even exaggerated after converting to caricature.
Our objective function for optimizing CariGeoGAN is:

19. APPEARANCE STYLIZATION
19

20. CARIGANS PERFORMANCE
20

21. PERFORMANCE
21
1. Our core algorithm is developed in PyTorch. All of our experiments are conducted on a PC with an Intel E5 2.6GHz CPU and an
NVIDIA K40 GPU. The total runtime for a 256 × 256 image is approximately 0.14 sec., including 0.10 sec. for appearance stylization,
0.02 sec. for geometric exaggeration and 0.02 sec. for image warping.
1. The perceptual study shows that caricatures generated by CariGANs are closer to the hand-drawn ones, and at the same time
better persevere the identity, compared to state-of-the-art methods.
1. Moreover, our CariGANs allow users to control the shape exaggeration degree and change the color/texture style by tuning the
parameters or giving an example caricature.

22. EXTENSIONS / SCOPE
22

23. EXTENSIONS / SCOPE
23
Video Caricature: We directly apply our CariGANs to the video frame by frame. Since the CariGANs exaggerate and stylize the face
according to facial features. The results are overall stable in different frames.

24. EXTENSIONS / SCOPE
24
Caricature-to-photo translation: Since both CariGeoGAN and CariGeoStyGAN are trained to learn the forward and backward mapping
symmetrically, we can reverse the pipeline to convert an input caricature into its corresponding photo.

25. THANK YOU
25

26. REFERENCES
26
CariGANs: Unpaired Photo-to-Caricature Translation
KAIDI CAO∗†,Tsinghua University JING LIAO‡,City University of Hong Kong, Microsoft Research LU YUAN,Microsoft AI Perception and Mixed Reality
Can an AI Learn To Draw a Caricature? From Two Minute Papers