Personalized Image Generation

Personalized image generation adapts a pretrained text-to-image model to a specific subject that the user supplies in a few photographs, such as one dog, one backpack or one toy, so that the model can then draw that subject in scenes, poses and styles that none of the photographs show. The subject is bound to a new word, and prompts that use the word (“a [V] dog on the beach”) steer the rest of the image as usual. The papers call the task personalization, subject-driven generation (Ruiz et al., 2023) or customization (Kumari et al., 2023). It differs from image editing, which changes one given image and keeps the rest of it, and from ordinary fine-tuning, which adapts a model to a whole distribution rather than to one instance. The methods described here are all built on diffusion models, most of them on Stable Diffusion.

Purpose

A text prompt cannot pin down a particular object. DreamBooth’s authors argue that even the most detailed description may yield instances with different appearances, and show a yellow alarm clock that Imagen and DALL·E 2 could not reproduce despite a long, hand-tuned prompt (Ruiz et al., 2023, §1 and Fig. 2; supplementary, Fig. 15). Personalization adds the missing information from images instead of words. The methods differ in where they put it: in a new word embedding, in all or some of the model’s weights, in a small low-rank or spectral update, or in an encoder that maps the images to conditioning in one forward pass.

When to Use It

Personalization fits when the same subject must reappear in many generated images, for example a product in different settings, a character across the panels of a comic (DreamBooth’s supplementary material shows one) or a pet in artistic renditions. DreamBooth worked with 3 to 5 images per subject; for the two subjects it tested with 1 to 5 images, 4 gave the highest subject fidelity (supplementary, Table 1; the prompt-fidelity table that follows repeats, for the backpack, the numbers of the dog’s subject-fidelity row, so it cannot be read as printed); Custom Diffusion reports results with as few as four (§1); encoder methods such as ELITE and InstantBooth also accept a single image.

It is a poor fit when the goal is to change one particular photograph while keeping everything else, which is editing, or when the subject is already well described by words, where a prompt costs nothing. The optimization-based methods cost minutes per subject on data-centre GPUs: about 5 minutes for DreamBooth on one A100 for Stable Diffusion (Ruiz et al., 2023, §3.3), and 6 minutes on two A100s for Custom Diffusion, whose authors put textual inversion at 20 minutes on two A100s and their DreamBooth baseline at about an hour on four (§4.1); each subject yields a stored model or weight update. Encoder methods remove the per-subject training, but need a large training set of their own; InstantBooth, for example, is trained separately for each category in the main paper, and its supplementary material adds a generic model trained on 10 million images of general objects. The subjects these papers use are objects, pets and, in InstantBooth, HyperDreamBooth and Anti-DreamBooth, people; personalizing a model on a real person raises consent and misuse questions taken up under Risks below.

How It Works

All methods start from a pretrained conditional denoiser $\epsilon_\theta(x_t, t, c)$, where $c = \Gamma(P)$ is the text conditioning computed from a prompt $P$ by a text encoder $\Gamma$, and from the usual noise-prediction loss on the few subject images $x$:

$$\mathcal{L}_{\text{subj}} = \mathbb{E}_{x, \epsilon, t}\,\big\| \epsilon - \epsilon_\theta(\sqrt{\bar\alpha_t}\,x + \sqrt{1-\bar\alpha_t}\,\epsilon,\; t,\; \Gamma(P_{\text{subj}})) \big\|^2 .$$

DreamBooth writes the same loss in the image-prediction form with a time weighting $w_t$ (Eq. 1), which differs from the noise-prediction form only by a time-dependent factor. What differs between methods is which parameters the loss updates and what keeps the rest of the model’s knowledge intact.

Textual inversion

Textual inversion (Gal et al., 2023; not read here, so this account comes from the papers that describe and compare against it) adds one new pseudo-word $S_*$ to the text encoder’s vocabulary and optimizes only its embedding vector on the subject images; the denoiser and the text encoder stay frozen (Ruiz et al., 2023, §2; Kumari et al., 2023, §4; Wei et al., 2023, §1). The result is a single vector that can be used in any prompt. Its limit, as DreamBooth’s authors put it, is the expressiveness of the frozen model: the new word can only point to images the model can already produce (§2). ELITE’s authors add that describing a concept with a single word may leave its local details inconsistent (§1). InstantBooth notes that the textual inversion it compared uses the LDM text-to-image model, which it calls weaker than Stable Diffusion (§4.3).

DreamBooth and prior preservation

DreamBooth (Ruiz et al., 2023) fine-tunes the model itself. Each subject image is captioned “a [identifier] [class noun]”, such as “a [V] dog”. The class noun ties the subject to what the model already knows about its class; in an ablation on five subjects, training with no class noun or a wrong one lowered subject fidelity (DINO 0.303 and 0.454 against 0.744, Table 4). The identifier should have a weak prior in both the text encoder and the denoiser. English words such as “unique” must first be unlearned, and random character strings can be split into letters that carry strong priors, so the authors pick rare tokens from the tokenizer’s vocabulary; for Imagen, uniformly sampled T5-XXL tokens with ids from 5,000 to 10,000 that decode to three or fewer characters worked well (§3.2).

The authors found the best subject fidelity by fine-tuning all layers, including those that read the text embeddings (§3.3). Two problems follow. Language drift, a term borrowed from language models fine-tuned for a narrow task, is the model gradually forgetting how to generate other members of the subject’s class, so that “a dog” starts producing the subject dog. Reduced diversity is outputs snapping to the poses and views of the few training images, which the authors often observed, especially when training runs too long. The class-specific prior-preservation loss addresses both: before fine-tuning, the frozen model generates samples $x_{\text{pr}}$ from the class prompt “a [class noun]” with the ancestral sampler, and fine-tuning supervises the model with its own samples alongside the subject images (Eq. 2):

$$\mathcal{L} = \mathcal{L}_{\text{subj}} + \lambda\, \mathbb{E}_{x_{\text{pr}}, \epsilon', t'}\,\big\| \epsilon' - \epsilon_\theta(\sqrt{\bar\alpha_{t'}}\,x_{\text{pr}} + \sqrt{1-\bar\alpha_{t'}}\,\epsilon',\; t',\; \Gamma(\text{“a [class noun]”})) \big\|^2 .$$

About 1,000 iterations with $\lambda = 1$, learning rate $10^{-5}$ for Imagen or $5 \times 10^{-6}$ for Stable Diffusion, 3 to 5 subject images and about 1,000 generated class samples were enough (§3.3). The authors call the loss autogenous: it needs no data beyond the model. Accounts differ on the text encoder. The DreamBooth supplementary says that with Stable Diffusion the U-Net “(and possibly the text encoder)” are trained; Custom Diffusion’s DreamBooth baseline, a third-party implementation, keeps the text transformer frozen and the identifier unoptimized (§4); HyperDreamBooth describes the original as fine-tuning the U-Net and the text encoder (§1).

Custom Diffusion: cross-attention keys and values

Custom Diffusion (Kumari et al., 2023) asks which weights need to change. Fine-tuning all of Stable Diffusion on one concept, they measured the relative change of each layer’s parameters and found it much larger in the cross-attention layers than elsewhere, although those layers are about 5% of the parameters (§3.1, Fig. 3). Because the text features enter the denoiser only through the key and value projections $W^k$ and $W^v$ of cross-attention, they train only these, together with a new modifier token $V_*$ initialized from a rarely used token’s embedding, as in “photo of a $V_*$ dog” (§3.1). Instead of generated images, they regularize with 200 real images from LAION-400M whose captions have CLIP text similarity above 0.85 to the target prompt, and they augment by resizing the subject images to 0.4–1.4 times and adding words such as “very small” or “zoomed in” to the caption (§3.1–3.2). Training took 250 steps at batch size 8 for one concept, and the stored update is 75 MB against 3 GB for a full model (§3.2, §4.1).

Low-rank and other compact updates

Several methods restrict the update to a small parameter space, which shrinks the stored model and, their authors argue, lessens overfitting:

• Low-rank adaptation (LoRA; Hu et al., 2022, not read here), proposed for language models, is described by HyperDreamBooth’s authors as training a residual $\Delta W = AB$ with $A \in \mathbb{R}^{n \times r}$, $B \in \mathbb{R}^{r \times m}$ and $r \ll n$ for frozen weight matrices, usually applied in diffusion models to the cross- and self-attention layers; they report that a rank-1 LoRA DreamBooth of Stable Diffusion 1.5 has about 386,000 parameters, about 1.6 MB (Ruiz et al., 2024, §3).
• SVDiff (Han et al., 2023) decomposes each weight matrix (convolution kernels reshaped to matrices) as $W = U \Sigma V^\top$ once, and trains only a shift $\delta$ of its singular values, $W_\delta = U\, \mathrm{diag}(\mathrm{ReLU}(\sigma + \delta))\, V^\top$, with DreamBooth’s prior-preservation loss (§3.2). The authors report about 2,200 times fewer parameters than full DreamBooth (a 1.7 MB checkpoint for Stable Diffusion) and that, in their single-image-editing comparison with LoRA, the delta checkpoint is 1/2 to 1/3 of LoRA’s size, while LoRA’s adjustable rank may help when much new content must be learned (§1, §4.5).
• Lightweight DreamBooth, part of HyperDreamBooth (Ruiz et al., 2024), factors the rank-1 LoRA matrices further through frozen random orthogonal bases, leaving about 30,000 trainable values, 120 KB (§4.1).

Encoder-based personalization

Optimization per subject takes minutes. Encoder methods instead train, once and on a large dataset, a network that maps subject images directly to conditioning:

• ELITE (Wei et al., 2023) maps CLIP image features from several encoder layers to several word embeddings; the one from the deepest layer becomes the editable subject word, and the others absorb background and other details and are dropped at generation. A local mapping network injects masked patch features through extra cross-attention for detail (§3.2–3.3). It was trained on the 125,000 images of the OpenImages test set and encodes a concept in 0.05 s (§4.1, Table 2).
• InstantBooth (Shi et al., 2024) trains per category (people, cats) in the main paper. A concept encoder turns the images into a global embedding that replaces the placeholder word before the class noun, and a patch encoder feeds 257 CLIP tokens per image into new gated self-attention adapter layers in the frozen U-Net. It needs no images of the same subject in different settings: during training the condition image is the masked, augmented object cut from the target image itself (§3.2). At inference a weight $\beta$ on the adapters and a factor $\alpha$ that rescales the concept embedding trade identity against prompt adherence; the authors chose $\beta = 0.3$, $\alpha = 0.4$ (§3.3; supplementary, Table 1).
• HyperDreamBooth (Ruiz et al., 2024) trains a hypernetwork, a ViT encoder with a transformer decoder, to predict the Lightweight DreamBooth weights from one face image, then refines them with 40 iterations of fine-tuning at a higher LoRA rank (“rank-relaxed fast finetuning”), about 20 seconds per subject, which the authors put at 25 times faster than DreamBooth (§4.2–4.3, abstract).

Several subjects in one image

Composing two personalized subjects (“The $V_1^*$ cat is sitting inside a $V_2^*$ wooden pot”, Kumari et al., 2023, Fig. 8) is harder than one. Custom Diffusion trains the cross-attention weights on both concepts jointly, with one modifier token each, or merges two separately fine-tuned models in closed form: the merged key and value matrices stay as close as possible to the pretrained ones on about 1,000 regularization captions, under the constraint that they reproduce each fine-tuned model’s outputs on its own concept words (§3.2, Eqs. 4–5). The merge takes about 2 seconds. In their tests, sequential training forgot the first concept (§4.2), and difficult compositions such as a dog and a cat together, or objects of similar categories such as a tortoise plushy and a teddy bear, still failed (§5, Fig. 9). SVDiff’s Cut-Mix-Unmix augmentation instead trains on hand-built collages of two subjects with prompts such as “photo of a [V2] sculpture and a [V1] dog”, applied with a probability usually set to 0.6, so that the model learns not to mix the two; when three concepts were trained together, it still failed to separate similar subjects such as a dog and a panda (§3.3, §4.2).

How to Apply It

The papers describe a common workflow, with choices at each step:

1. Choose where the subject will live. A new embedding (textual inversion) leaves the model untouched and is the smallest to store, but is limited by what the frozen model can already draw. Full fine-tuning (DreamBooth) gave the highest subject fidelity in its authors’ tests at the cost of a full model per subject; cross-attention, low-rank or spectral updates store far less and suit composing several subjects; an encoder avoids per-subject training if one exists for the subject’s category.
2. Collect the subject images. DreamBooth used 3 to 5 casually captured images with no restriction on their settings (§3); InstantBooth masks out the background before encoding (§3.2), ELITE’s local mapping reads only the masked subject (§3.3), and Custom Diffusion trains on randomly rescaled copies with matching size words in the caption (§3.2).
3. Write the training caption. For fine-tuning methods, “a [V] [class noun]” with a rare identifier; for textual inversion and ELITE, a pseudo-word $S_*$ inside neutral templates such as “a photo of a $S_*$” (Wei et al., 2023, §3.2).
4. Regularize. With weight updates, add the prior-preservation term on generated class images or a retrieved set of real class images, at the weight and sample counts given above.
5. Evaluate on prompts the training images do not show, for recontextualization, accessories, property changes and styles, scoring subject fidelity, prompt fidelity and, after fine-tuning, what the class prompt alone now generates, since that is where language drift shows (Ruiz et al., 2023, §4.3; supplementary, Fig. 11).

The reported settings are for models with hundreds of millions to billions of parameters on data-centre GPUs; they are starting points from the papers, not values that transfer to other models.

Variants

The methods above differ along three axes, which papers mix freely:

Method	What is trained per subject	Regularization	Per-subject time reported
Textual inversion	One word embedding	None needed: model frozen	20 min on 2 A100s (Custom Diffusion, §4.1)
DreamBooth	All denoiser weights (text encoder in some accounts)	Prior preservation with generated class images	About 5 min on one A100 for Stable Diffusion
Custom Diffusion	Cross-attention $W^k$, $W^v$ and a modifier token	200 retrieved real images	About 6 min on 2 A100s
LoRA, SVDiff, LiDB	Low-rank residuals, singular-value shifts or a smaller subspace	Prior preservation (SVDiff)	—
ELITE, InstantBooth	Nothing: an encoder trained once	—	0.05 s (ELITE), 6 s (InstantBooth)
HyperDreamBooth	A hypernetwork’s prediction, then 40 fine-tuning steps	—	About 20 s

The times come from different papers, hardware and step counts and are not comparable across rows; InstantBooth puts textual inversion at about 1,500 s and DreamBooth at about 600 s per subject in its own set-up (Table 1).

Evaluation

DreamBooth introduced a dataset and protocol that later papers reuse: 30 subjects (21 objects, 9 live subjects or pets), each with a few images collected by the authors or from Unsplash, and 25 prompts per subject (for objects, 20 recontextualization and 5 property-modification prompts; for live subjects, 10 recontextualization, 10 accessorization and 5 property modification). Evaluation generates four images per subject and prompt, 3,000 images in all (§4.1; supplementary). It measures:

• Subject fidelity with DINO, the average pairwise cosine similarity between DINO ViT-S/16 embeddings of generated and real subject images, and CLIP-I, the same with CLIP image embeddings. The authors prefer DINO: CLIP is trained to match captions and may not separate two subjects with near-identical descriptions, whereas DINO’s self-supervised objective does not teach it to ignore such differences. In their data DINO correlated with human preference at 0.32 (Pearson) against 0.27 for CLIP-I (§4.1; supplementary).
• Prompt fidelity with CLIP-T, the average cosine similarity between the prompt’s and the image’s CLIP embeddings (text conditioning discusses what such scores miss).
• In the prior-preservation ablation, PRES, the DINO similarity between images generated from the class prompt and real images of the subject (higher means the class has collapsed toward the subject), and DIV, an LPIPS-based diversity among images of the subject from the same prompt (§4.3).

Other papers vary the protocol: Custom Diffusion measures text and image alignment with CLIP and a KID against 500 retrieved real images of the class to detect forgetting (§4); ELITE uses DreamBooth’s prompts with five images per subject and prompt and 20 subjects drawn from several papers (§4.1); InstantBooth and HyperDreamBooth add face-recognition similarity for people. Scores from different papers are therefore not comparable.

Best Practices

• Use the class noun with the identifier, and choose an identifier with a weak prior (Ruiz et al., 2023, §3.2, Table 4).
• Regularize against drift when weights change: generated class images (DreamBooth) or retrieved real images (Custom Diffusion). Custom Diffusion found that without regularization, or with generated rather than real regularization images, KID on real class images rose from 20.96 to 32.64 and 34.70 (×10³, Table 3).
• Keep training short. DreamBooth’s authors observe snapping to the training views especially when training runs long (§3.3).
• Report the trade-off, not one number. Subject fidelity and prompt fidelity compete (see Evidence and Impact), and InstantBooth’s own inference weights move both in opposite directions (supplementary, Table 1).
• Restrict the update when subjects will be composed: Custom Diffusion’s comparisons favour updating only cross-attention for two concepts (§3.2, §4.2).

Pitfalls

• Language drift and loss of diversity after full fine-tuning without regularization (Ruiz et al., 2023, §3.3; supplementary, Fig. 11).
• Overfitting to the training context, so that prompts close to the original setting reproduce the training images, and context–appearance entanglement, such as a backpack changing colour with the scene (DreamBooth §4.5, Fig. 8).
• Weak priors for rare subjects or contexts: some subjects, such as dogs and cats, are easier to learn than rarer ones, and some prompted contexts are not generated (§4.5).
• Leakage between concepts: training on “moongate” made “photo of a moon” produce moongates when no regularization set was used (Kumari et al., 2023, Fig. 5).
• Metrics that reward copying. In our reading, a method that reproduces a training image scores high on CLIP-I or DINO. InstantBooth’s reconstruction score was lower than DreamBooth’s even though, the authors argue, its outputs varied pose and background while all the compared methods copied the pose, and textual inversion and DreamBooth also the background (§4.3).

Evidence and Impact

On the DreamBooth dataset, DreamBooth scored DINO 0.696, CLIP-I 0.812 and CLIP-T 0.306 with Imagen, and 0.668, 0.803 and 0.305 with Stable Diffusion, against 0.569, 0.780 and 0.255 for textual inversion on Stable Diffusion; real images of the same subject scored 0.774 DINO and 0.885 CLIP-I against each other (Table 1). In two user studies of 1,800 answers each, DreamBooth was preferred over textual inversion for subject fidelity 68% to 22% and for prompt fidelity 81% to 12% (Table 2). The prior-preservation ablation on 15 subjects with Imagen lowered PRES from 0.664 to 0.493 and raised DIV from 0.371 to 0.391, at a small cost in subject fidelity (DINO 0.712 to 0.684), with CLIP-T 0.306 and 0.308 (Table 3).

Other comparisons rank the methods differently, because protocols differ. Custom Diffusion’s single-concept averages, with a third-party DreamBooth, give textual inversion the highest image alignment (0.827, against 0.776 for DreamBooth and 0.775 for Custom Diffusion) but the lowest text alignment (0.670 against 0.781 and 0.795); its authors read this as textual inversion overfitting to the target images (Table 1, Table 2 caption). In ELITE’s comparison, where every method learned each subject from a single image (§4.3), DreamBooth and Custom Diffusion led on CLIP-I (0.785, 0.801) and on DINO, which ELITE calls DINO-I (0.674, 0.695), ELITE on CLIP-T (0.255), and textual inversion trailed on all three (Table 2). InstantBooth reports, for 50 people from the PPR10K dataset, a higher CLIP alignment (0.3140 against 0.3088) and face similarity (0.2456 against 0.2408) than DreamBooth but a lower reconstruction score (Table 1). HyperDreamBooth reports DINO 0.473 against 0.441 for DreamBooth and 0.289 for textual inversion on 100 CelebA-HQ identities with 30 prompts (Table 1).

Risks

DreamBooth’s authors note that malicious parties might use personalized images to mislead viewers, a problem they describe as common to generative and content-manipulation methods, and call for continued study (supplementary, Societal Impact). Van Le et al. (2023) argue more specifically that a model personalized on a few photos of a person can produce fake news or disturbing content targeting that person, and classify this as a form of deepfake. Their Anti-DreamBooth adds small adversarial perturbations to a person’s photos before they are published, so that a DreamBooth model trained on them produces degraded images; they evaluated it on the CelebA-HQ and VGGFace2 face datasets and report that its protection weakens as more unperturbed photos are mixed in (§1, §5.5). The privacy issue also shapes research practice: HyperDreamBooth’s authors show only synthetic faces in their figures “for privacy reasons” (§5). Personalization also starts from the base model’s limits: Custom Diffusion’s authors note that their models inherit the pretrained model’s difficulty with hard compositions (§5).