Tuning-Free Noise Rectification for High Fidelity Image-to-Video Generation

Abstract

Image-to-video (I2V) generation tasks always suffer from keeping high fidelity in the open domains. Traditional image animation techniques primarily focus on specific domains such as faces or human poses, making them difficult to generalize to open domains. Several recent I2V frameworks based on diffusion models can generate dynamic content for open domain images but fail to maintain fidelity. We found that two main factors of low fidelity are the loss of image details and the noise prediction biases during the denoising process. To this end, we propose an effective method that can be applied to mainstream video diffusion models. This method achieves high fidelity based on supplementing more precise image information and noise rectification. Specifically, given a specified image, our method first adds noise to the input image latent to keep more details, then denoises the noisy latent with proper rectification to alleviate the noise prediction biases. Our method is tuning-free and plug-and-play. The experimental results demonstrate the effectiveness of our approach in improving the fidelity of generated videos. For more image-to-video generated results, please refer to the project website: https://noise-rectification.github.io/.

1 Introduction

With the remarkable breakthroughs of diffusion models in generating exquisite images [7, 37, 36, 40, 39], researchers are exploring the further potential of diffusion models to achieve more coherent video generation. Some recent works [16, 43, 15, 49, 51] have made incremental progress in the text-to-video (T2V) task to generate videos that align with the input text. However, a textual description can correspond to various imaginable videos, which may not necessarily meet people’s specific expectations. Therefore, the reference image is proposed to guide the video generation process, aiming to generate videos that closely align with the given image or even strictly start from the still image, which is called the image-to-video (I2V) task.

The concept of image-to-video is not novel and has long existed in traditional computer vision tasks, such as facial animation [50, 42], body motion synthesis [33], nature-driven animation [28, 17], and video prediction [24, 9, 18], which can all be considered as I2V tasks. However, these tasks were either limited to specific domains (such as faces, human poses, and simple natural scenes) or focused on relatively simple scenarios (such as animating fonts [9], drawing [44] or moving rigid objects [19]). The proposed solutions for these specific tasks are difficult to be applied to open-domain images. Moreover, previous studies [28, 53, 19, 18, 48, 23] adopted the autoregressive approach to generate the video sequences, which is computationally expensive and still faces challenges in complex open-domain scenarios. Recently, the emerged diffusion models have demonstrated strong generative capabilities and significant extensibility by learning the data distribution from noise. As a video can be considered as a temporal sequence (batch) of highly correlated images, it is feasible to process videos in batches using diffusion models. Consequently, there is a growing focus on leveraging diffusion models for image-to-video task, attracting significant attention from both research and industry.

However, current I2V research [11, 4, 51, 59] primarily relies on enhancing the supervision of image signals to guide the video diffusion model. As a result, the generated videos are only able to resemble the given image. In our view, existing video diffusion models in these works already exhibit strong capabilities for generating dynamic motions, but they struggle to maintain fidelity, which can be attributed to two main factors. One is the loss of image details, such as adopting IP-Adapter [54] or ControlNet [58] only extracts partial image representation. Another is the noise prediction biases during the denoise process, due to the unattainable perfect zero loss in training the video diffusion model, even when the entire image information has been injected or concatenated. Inspired by the transition refinement with the pivotal noise vector in recent image editing work [6, 29, 30], we propose to set the direction of initial noise as the pivotal reference in the denoising process. Specifically, we design an image-to-video pipeline, which adopts the “noising and rectified denoising” reverse process to improve the fidelity of generated video. Our method utilizes pre-trained video latent diffusion models (VLDM) to generate fluent motion between frames. During the inference, we first add initial noise to the given image to alleviate the loss of detail information, denoted as “noising” stage. Then we properly rectify the predicted noise using the pivotal reference noise in the reverse denoise timesteps to alleviate the noise prediction biases, denote as “rectified-denoising” stage. Additionally, in order to control the retention degree of the reference image, we further introduce a practical step-adaptive intervention strategy based on noise rectification.

In general, we propose an effective method to utilize the existing pre-trained video diffusion models for image-to-video tasks. The comparison experiments with current public I2V works and several I2V attempts in the active community have demonstrated the effectiveness of our methods in generating videos with higher fidelity. Moreover, our method does not require extra training and is simple to implement, which can be seamlessly integrated with current pre-trained open-domain video diffusion models in a plug-and-play manner, enabling high-fidelity I2V generation in open domains.

2 Related Work

The diffusion models have achieved great success in generative tasks in recent years. Due to the high correlation between image and video modalities, many ideas and insights in current video generation work have been inspired by extensive image generation work. Therefore, we introduce the related work on image and video generation here.

2.1 Image Generation with Diffusion Model

Compared to the traditional GAN [10] and VAE [22] based methods, the diffusion models  [45, 14, 7, 47, 46, 36, 40] have demonstrated more powerful capabilities to produce high-quality images with realistic textures and fine details. The U-Net [38] with the attention layer is the widely adopted structure in image diffusion models to predict noise. To save computation costs, Stable Diffusion (SD) [37] proposed the latent diffusion model (LDM), which utilized VAE [22] to encode the image into a latent space and perform the diffusion process on the latent space. To enhance the controllability and support various control conditions such as depth, reference image, normal map and canny map, ControlNet [58] and T2I-Adapter [31] introduced a flexible adapter based on the SD [37] for controlled generation. Recently, IP-Adapter [54] also proposed an image prompt adapter for T2I models to guide the image generation with the reference image. Besides, SDEdit [29] added noise to the input stroke image and progressively denoised the resulting image to increase the realism of the synthesized image. In our image-to-video work, we adopt an inflated 3D U-Net similar to the T2I task and our noise rectification also takes inspiration from the transition refinement in the image editing works [6, 29, 30].

2.2 Video Generation with Diffusion Model

Thanks to the significant progress of text-to-image generation, video generation has also started to develop from the text-to-video (T2V) task. VDM [16] introduced a pioneering video diffusion model that extends the 2D U-Net to a 3D U-Net structure, jointly training both images and videos in the pixel space. Subsequent methods [60, 13, 49, 1, 12, 57, 3, 52, 26] mostly adopted the latent space to reduce memory requirements and speed up the training and inference. To optimize the running time required for video generation, most works (Make-A-Video [43], ModelScopeT2V [49], Latent-Shift [1], AnimateDiff [12]) were built upon the pre-trained T2I models and incorporated temporal modules, enabling batch generation of all video frames simultaneously. Particularly, AnimateDiff [12] only trains a motion module that can be adapted to various personalized T2I models. Text2Video-Zero [21] proposed a training-free sampling method to enrich motion dynamics and maintain temporal consistency with cross-frame attention. Besides, the cascade framework of video diffusion models is also used to generate high-resolution [15, 3, 52] and longer videos [13, 56].

Similar to image generation, introducing more control conditions in video generation is also crucial. Recently, to make the generated videos more controllable, recent work has introduced various conditions into the video diffusion models, including depth [43, 8], pose [27, 20], guided motion from trajectory [55], stroke painting [5] or frequency analysis  [25]. As to the image condition, existing video generation work mainly draws on experiences from the image generation field, e.g., enhancing the image guidance using ControlNet [58] and IP-Adapter [54]. Besides, Seer [11] concatenated the conditional image latent with the noisy latent and employed causal attention in the temporal module of 3D U-Net for image-to-video tasks. VideoComposer [51] proposed to concatenate the image embedding with the noisy latent along the feature channel, as well as support forwarding the style of the given image into the video latent diffusion model (VLDM). Recently, VideoCrafter [4] extracted the image feature into the VLDM for the image-to-video task. Similarly, I2VGen-XL [59] both added the image latent with the noisy latent in the input layer and built a global encoder to extract the image CLIP feature into the VLDM. However, these image-to-video works either have limited fidelity or require fine-tuning the whole VLDM. In comparison, our noise rectification method is tuning-free and maintains high fidelity.

3 Preliminary

Figure 1: The general framework of image diffusion model and video diffusion model with inflated 3D U-Net structure.

3.1 Image-to-Video Task Definition

All video generation tasks require generating coherent frames that maintain both visual consistency and logical motion. Specifically, image-to-video (I2V) task is defined as generating a video from a specified reference image. Its goal is to transform the static nature of an image into the dynamic visual representation, adding motion and fluidity to the content. Compared to the text-to-video (T2V) task, I2V prioritizes high fidelity with the conditional image, while dynamic motion in the video can be learned through common prior knowledge or driven by the given conditions like the text description or other data forms. Here we focus on the text-conditioned image-to-video task, and this definition can be formulated as, given a still image $I$ and a text description $c$, the generative system outputs a predicted video ${V}^{0:L-1}=\left\{\bar{I}^{0},\dots,\bar{I}^{L-1}\right\}$, where $L$ represents the video length. The objective is to keep appearance consistent with the given initial image $I$, as well as ensure the generated video aligns with the text description $c$.

3.2 Video Latent Diffusion Models

The diffusion models [45, 14, 47, 46] are a class of generative models inspired by non-equilibrium thermodynamics, which define the Markov chain to perturb data to noise in the diffusion process and then learn to convert noise back to data in the reverse process. Formally, in the diffusion process, given a data distribution $z_{0}\sim q(z_{0})$, the forward Markov chain gradually adds Gaussian noise to the data sample in the $T$ timesteps, thus obtaining a sequence of noisy data $\left\{z_{1},z_{2},\dots,z_{T}\right\}$ conditioned on $z_{0}$, following the transition formula, which can be denoted as:

$$q(z_{1:T}|z_{0}) =\prod\limits_{t=1}^{T}q(z_{t}|z_{t-1}),$$ (1)

$$q(z_{t}|z_{t-1}) =\mathcal{N}(z_{t};\sqrt{\alpha_{t}}z_{t-1},(1-\alpha_{t})\mathbf{I}),$$ (2)

where $\left\{\alpha_{t}\in(0,1)\right\}^{T}_{t=1}$ is a variance schedule to control the step size. In the reverse process, a model $p_{\theta}$ is learned to denoise from the noisy prior $z_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ to gradually generate the desired data iteratively following:

$$p_{\theta}(z_{t-1}|z_{t})=\mathcal{N}(z_{t-1};\mu_{\theta}(z_{t},t),\Sigma_{\theta}(z_{t},t)),$$ (3)

where $\theta$ is the model parameters, $\mu_{\theta}(z_{t},t)$ and $\Sigma_{\theta}(z_{t},t)$ denote the predicted mean and variance by the model.

In the image generative tasks, the denoising model is usually designed as the U-Net network architecture and learned with the objective function

$$\min\limits_{\theta}\mathbb{E}_{z_{0}\sim p_{data},t,\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})}[{\|\epsilon-\epsilon_{\theta}(z_{t},c,t)\|}^{2}_{2}],$$ (4)

where $\epsilon$ and $\epsilon_{\theta}$ are the actual and predicted noise respectively, $c$ represents various conditions like text, image, or other control signals. Furthermore, to reduce the computational complexity, diffusion models are utilized in a lower-dimensional latent space rather than the pixel space, which is denoted as the latent diffusion model [37].

Figure 2: Two basic approaches in existing research and community regarding image-to-video generation.

Figure 3: The framework of our tuning-free image-to-video method. (a) represents the inference pipeline, where the input image is noised into the initial latent and the predicted noise of the inflated 3D U-Net will be rectified during the denoising process. (b) illustrates the detailed generation process of our method. The visualization of intermediate steps shows our method can effectively refine the denoising direction, making intermediate results closer to the given image.

Similar to image diffusion generation, video diffusion generation can be regarded as dealing with a batch of images together (see Fig.1). Recently, video latent diffusion models (VLDM) were also developed upon the text-to-image generation and followed the aforementioned diffusion process, aiming to model the video data from Gaussian noise. Formally, a given video data $V^{0:L-1}\in\mathbb{R}^{L\times 3\times H\times W}$ will be converted to the latent representation $z_{0}^{0:L-1}\in\mathbb{R}^{L\times C\times H^{{}^{\prime}}\times W^{{}^{\prime}}}$ through a VAE encoder [22], where $C$ is the number of feature channels. Besides, due to the temporal consistency and content relevance requirements in the video, the VLDM often involves the additional temporal module [16, 49, 1, 12], thus inflating the denoising model from 2D U-Net to the 3D U-Net. Through the diffusion process $z_{t}^{0:L-1}=q(z_{0}^{0:L-1},t)$ and reverse process $z_{t-1}^{0:L-1}=p_{\theta}(z_{t}^{0:L-1},t)$, the finally denoised video latent $\bar{z}_{0}^{0:L-1}$ will be processed via the VAE decoder to generate the video.

Inspired by the mainstream text-to-video framework, to generate a video from a still image, we also model the video motion with temporal attention in the inflated 3D U-Net [12]. As shown in Fig. 1(b), to improve the computation efficiency, the video’s frame dimension is treated as the batch axis in the forward of the spatial modules, and the video’s spatial dimensions are treated as the batch axis in the forward of the temporal modules.

4 Method

4.1 Enhance Image Condition Analysis

Although the text-to-video framework can generate a video clip with relatively coherent motion, the semantic content of the generated video is mainly aligned with the given text description at a coarse-grained level. To control the content consistency between the generated video and the reference image, the mainstream I2V works in existing research and the community can be summarized into two basic types (see Fig.2): One is to incorporate the image condition at the beginning of the reverse process. This approach is mainly inspired by the image generation field like the img2img tasks, such as the image editing task [6, 29, 30], whose idea is to inject the image latent into the initial noise latent. In this way, the reverse denoising process could be implicitly guided towards the direction of the image latent in the latent space. However, this approach can only achieve a resemblance to the given image and there is still a certain gap to high fidelity. A different method involves concatenating the full clean image with the initial noise to introduce more fine-grained details [11, 51, 59]. While this approach improves fidelity, the entire generation framework must be retrained, leading to low scalability and challenges in integrating with existing pre-trained modules like ControlNet [58]. Another method to enhance image fidelity introduces more image feature signals and conditions into the internal computation of the diffusion model at each timestep [4], such as using various ControlNets [58] and IP-Adapter [54]. The image features act as strong supervision to improve the fidelity. However, since feature extraction inevitably loses image details, these approaches tend to learn the overall style or general layout from the original image, making it difficult to achieve high fidelity in terms of fine details.

All the above methods aim to enhance the guidance and control of the initial image in video generation to improve fidelity. However, as shown in Fig.3(b), the denoising process (represented in gray arrow) can not restore the given image even when the initial noisy latents are obtained by adding noise to the given image (represented in dashed blue arrow), we analyzed that the reason why these methods [4, 51, 59, 11] fail to achieve perfect fidelity lies in the accumulated noise biases during the denoising process, causing the generated frame latents to deviate from the given image latent. In the training process, although the MSE loss function is utilized to make the predicted noise close to the initial input noise, the training process cannot completely achieve a perfect loss of 0. Therefore, there will always be a discrepancy between the predicted noise and the true noise. To further improve fidelity, we draw inspiration from the noise latent and aim to alleviate the noise gap during the denoising process.

4.2 Noise Rectification Strategy

Our method includes the “noising and rectified denoising” process. Similar to [29], our approach starts by injecting the image latent into the initial noise. Without introducing any additional operations, such a setting could generate a coherent video that resembles the given image in the whole style and layout. Taking a different perspective, if the denoising process adopts the known initial noise rather than the predicted biased noise at each timestep, it would result in a video sequence that is entirely faithful but also lacks any motion or dynamics. Therefore, to strike a balance between complete fidelity and dynamics, we propose a noise rectification method. The pipeline of our inference process is shown in Fig.3(a), in some intermediate steps of the denoising process, we rectify the predicted noises by adaptively compensating them with the known initial noise, which is formulated as

$${\color[rgb]{0.5234375,0.7421875,0.52734375}\widetilde{\color[rgb]{0.97265625,0.58984375,0.55859375}\widetilde{\color[rgb]{0,0,0}n}}}_{t}^{0:L-1}=Rectify({\color[rgb]{0.97265625,0.58984375,0.55859375}\widetilde{\color[rgb]{0,0,0}n}}_{t}^{0:L-1},n^{0:L-1},t,\omega^{0:L-1},\tau),$$ (5)

where ${\color[rgb]{0.5234375,0.7421875,0.52734375}\widetilde{\color[rgb]{0.97265625,0.58984375,0.55859375}\widetilde{\color[rgb]{0,0,0}n}}}_{t}^{0:L-1}$ denotes the rectified noise at $t^{th}$ timestep, ${\color[rgb]{0.97265625,0.58984375,0.55859375}\widetilde{\color[rgb]{0,0,0}n}}_{t}^{0:L-1}$ denotes the predicted noise of 3D-UNet, $n^{0:L-1}$ denotes the initial sampled noise that is added to a given image, $\omega^{0:L-1}$ and $\tau$ denote the rectification weight and timestep period.

Algorithm 1 Noise Rectification for Image-to-Video

0:  The given image latent $z^{0}$, optional text embedding $c$, video length $L$, rectification weight $\omega^{0:L-1}$ and timestep period $\tau$.

0:  The generated video latent $z_{0}^{0:L-1}$.

1:  $n^{0:L-1}\sim\mathcal{N}(0,\mathbf{I})$

2:  $z_{T}^{0:L-1}\leftarrow AddNoise(Repeat(z^{0}),n^{0:L-1},T)$

3:  for $t=T,…,1$ do

4:     Predict noise ${\color[rgb]{0.97265625,0.58984375,0.55859375}\widetilde{\color[rgb]{0,0,0}n}}_{t}^{0:L-1}=\epsilon_{\theta}(z_{t}^{0:L-1},c,t)$

5:     Compute noise gap $\Delta^{0:L-1}_{t}={n^{0:L-1}-{\color[rgb]{0.97265625,0.58984375,0.55859375}\widetilde{\color[rgb]{0,0,0}n}}_{t}^{0:L-1}}$

6:     if $t$ in $\tau$ then

7:        Rectify ${\color[rgb]{0.5234375,0.7421875,0.52734375}\widetilde{\color[rgb]{0.97265625,0.58984375,0.55859375}\widetilde{\color[rgb]{0,0,0}n}}}_{t}^{0:L-1}={\color[rgb]{0.97265625,0.58984375,0.55859375}\widetilde{\color[rgb]{0,0,0}n}}_{t}^{0:L-1}+\omega^{0:L-1}\cdot Repeat(\Delta^{0}_{t})$ $\hskip 31.2982pt+(1-\omega^{0:L-1})\cdot\Delta^{0:L-1}_{t}$

8:     else

9:        ${\color[rgb]{0.5234375,0.7421875,0.52734375}\widetilde{\color[rgb]{0.97265625,0.58984375,0.55859375}\widetilde{\color[rgb]{0,0,0}n}}}_{t}^{0:L-1}={\color[rgb]{0.97265625,0.58984375,0.55859375}\widetilde{\color[rgb]{0,0,0}n}}_{t}^{0:L-1}$

10:     end if

11:     $z_{t-1}^{0:L-1}\leftarrow Sample(z_{t}^{0:L-1},{\color[rgb]{0.5234375,0.7421875,0.52734375}\widetilde{\color[rgb]{0.97265625,0.58984375,0.55859375}\widetilde{\color[rgb]{0,0,0}n}}}_{t}^{0:L-1})$

12:  end for

13:  return  $z_{0}^{0:L-1}$

Concretely, in our noise rectification strategy, the noise ${\color[rgb]{0.97265625,0.58984375,0.55859375}\widetilde{\color[rgb]{0,0,0}n}}_{t}^{0:L-1}$ predicted by U-Net at each step $t$ is first obtained:

$${\color[rgb]{0.97265625,0.58984375,0.55859375}\widetilde{\color[rgb]{0,0,0}n}}_{t}^{0:L-1}=\epsilon_{\theta}(z_{t}^{0:L-1},c,t),$$ (6)

where $z_{t}^{0:L-1}$ is the input latent map at step $t$ and $\epsilon_{\theta}(\cdot)$ is the denoise model (an inflated 3D U-Net). $c$ and $L$ are the text embedding and video length respectively. Then, we can naturally calculate the noise gap (dubbed $\Delta^{0:L-1}_{t}$) between the initial sampled noise in our noising process and the noise predicted during the denoising process.

$$\Delta^{0:L-1}_{t}={n^{0:L-1}-{\color[rgb]{0.97265625,0.58984375,0.55859375}\widetilde{\color[rgb]{0,0,0}n}}_{t}^{0:L-1}}.$$ (7)

We further calibrate the predicted biased noise, which is the key procedure of our method. By introducing the rectification weight factor $\omega^{0:L-1}$, we balance the first frame noise gap and the subsequent frames’ noise gap to obtain the weighted rectification offset, which is then used to frame-wise update the originally predicted noise.

$${\color[rgb]{0.5234375,0.7421875,0.52734375}\widetilde{\color[rgb]{0.97265625,0.58984375,0.55859375}\widetilde{\color[rgb]{0,0,0}n}}}_{t}^{0:L-1} ={\color[rgb]{0.97265625,0.58984375,0.55859375}\widetilde{\color[rgb]{0,0,0}n}}_{t}^{0:L-1}+\omega^{0:L-1}\cdot Repeat(\Delta^{0}_{t})$$ (8)

$$+(1-\omega^{0:L-1})\cdot\Delta^{0:L-1}_{t},$$

where $Repeat(\cdot)$ is the broadcasting operation to align the temporal dimension.

Figure 4: Visual comparison with current image-to-video methods. We use AnimateDiff [12] as the VLDM. “Ctrl.R.O.” means ControlNet Reference-Only method [58]. Our method achieves higher fidelity in the video sequences with the given image.

The whole process of our image-to-video method is detailed in the Algorithm 1. Such a noise rectification method is simple but effective. As illustrated in Fig.3(b), through noise rectification (represented in the green arrow), the accumulation noise gap could be effectively alleviated and thus the noisy latent of generated frames are closer to the image latent. In this way, the fine-grained content details of the reference image can be well preserved in the generated video. In addition, to control the retention degree of the reference image, we further introduce a step-adaptive intervention strategy based on noise rectification. Specifically, by adjusting the parameter of rectification steps $\tau$ and weight $\omega^{0:L-1}$, our method could control the fidelity degree of the generated video. It is worth mentioning that our method is tuning-free and can be applied to most current video diffusion models.

5 Experiments

5.1 Experimental Setup

Dataset. We utilized two public datasets WebVid10M [2] and LAION-Aesthetic in  [41] to evaluate our method. As for the WebVid10M dataset, we randomly sampled 1000 video-text pairs in proportion to the different categories in its validation subset. For quantitative evaluation, to avoid buffering frames at the beginning of the videos, we selected the 10th frame as the image input for the video generation, along with the video’s text description. As for the LAION-Aesthetic dataset, we also randomly chose 1000 image-text pairs for the qualitative evaluation.

Methods	Image fidelity$\uparrow$	Temporal coherence$\uparrow$	Video-text alignment$\uparrow$
VLDM [12] + SDEdit [29]	0.7425	0.9888	0.2548
VLDM [12] + ConcateImage	0.6944	0.9427	0.2084
VLDM [12] + Ctrl.R.O. [58]	0.7689	0.9919	0.2466
VLDM [12] + IP-Adapter [54]	0.7650	0.9918	0.2287
VideoComposer [51]	0.7483	0.9352	0.2447
VideoCrafter1-I2V [4]	0.7695	0.9689	0.2206
I2VGen-XL [59]	0.7717	0.9560	0.2208
Ours	0.7907	0.9882	0.2517
Ours + IP-Adapter [54]	0.8042	0.9934	0.2405

Table 1: Quantitative comparison results on the WebVid dataset [2].

Evaluation Metrics. For the image-to-video generation task, the focus lies on the fidelity and smoothness of the generated videos. Therefore, we assess the generated video from three aspects: image fidelity, temporal coherence, and video-text alignment. Specifically, to evaluate the fidelity between the generated video and the given image, we calculate the CLIP [35] image similarity for each generated frame and the given image. Considering the temporal consistency in the video, we evaluate the CLIP score between the generated frames. Besides, since the text description is input as a condition, we also calculate the CLIP text-image similarity to evaluate the semantic relevance between the generated video and the text description.

5.2 Comparisons

Comparison Methods. We categorized the comparison methods into these two types as Fig.2. One is to incorporate the image condition into the input layer. (1) SDEdit [29], a semantic image translation method, which can also be used for I2V tasks by simply adding noises to the given image and then denoising. (2) ConcateImage, another simple baseline to concatenate the image condition on initialization noises, which needs to be finetuned to learn the structural information of the given image. Another type of approach is to perform image condition injection at each layer of VLDM. (3) ControlNet Reference-Only [58], an effective way to directly link the attention layer of the VLDM to the reference image. (4) IP-Adapter [54], using an additional cross-attention layer for image prompts to achieve semantic and structural preservation. (5) VideoCrafter1-I2V [4], similar to IP-Adapter, is another implementation of image prompts injection into the VLDM. Besides, (6) VideoComposer [51] and (7) I2VGen-XL [59] combine the above two types of ideas for image injection both at the input and middle layers of VLDM.

Benefiting from plug-and-play and tuning-free properties, our method can combine with other image-condition enhancing modules mentioned above. In order to make an intuitive and fair comparison, we conduct our method on both the two above types, denoted as Ours and Ours+IP-Adapter [54]. For fairness, we select AnimateDiff [12] as the pre-trained VLDM.

Figure 5: Ablation study on the weight of noise rectification. We fix the rectification timestep $\tau$ and change the rectification weights $\omega$ for different frames.

Qualitative Comparison. As shown in Fig.4, the method [29] and ConcateImage which only incorporate the image condition with the noisy latent at the beginning of the reverse stage are only able to maintain a similar style of the given image. In contrast, those methods [58, 54, 51, 4, 59] that iteratively utilize the image information in the model’s intermediate computation process can preserve more visual features of the given image. In comparison, our method maintains more visual details and achieves high fidelity to the input image. For clearer video samples please refer to the project website.

Figure 6: Ablation study on the timestep of noise rectification. We fix the rectification weight $\omega$ and the green panels show the the rectification start and end timesteps $\tau$.

Figure 7: A plug-and-play extension of our method on current T2V frameworks to realize I2V. (a) Text-to-video generation results for different T2V models. (b) Different T2V frameworks combined with our method for high-fidelity image-to-video generation.

Quantitative Comparison. As shown in Tab.1, our noise rectification method effectively improves the fidelity. Combined with the additional image prompt module [54], our method can obtain the highest video-image fidelity value of 0.8042 and temporal coherence value of 0.9934. Besides, our method still obtains acceptable video-text alignment although we mainly focus on the high fidelity image-to-video task.

5.3 Ablation Study

Our method rectifies the predicted noise in the reverse steps and contains two adjustable parameters: rectification weight $\omega$ and rectification timestep $\tau$ as introduced in Algorithm 1. Therefore, we take the ablation study on these two parameters, respectively. Specifically, $\tau=[s,e]$ indicates that noise rectification is performed from $s$ ratio to $e$ ratio of the total timestep.

Rectification Weight. We fix the rectification timestep $\tau=[0\%,60\%]$ and change the rectification weights $\omega$ for different frames. The ablation results on $\omega^{i}$ are shown in Fig.5, where the plots above the video sequences indicate the rectification weights $\omega^{i}$ for the $i^{th}$ frame. It can be observed that $\omega^{i}$ could affect the fidelity and temporal consistency of subsequent frames. For example, the results in the third or fourth column may result in abrupt changes in the image detail or motion effects. Therefore, we empirically select the setting of the second column for maintaining high fidelity and natural motion.

Rectification Timestep. The rectification timestep period $\tau$ determines in which denoising steps the predicted noise needs to be corrected. As shown in Fig.6, we fix the rectification weights $\omega$ and the green panels show the rectification start and end timestep. If noise rectification is not performed, i.e. the first column $\tau=[0\%,0\%]$, the fidelity of the generated video will be poor. Starting from the initial denoising, as the noise rectification period increases (from $\tau=[0\%,20\%]$ to $\tau=[0\%,100\%]$), the fidelity will gradually be improved. However, if the rectification only happens on the latter denoising process (i.e.,$\tau=[30\%,70\%]$ or $\tau=[60\%,100\%]$), the generated video will still get poor fidelity. These results indicate that accurately predicting noise at the start of the reverse process is crucial for maintaining image fidelity. Considering that a perfect fidelity will scarify the motion intensity, we strike a balance to set $\tau=[0\%,60\%]$ for all experiments.

Extension to More VLDMs. Our method utilizes the motion prior of VLDM to model the dynamic motion, which is actually tuning-free and can be adapted to other video diffusion models. To evaluate the extension performance of our method, we selected several recent T2V models and applied our noise rectification method to implement I2V. Besides AnimateDiff [12], ModelScopeT2V [49] is a diffusion-based text-to-video model that utilizes the spatio-temporal block to model dynamics. Hotshot-XL [32] is an open-sourced text-to-GIF model developed to work alongside the Stable Diffusion XL (SD-XL) model [34]. We evaluate these three T2V models and extend them to I2V using our plug-and-play noise rectification method. As shown in Fig.7, based on pre-trained motion priors, our method can maintain high fidelity and consistent animation.

6 Conclusion

In this work, we propose a simple but effective noise rectification method for image-to-video generation in open domains. We deeply analyze the challenges in I2V and propose a tuning-free approach to ensure high fidelity through a noising and rectified denoising process. Our method is plug-and-play and can be applied to other video latent diffusion models to realize I2V. Experimental results demonstrate the effectiveness of our method. We hope that our method provides a new idea to improve fidelity in the video synthesis field. Notably, our method achieves higher fidelity while losing some motion intensity. Therefore, in future exploration, we will continue to focus on increasing the motion intensity while maintaining high fidelity.