Will It Render?
Learned Attention Steering for Object Presence in Text-to-Video

Ben Fishman1, Dvir Samuel2, Gal Chechik1,2, Idan Schwartz1
1Bar-Ilan University    2NVIDIA
TL;DR The problem: not all requested objects are rendered by text-to-video models. Our Solution: trainning a tiny model (37k parameters) on the generator's attention maps which predict whether each requested object will render. Then, we use it as a differentiable steering signal at inference time — no model retraining, no hand-crafted objectives needed.

Abstract

Text-to-video diffusion models generate increasingly realistic videos, yet often fail to render all objects requested by the prompt, particularly as the number of objects grows. Improving such fine-grained adherence through training requires substantial video supervision, motivating inference-time methods that steer intermediate diffusion latents. Existing approaches typically define hand-crafted objectives over cross-attention maps, relying on indirect attention statistics rather than whether the model ultimately renders a requested object.

We propose AttnSteer, a learned attention-steering model that predicts object rendering directly from intermediate cross-attention. AttnSteer is trained entirely on the generator's own trajectories using automatically obtained detector labels and contains only 37k parameters. At inference time, its predictions define differentiable objectives for steering object presence; the same framework also extends to frame-level temporal control. AttnSteer improves all-objects accuracy over the strongest baseline at comparable video quality.


Key Results

All-objects accuracy

LTX-2
Wan 2.1
CogVideoX
Un-steered
38.3%
27.1%
30.5%
↓
↓
↓
A&E (baseline)
49.0%
42.1%
56.1%
↓
↓
↓
AttnSteer (ours)
61.9%
46.5%
65.9%

Method

Inference

At inference time, AttnSteer's predictions define differentiable objectives for steering object presence. The video latent is optimized while the generator stays fixed. For multi-object prompts, independent per-object updates can conflict: we filtering out objects which are likely to be rendered, for the remain objects we normalize each ones gradient, and aggragate all objects' gradients while suppress locally opposing contributions.

Inference-time steering with AttnSteer
Inference-time steering. At selected denoising steps (yellow), we run a few optimization steps (purple). In optimization steps we extract object's cross-attention maps and pass them throuh frozen AttnSteer predictor (blue). Then we backpropagate the gradient, together with filtering, normalization and aggragation logic (red pass) to update the latent.

AttnSteer Training

We train a tiny model - AttnSteer to predict whether each requested object will be rendered. We use per-step, per-layer, per-head cross-attention features from the frozen generator. Labels come from detectors run on the generator's own final videos.

Training AttnSteer with self-generated supervision
Training with self-generated supervision. Cross-attention features from the frozen generator are paired with detector labels from its final video.

Steering Process Visualization

Each row shows the predicted clean video (x̂₀) at successive denoising steps for the prompt "a truck and a bird and a cow" steered toward "a bird". The highlighted cell (step 7) is expanded below: the four optimization iterations within that single denoising step show the probe score jumping from 0.61 → 0.99 within two iterations.

Top: x̂₀ previews at selected denoising steps (step 7 highlighted in orange). Bottom: x̂₀ at every other optimization iteration inside step 7.


Quantitative Results

We evaluate over our own benchmark: extended 473 prompts of VBench multiple-objects dimension.

All Objects Accuracy

Wan 2.1 all-object accuracy by k
LTX-2 all-object accuracy by k
CogVideoX all-object accuracy by k
Un-steered Attend-and-Excite Best-of-N Ours

All-object accuracy vs. prompt complexity (k objects). Bands show ±1 SE.

VBench Quality Dimensions

AttnSteer preserves or improves video quality across all VBench dimensions. Each bar shows the per-dimension score for one backbone.

Wan 2.1 VBench quality dimensions
LTX-2 VBench quality dimensions
CogVideoX VBench quality dimensions
Un-steered Attend-and-Excite Best-of-N Ours

Per-dimension VBench scores. Higher is better for all dimensions.

Full Results Table

The table reports the detection ratio and the fraction of clips where all requested objects are detected, overall and by k, the number of requested objects. The gain grows with k.

Model Det. ratio
(% ↑)
All-object accuracy (%) ↑ Quality (%) ↑ Inference cost
meank=234567 VBenchMUSIQ Upd. ↓Time(s) ↓×base ↓
Wan 2.278.938.387.552.535.035.07.59.684.475.4———
LTX-2.562.117.545.016.317.511.38.75.586.973.6———
MiniMax-H384.649.790.055.033.848.736.232.981.564.8———
Wan 2.173.427.180.631.524.322.71.62.179.975.5026.21.00×
 + Best-of-N81.740.493.659.535.928.211.813.280.175.5078.53.00×
 + A&E79.839.283.152.147.627.511.113.779.271.85366.42.54×
 + Ours89.359.990.972.460.954.941.339.382.574.62370.22.68×
LTX-278.938.375.660.335.526.320.012.183.770.6059.81.00×
 + Best-of-N85.951.188.667.549.445.630.425.084.370.70179.43.00×
 + A&E78.337.874.752.633.327.825.712.584.170.555149.22.50×
 + Ours93.975.595.086.784.468.965.752.383.571.330145.52.43×
CogVideoX-5B73.130.580.054.120.313.48.66.579.566.60135.01.00×
 + Best-of-N82.243.992.473.440.527.021.18.879.367.40404.93.00×
 + A&E87.356.192.469.752.949.343.328.879.165.654358.52.66×
 + Ours91.265.994.981.864.461.157.136.278.470.613278.92.07×

Bold = best within backbone block; underline = best overall. Inference cost measured on a single H100 GPU; ×base = multiple of un-steered generation time.


User Study

To confirm the improvement is visible to people, we ran a paired human-preference study on LTX-2 and Wan 2.1, comparing AttnSteer against Attend-and-Excite on object presence, photorealism, and motion smoothness. 37 volunteer annotators completed 622 clip judgements. AttnSteer is preferred on every question and both backbones. The largest margin is object presence on Wan 2.1 (73.9% of decisive votes); photorealism and motion smoothness — included as quality guardrails — also come out in our favor (58–67%).

LTX-2 user study win rates vs Attend-and-Excite
Wan 2.1 user study win rates vs Attend-and-Excite
Ours About the same Attend-and-Excite

Bars show share of all votes; percentage at right = decisive-vote win rate for Ours.


Single-Object Guardrail

We evaluate on 79 VBench object_class single-object prompts as a guardrail: with only one requested object, backbones already succeed on most clips, and the goal is to verify that steering does not harm this easy regime. AttnSteer in fact improves single-object presence on LTX-2 under both detectors, while leaving VBench quality unchanged.

Metric CogVideoX-5B Wan 2.1 LTX-2
Base+A&E+Ours Base+A&E+Ours Base+A&E+Ours
Presence 3-det. (%) ↑ 92.390.996.2 92.287.288.3 96.291.196.2
Presence GRiT (%) ↑ 81.072.282.3 81.081.081.0 88.678.589.9
VBench (%) ↑ 78.375.475.9 77.076.079.7 82.982.181.9

Bold = best within backbone. Underline = best across all backbones.

Qualitative examples


Temporal Control

The same learned objective applies frame by frame, so it can control when an object appears. On single-object prompts, where an object must enter at a prescribed time, AttnSteer raises specificity from 31.2% to 52.6%, meaning fewer frames before the requested onset are wrongly filled. Temporal alignment (t-IoU) rises from 40.6% to 45.0%, and VBench quality is unchanged (74.7 vs. 75.1).

Model Single object (n=80) Static + dynamic (n=82)
PrecisionRecallSpec.F1t-IoUR@0.5VBench PrecisionRecallSpec.F1t-IoUR@0.5VBench
LTX-255.978.931.258.640.637.5—86.477.154.183.067.478.7—
 + TempoControl57.780.935.760.541.837.5—87.180.553.783.170.881.7—
 + AttnSteer63.868.152.662.545.045.0—87.284.152.285.574.585.4—

Temporal control examples

TempoControl (left) vs. AttnSteer (right), played in sync. The timeline shows the requested control signal and the actual per-frame detections for each method.

Car
Requested
TempoControl
AttnSteer (ours)
Fire hydrant
Requested
TempoControl
AttnSteer (ours)
static Boat + controlled Airplane
Requested
TempoControl
AttnSteer (ours)
Giraffe
Requested
TempoControl
AttnSteer (ours)
static Bicycle + controlled Car
Requested
TempoControl
AttnSteer (ours)
Teddy bear
Requested
TempoControl
AttnSteer (ours)

Dataset

We extend VBench's multiple_objects dimension from k=2 (80 prompts) up to k=7, totaling 473 prompts.

473

prompts

2–7

objects / prompt

76

classes

73–80

prompts / k

VBench's template

"a X and a Y and a Z …"  (examples) "a kite and a baseball bat"
"a pizza and a surfboard and a kite"
"a tie and a book and a dog and a frisbee and a fork"
"a bicycle and a vase and a cup and a toaster and a microwave and a horse"

Random k objects

drawn from the same 76-class list

Pairwise-unique

no object pair co-occurs in two prompts


BibTeX

@inproceedings{anonymous2027willitrender,
  title     = {Will It Render? Learned Attention Steering for Object Presence in Text-to-Video},
  author    = {TODO},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027}
}