SparSTAR: Sparse Attention for SpaceTime AutoRegressive Video Synthesis

Preprint
Jongbeom Lee1*, Hyunwoo Yu1*, Jincheol Yang1, Jaemin Choi1, Suk-Ju Kang1†
1Sogang University
*Equal contribution †Corresponding author
Paper arXiv Code (soon)

SparSTAR Gallery

Cross-clip sparse pattern reuse versus SparSTAR dynamic masks

Core idea. Sparse support transferred from an earlier scale or clip can miss newly active blocks. SparSTAR instead builds a fresh per-head mask at each sparsified scale with low selector overhead.

Abstract

InfinityStar extends visual autoregressive generation to video through a sequence of image and clip pyramids. Its changing scale and cross-clip context, however, leave late-stage attention costly and make sparse patterns reused from diffusion or image VAR models unreliable. We introduce SparSTAR, a training-free block-sparse attention method tailored to this setting. At each expensive scale and attention head, SparSTAR scores contiguous key blocks from the current query and key activations, retains required conditioning context, and executes the selected blocks through a forward-only sparse path. We analyze cross-scale consistency within a clip, pattern persistence across clip boundaries, and quality degradation as reuse spans increasingly distant scales. Across these analyses, important key blocks shift, showing that recomputing block selection at each target scale is more reliable than reusing a transferred mask. On 720p text-to-video and image-to-video generation, SparSTAR preserves every token and refinement scale while providing about a 1.6× end-to-end speedup and maintaining VBench and paired-output reconstruction fidelity close to dense InfinityStar.

Analysis-driven design

Three observations motivate recomputing the sparse mask at every expensive scale rather than transferring it from an earlier scale or clip.

Walkthrough of the analysis (1:06). Cross-scale pattern reuse, the clip boundary, and how often the mask has to be recomputed — the three findings detailed below.

Dense-attention patterns across clips and scales

Dense-attention patterns across clips and scales in a 480p two-clip T2V trace. Attention concentrates on block-structured regions, but which blocks are active changes at clip boundaries (left) and between refinement scales (right).

Retained attention mass across scales

Patterns shift across scales

Cross-scale reuse loses substantial attention mass, while freshly recomputed block-wise selection stays close to the token-wise upper bound.

Fresh versus reused selection across clip boundaries

Only partial persistence across clips

At a 15% preceding-clip context budget, fresh Clip-2 selection retains 85% of attention mass versus 56% for reusing Clip-1 support.

PSNR latency trade-off for recalibration frequency

Recalibration is worth the overhead

Updating every scale recovers 1.73 dB over single transfer for only 0.30 seconds of additional latency at 720p.

Fresh, clip-aware block selection

SparSTAR sparsifies attention without token pruning, KV eviction, or skipped refinement scales — every token still passes through every scale, but each query reads fewer key blocks.

Walkthrough of the method (1:16). Where the attention cost concentrates, how the aggregated query–key selector picks blocks, and how the density budget is spread across scales.

Overview of the SparSTAR selection and sparse execution pipeline

Overview of the selection and sparse execution pipeline.

  • Aggregate contiguous Q/K blocks. Queries and keys are partitioned into 128-token blocks, and per-head block compatibility is scored from the current activations.
  • Apply clip-dependent retention. Required conditioning context stays dense, while eligible current-scale and preceding-clip blocks are ranked under a scale-wise density budget.
  • Execute a forward-only sparse path. Selected blocks are passed to a FlexAttention-based implementation that omits backward-only indexing structures during inference.

Quality–efficiency results

Tables are reproduced from the paper. Reconstruction metrics are measured against paired dense InfinityStar outputs at matched prompts and seeds.

T2V speedup1.60×VBench 83.83
T2V fidelity31.22 dBLPIPS 0.065
I2V speedup1.62×VBench 80.96
I2V fidelity27.96 dBLPIPS 0.080
Table 1: 720p T2V and I2V against token-reduction baselines

Main comparison — 720p, five seconds, 81 frames. Against the published token-reduction baselines, SparSTAR improves paired-output PSNR by 2.92 dB on T2V and 2.31 dB on I2V over the strongest of them, while VBench stays within 0.04 and 0.08 points of dense InfinityStar. † marks the rewritten T2V-prompt variant used for the dense-reference score; dashes are non-applicable metrics.

Table 2: compatibility with complementary acceleration

Composable with other accelerations. Guidance-off skips the unconditional CFG branch at all four sparsified refinement stages; Scale-skip omits the final stage. Stacking either onto SparSTAR pushes end-to-end speedup to 2.2–2.5× for 0.6–1.3 dB of paired-output PSNR.

480p five-second T2V and I2V quality and efficiency

Lower resolution, same picture. At 480p, five seconds, SparSTAR stays closer to the dense output than FastSTAR by 2.60 dB on T2V and 2.28 dB on I2V and roughly halves LPIPS on both, landing within 0.41 and 1.12 VBench points of dense — at 1.30× against FastSTAR’s 1.47×, since 480p leaves less attention work to remove.

7.08×sparse operator speedup over FlashAttention-2 in the matched final-scale benchmark
<1.7%end-to-end latency added by selection and mask construction
74.6%of sparsified-scale attention cost comes from the 72k-token final scale
Ablation of increasing, constant and decreasing density schedules

Hierarchical density schedule. The decreasing schedule allocates more density to earlier, cheaper refinement scales and the most aggressive budget to the final 72k-token scale.

Dense InfinityStar vs. SparSTAR

Full 720p, 5-second, 81-frame generations at matched prompts and seeds — the same setting as Table 1. Left is dense InfinityStar, right is SparSTAR (sparse only, 1.60×). Clips are drawn from those with the lowest LPIPS against the paired dense output.

“A person is doing aerobics”

“a refrigerator and skis”

“A cute fluffy panda eating Chinese food in a restaurant”

“a truck”

BibTeX

@article{lee2026sparstar,
  title   = {SparSTAR: Sparse Attention for SpaceTime AutoRegressive Video Synthesis},
  author  = {Lee, Jongbeom and Yu, Hyunwoo and Yang, Jincheol and Choi, Jaemin and Kang, Suk-Ju},
  journal = {arXiv preprint arXiv:2608.10519},
  year    = {2026}
}

Replace with the @inproceedings entry once the paper appears in proceedings.