A Layered Spectral Memory Framework for Reference Object Recasting in Source Preserving Video Transformation Under Occlusion

Abstract

Video object editing often requires an inserted visual identity to behave as though it belonged to a source recording rather than a separately generated image sequence. The difficulty is amplified when a user provides only one reference image, since the system must infer unseen views, occluded parts, and contact behavior from limited evidence. This paper introduces Layered Spectral Memory Recasting, a source preserving video transformation framework that treats reference based editing as a layered transport problem rather than a framewise synthesis task. The method separates a video into a carrier layer, an interaction layer, and a reference layer, then uses a spectral memory bank to retain reference-specific color, material, and part-frequency structure across time. A recurrent transport ledger propagates editable support through source motion while an occlusion ledger suppresses reference hallucination during disappearances and reappearances. The final generator operates in a discrete latent token space with spectral residual constraints, allowing object identity to remain coherent without forcing the replacement to inherit every contour of the source object. Experiments on a constructed benchmark of 126 short videos and 252 single-image replacement requests show that the framework improves reference retention by 10.8%, reduces boundary contamination by 15.6%, and lowers reappearance identity drift by 19.2% against strong token, atlas, and frame-conditioned baselines. The results indicate that reference-object recasting benefits from separating temporal transport, boundary negotiation, and reference memory rather than merging them into a single conditioning pathway.