Forecasting riverbank deformation requires characterizing complex spatiotemporal heterogeneity governed by coupled hydro-meteorological forcing and site-specific geomorphology. Although deep learning has shown promise for geohazard prediction, existing models for fluvial instability largely operate as black boxes, lacking both physical transparency and spatial transferability. This study proposes the knowledge-infused variational mode decomposition (VMD)-temporal fusion Transformer (TFT), knowledge-infused VMD-TFT (KI-VTFT), a physically constrained and interpretable framework that embeds domain knowledge into data-driven forecasting through two complementary pathways: 1) signal-level priors, injected via Bayesian-optimized VMD (BO-VMD), which partitions displacement into trend, periodic, and residual components; and 2) spatial-level priors, encoded through seven static geomorphic attributes that condition predictions on terrain-controlled susceptibility. The framework was applied to continuous displacement fields derived from 177 Sentinel-1 images (2021–2024) using small baseline subset interferometric synthetic aperture radar (SBAS-InSAR) across three subreaches of the Lower Yangtze River (LYR). At the five-step (60-day) forecasting horizon, KI-VTFT achieved a root-mean-square error (RMSE) of 2.29 mm and an R2 of 0.97, reducing RMSE by 25.96%–65.12% relative to gated recurrent units (GRUs), long short-term memory (LSTM), temporal convolutional networks (TCNs), and Transformer baselines. Interpretability analysis revealed that VMD components drove 64.52% of encoder feature importance, while distance to thalweg (22.51%) and elevation (20.78%) dominated the static controls; temporal attention weights peaked during the May–August flood season, consistent with regional hydrological cycles. Independent validation using uncrewed aerial vehicle and uncrewed surface vessel (UAV–USV) bathymetric surveys, acoustic Doppler current profiler (ADCP) flow measurements, and sediment sampling supported the physical plausibility of model-identified drivers. Furthermore, zero-shot transfer to the separate Wuwei Reach maintained an RMSE of 2.77 mm without retraining, outperforming the strongest baseline, the Transformer (3.65 mm), under identical conditions and demonstrating the framework’s potential for cross-reach generalization.