Stable Diffusion From Sidelines