Federated Unlearning With Diffusion Models Already Have A Semantic Latent