Point Diffusion Implicitly Understood