Lightweight Zero Shot Text To Speech With Mixture Of Adapters