Less Is More For Alignment