Aligning Agents Like Large Language Models