Learning To Generate Better Than Your Llm