Language Models Mostly Know What You Did Last Summer