Paying More Attention To Attention