Simplifying Deep Temporal Model