Reliable Reasoning Beyond Natural Language Paradigm