Orca: Progressive Learning from Complex Explanation Traces of GPT-4 ...

• Main Authors: Mukherjee, Subhabrata, Mitra, Arindam, Jawahar, Ganesh, Agarwal, Sahaj, Palangi, Hamid, Awadallah, Ahmed
• Format: Report
• Language: unknown
• Published: arXiv 2023
• Subjects: Computation and Language cs.CL; Machine Learning cs.LG; FOS Computer and information sciences; Orca
• Online Access: https://dx.doi.org/10.48550/arxiv.2306.02707; https://arxiv.org/abs/2306.02707

Recent research has focused on enhancing the capability of smaller models through imitation learning, drawing on the outputs generated by large foundation models (LFMs). A number of issues impact the quality of these models, ranging from limited imitation signals from shallow LFM outputs; small scale homogeneous training data; and most notably a lack of rigorous evaluation resulting in overestimating the small model's capability as they tend to learn to imitate the style, but not the reasoning process of LFMs. To address these challenges, we develop Orca (We are working with our legal team to publicly release a diff of the model weights in accordance with LLaMA's release policy to be published at https://aka.ms/orca-lm), a 13-billion parameter model that learns to imitate the reasoning process of LFMs. Orca learns from rich signals from GPT-4 including explanation traces; step-by-step thought processes; and other complex instructions, guided by teacher assistance from ChatGPT. To promote this progressive ...