New Algorithm Aces University Math Course Questions

Machine learning models have consistently struggled with topics like multivariable calculus, differential equations, and linear algebra, which many MIT students can easily master.

Best Version

The best versions have only been able to answer elementary or high-school-level mats questions, and they do not always find the correct solutions. At present, a group of researchers with varied expertise from MIT and other institutions, headed by Iddo Drori, who teaches in the Department of Electrical Engineering and Computer Science (EECS) at MIT, has successfully used a neural network model to solve college-level math problems in just a few seconds, matching human-level performance.

The model can generate solutions and create new problems in various math topics taught at the university level. When presented with machine-generated questions, university students were unable to distinguish them from questions created by humans. This technology could simplify the process of generating course content, particularly for large residential courses and massive open online courses (MOOCs) with a large number of students. Additionally, the system could serve as an automatic tutor, demonstrating the necessary steps for solving undergraduate math problems.

Drori, who will be a faculty member at Boston University starting this summer, and also the primary author of the study conducted in the Division of Computer Science at Columbia University, believes that this technology could boost higher education by augmenting student learning, aiding educators in generating fresh material, and raising the level of complexity in select courses. It also enables us to build a graph of questions and courses, which helps us understand the partnership between courses and their prerequisites, not just by historically contemplating them, but based on information. “The work is a collaboration involving students, researchers, and professors at MIT, Columbia College, Harvard College, and the College of Waterloo. The Proceedings of the National Academy of Sciences has published the research, which was conducted by a team of researchers led by Gilbert Strang, a professor of mathematics at MIT.

A “eureka” moment

Drori and his students and colleagues have worked on this project for nearly two years. Their discovery indicated that models pre-trained using text were incapable of achieving more than eight percent accuracy on high school math problems. People who use neural chart networks could excel in answering questions in a machine learning course, but training the network would take a week.

However, Drori had a breakthrough moment when he came up with an idea to turn questions from MIT and Columbia College math courses into programming tasks using program synthesis and few-shot learning. This involves converting the question into a programming task, such as changing “find the distance between two factors” to “write a program that finds the difference between 2 points,” and using a few question-program pairs as samples. To improve their previous attempts, the researchers added a new step before inputting the programming tasks into the neural network, resulting in better performance.

GPT-3

Previously, both they and others attempting to solve this issue have utilized a neural network, like GPT-3, that was pre-trained only on text to comprehend the intricacies of natural language. To achieve proficiency, thousands of text examples were required for learning. However, this time, they used a neural network called Codex, produced by OpenAI, which was pre-trained on text but also “fine-tuned” on code. Fine-tuning is an additional pre-training step that can enhance the performance of a machine-learning model.

Millions of examples of code from online repositories were presented to the pre-trained model for it to learn. As this model was trained on millions of natural language words and numerous lines of code, it developed an understanding of the relationship between textual elements and coding elements. Many mathematical problems can be resolved using a computational graph or tree. However, it is hard to turn a problem written in text into this kind of representation, Drori explains. As this version has acquired knowledge of the connections between text and code, it can translate a text-based query into code by utilizing only a few examples of question-and-code pairings, and then execute the generated code to provide a solution to the problem.

“He explains that when a question is presented solely in text format, it can be challenging for a machine learning model to generate a suitable response, even if the answer is contained within the text. This research bridges the gap by incorporating code and program synthesis to complement the textual content.” This work is the 1st to solve undergraduate math problems and also moves the needle from eight percent accuracy to over eighty percent, Drori includes.

Adding context

Turning mathematic questions into programming tasks is not always straightforward, Drori says. Some problems require scientists to add context so the neural network can process the question correctly. While a student would grasp this context while attending the course, a neural network lacks this prior knowledge unless it is explicitly provided by the researchers.

For instance, they might be required to clarify that the “network” in a question’s text refers to “neural networks” rather than “communications networks.” Alternatively, they might need to tell the model which programming package to use. They may also need to provide specific definitions; in a question about poker hands, they might need to inform the model that each deck contains fifty-two cards. The programming tasks, along with their context and examples, are fed into a neural network that has been pre-trained and fine-tuned. The output of this neural network is usually a program that can produce the correct answer. It was correct for more than eighty percent of the questions.

The scientists additionally utilized their model to produce questions by giving the neural network a series of math problems on a topic and then asking it to produce a brand new one. “In some subjects, it surprised us. As an example, when questions were asked about the quantum detection of horizontal and vertical lines, new questions were created about the quantum detection of diagonal lines. So, it is not just creating new questions by replacing values and variables in the existing questions,” Drori states.

Human-generated vs machine-generated questions

The researchers examined the machine-generated questions by showing them to university students. The researchers randomly gave students ten questions from each undergraduate math course; humans developed five, and five were machine-generated.

Students could not tell whether an algorithm or a human produced the machine-generated questions, and they gave human-generated and machine-generated questions similar marks for the level of difficulty and appropriateness for the course. Drori clarifies that the purpose of this work is not to substitute human professors.

“Automation is now at eighty percent, but automation will never be 100 percent accurate. Whenever you solve something, someone will come up with a harder question. Nevertheless, this work opens the field for people to start solving tougher and also tougher questions with machine learning. We think it will greatly impact higher education,” he says.

The group is excited by their approach’s success and has extended the work to handle mathematics proofs, but there are some limitations they plan to tackle. At present, the model is incapable of responding to questions that involve visuals and solving problems that are computationally intractable because of their complexity. In addition to overcoming these hurdles, they are working to scale the version up to hundreds of programs. With those hundreds of courses, they will create more data that can enhance automation and offer insights into course design and educational programs.

Reference: Iddo Drori et al, A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level, Proceedings of the National Academy of Sciences (2022). DOI: 10.1073/pnas.2123433119

Read the original article on Tech Xplore.