Beyond the Final Answer: Process-Oriented Math Training Data

Whenever model reasoning is discussed, math is often the first stop: answers are objective, explicit, and verifiable. Compared with open-ended Q&A or creative tasks, math seems easier to grade automatically—right is right, wrong is wrong. For that reason, math is widely treated as a proxy for reasoning ability: solve harder problems, and the model seems logically stronger.

Yet behind this scorecard lies a problem that is easy to underestimate: an answer that is easy to verify does not mean the reasoning process has been verified.

When training and evaluation lean too heavily on final answers, models are pushed toward outcome-driven learning. They may not master derivation, but become skilled at shortcuts across problem types, numbers, and formula patterns. The deeper issue is that a final answer is only the terminal node of a reasoning chain—it cannot constrain whether intermediate steps are rigorous. A correct answer may come from misapplied theorems or pattern matching, while a sound approach may be marked wrong because of a final arithmetic slip.

If we want models to participate in more complex, open-ended mathematical reasoning, training data must provide richer supervision than answers alone. SoTALab argues that valuable math training data should record not only whether a model reached a final result, but how it got there—whether it understood problem structure, chose a valid path, and identified or corrected errors along a long chain. Only when training signals extend from “is the answer correct?” to “is the reasoning process reliable?” can models truly help human mathematical research.

Public benchmarks already signal the gap

This is also why hard math benchmarks have proliferated in recent years. Epoch AI’s FrontierMath pushes math problems closer to the level of modern mathematical research. It is not asking whether AI can “do math problems,” but whether AI can face the genuinely difficult reasoning structures found in mathematical research.

Public geometry data points to the same issue. The authors of Geometry3K emphasize that geometry solving requires abstract problem understanding, symbolic reasoning, and axiomatic knowledge. GeoQA stresses that geometry problems require jointly understanding textual descriptions, visual diagrams, and theorem knowledge, with programmatic solution-process annotations. More recent multimodal geometry work shows that in zero-shot settings, GPT-4 scores 60%, 36%, and 51% on GPSM4K, PGPS9K, and Geometry3K respectively; Gemini-Pro scores 44%, 30%, and 47%; while some LLaVA-family models remain in single digits or low double digits on multiple geometry datasets. MathVista also shows that even GPT-4V at 49.9% overall accuracy still trails human performance and often fails on complex diagram understanding and strict reasoning.

This suggests a simple but important point: if we train models only in a “problem—answer” format, they may become better at guessing answers without becoming better at mathematics.

A small experiment: five problems, five failure modes

To observe this more concretely, SoTALab selected five representative math examples spanning holomorphic symplectic geometry, algebraic geometry and symbolic computation, numerical PDEs and operator learning, differential geometry, and symbolic integration. We asked multiple frontier models to solve the same set under a unified rubric, averaged scores over five runs, and followed up with human review.

The goal was not to prove that any particular model “cannot do math.” It was to answer a more important question: where exactly do models fail?

1. A quartic radical integral: the issue is not calculation, but hidden structure

The first problem is a quartic radical integral that looks like a routine integration exercise:

$$I=\int \frac{dx}{(x-1)\sqrt[4]{x^3+x}}.$$

What makes it hard is not the length of the integrand, but how hidden the entry point is. Standard substitutions stall quickly. The real breakthrough is recognizing a hidden identity:

$$(x+1)^4-8(x^3+x)=(x-1)^4.$$

This identity suggests a non-obvious normalization variable

$$u=\frac{\sqrt[4]{8x^3+8x}}{x+1}.$$

Once this variable is found, the quartic radical structure collapses into a rational integral and the problem becomes genuinely solvable.

This example exposes a typical symbolic-computation failure mode: models often try familiar routines without discovering the true structure of the problem.

In our runs, some models concluded that the integral “has no elementary antiderivative,” pushing an elementary problem toward special functions or numerical methods. Others reached nearly the right substitution but made mistakes in final coefficients, back-substitution, or domain conditions. The first error reflects weak structure recognition; the second reflects weak result checking.

Why this matters for training data: complex mathematical computation is not mere mechanical algebra. In many advanced problems, the key is to see hidden structure first, then perform valid transformations. For training data, this problem cannot be reduced to a final answer. It must annotate:

• which identity is the breakthrough;
• why ordinary substitutions are insufficient;
• which step turns radicals into a rational integral;
• which near-correct answers differ only by coefficient errors;
• how differentiation can be used to verify the final result.

Such data can move models from “applying formulas” toward “recognizing structure.”

2. Geodesics on an elliptic cylinder: local computation without global minimality

The second problem comes from differential geometry. The surface is an elliptic cylinder:

$$\mathbf r(u,v)=(a\cos u,b\sin u,v),$$

and the task is to find general geodesics; in the cylindrical case $a=b$, find the shortest curve between two points.

The first step is not hard. Models can usually compute the first fundamental form:

$$ds^2=(a^2\sin^2u+b^2\cos^2u)\,du^2+dv^2.$$

Then introduce the elliptic arc-length variable:

$$s(u)=\int_0^u\sqrt{a^2\sin^2t+b^2\cos^2t}\,dt.$$

In $(s,v)$ coordinates, the surface is locally flat, so geodesics locally correspond to straight lines.

The real trap is the second part. For a cylinder, unfolding to the plane does not give a unique image of a point. Because the cylinder wraps around, the same endpoint corresponds to many positions on the unfolded plane:

$$Q,\quad Q+2\pi a,\quad Q-2\pi a,\quad Q+4\pi a,\ldots$$

Each image corresponds to a different path on the cylinder: direct, one wrap, one wrap the other way, two wraps, and so on. The true shortest curve requires comparing the lengths of these candidate lines, not stopping at one unfolded straight segment.

So this problem is not testing “can you write geodesic equations?” It tests whether a model can distinguish:

• local geodesics;
• globally shortest curves;
• different wrapped copies on the unfolded plane;
• how the parameter range $-\pi<u_0<\pi$ affects minimality.

In practice, models often complete the local differential-geometry computation but equate “a straight line after unfolding” with “this must be the shortest path.” That is a classic geometric reasoning error: local correctness does not imply global correctness.

This kind of example matters for training data because it teaches spatial structure, periodicity, parameterization, and global constraints. High-quality samples cannot stop at “the answer is a straight line.” They must explain why other wrapped paths are not shortest.

3. Specialization stability of parametric RUR: fluent conclusions without real proof

The third problem comes from algebraic geometry and symbolic computation, on the specialization stability of parametric Rational Univariate Representations (RUR).

The core task: given a parametric ideal, a Gröbner basis, a separating element, and a family of RURs, prove that when

$$F(p)\Delta_S(p)\neq0,$$

the specialized tuple remains a valid RUR for $I_p$.

Many models, upon seeing this condition, naturally write:

“Since $F(p)\neq0$, the Gröbner basis specializes well; since $\Delta_S(p)\neq0$, roots remain separated, so the RUR is valid.”

This sounds reasonable, but mathematically it is far from enough.

A real proof must simultaneously explain:

1. how $F(p)\neq0$ ensures leading coefficients do not vanish;
2. why leading monomials are preserved under specialization;
3. why the standard monomial basis is preserved;
4. why the dimension of the quotient space is preserved;
5. why the multiplication matrix of the separating element can be specialized term by term;
6. why the characteristic polynomial specializes to the corresponding $X_S(\lambda)$;
7. how $\Delta_S(p)\neq0$ ensures square-freeness and separation;
8. why the denominator $G_S(1,\lambda)$ is nonzero on the relevant roots;
9. why the final coordinate ratios indeed give points of $V(I_p)$.

This problem exposes a common failure in abstract-algebra proofs: knowing the name of a theorem without filling in the condition chain required to apply it.

In our runs, models often recognize the two nonzero conditions but compress the crucial stability argument into “specialization works.” That is dangerous because the answer sounds fluent and directionally correct while skipping the part that most needs verification.

These examples are especially useful for training proof rigor. The data must include not only a standard proof, but a rubric that decomposes each technical bridge: preserving leading terms, preserving the standard monomial basis, explaining dimensional stability, handling multiplication matrices, handling denominators, and stating final RUR validity with care.

Without such process annotations, models easily learn to “sound like a proof” without learning to prove.

4. A fixed-point-free symplectic action: connecting theorems into a closed proof chain

The fourth problem comes from holomorphic symplectic geometry. Given a projective Lagrangian fibration germ $f:M\to X$, where $M$ is a holomorphic symplectic manifold, a finite group $G$ acts equivariantly on $f$, preserves the symplectic form, and acts freely on $M$. If $x\in X$ is fixed by $g\in G$, prove:

$$\det(\operatorname{id}_{T_xX}-dg|_x)=0.$$

The difficulty is organizing multiple given facts into one closed proof chain.

The full reasoning is roughly:

Step 1: because $G$ acts freely on $M$, $g$ has no fixed point on the fiber $F=f^{-1}(x)$.

Step 2: apply the Lefschetz trace formula to this fixed-point-free finite-order automorphism, yielding $\sum_i(-1)^i\operatorname{Tr}(g|H^i(F,\mathcal O_F))=0$.

Step 3: use equivariant base change: $H^i(F,\mathcal O_F)\cong R^if_*\mathcal O_M|_x$.

Step 4: use the cohomology identification for a Lagrangian fibration: $R^if_*\mathcal O_M|_x\cong \Omega_X^i|_x\cong \wedge^iT_x^*X$.

Step 5: turn the alternating trace into a determinant identity on exterior powers: $\det(\operatorname{id}-A)$.

Finally, explain that passing from cotangent space to tangent space via duality does not change whether $\det(\operatorname{id}-A)$ vanishes, yielding the target conclusion.

This problem exposes a core capability in advanced proofs: can the model connect multiple local facts into a proof chain without gaps?

In practice, models often know each fact in isolation but miss the joints: why free action implies no fixed point on the fiber; why the base change is equivariant; why $\Omega_X^i|_x$ is $\wedge^iT_x^*X$; why determinants agree on tangent and cotangent spaces.

This is closer to research-level proof. Research proofs are rarely single-step tricks; they stitch concepts, theorems, and isomorphisms together precisely. If one bridge is missing, the conclusion is unreliable.

5. D2D versus D2E regularity: lower error does not always mean better mathematics

The fifth problem comes from numerical PDEs and operator learning. It compares two neural-operator strategies:

• D2D: predict on a reference domain, then map back to the physical domain via an inverse transform;
• D2E: predict on a larger ambient domain, then restrict to the target region.

Given $D_\alpha(s,t)=(s,t^\alpha)$ with $\alpha>1$, suppose the reference-domain prediction is $\widehat u_{\rm ref}(s,t)=t$. Then D2D maps back to the physical domain as $\widehat u_{\rm D2D}(x,y)=y^{1/\alpha}$.

On the surface this looks like a simple calculation. The real mathematical issue is near the boundary. Its first derivative is $\frac1\alpha y^{1/\alpha-1}$. Because $\alpha>1$, this derivative blows up as $y\to0$. The second derivative behaves like $\partial_{yy}\widehat u_{\rm D2D}\sim y^{1/\alpha-2}$.

To decide membership in $H^2((0,1)^2)$, one cannot stop at “the derivative is unbounded.” One must check whether the square of the second derivative is integrable. Squaring leads to $\int_0^1 y^{2/\alpha-4}\,dy$. Convergence requires $\frac{2}{\alpha}-4>-1$, i.e. $\alpha<\frac23$. But the problem assumes $\alpha>1$, so the D2D prediction is not in $H^2$. By contrast, D2E gives $\widehat u_{\rm D2E}(x,y)=y$, which is smooth.

The problem also reports scalar errors: $E_{\rm D2E}=3.51\%$, $E_{\rm D2D}=3.46\%$, $E_{\rm GeoFNO}=6.52\%$. By $L^2$ scalar error alone, D2D is slightly better; but if evaluation considers function regularity, derivative stability, and downstream PDE post-processing needs, D2E may be preferable.

This example challenges a common intuition: a lower error score does not necessarily mean a more mathematically reliable model.

In our runs, models often compute the inverse map and error percentages correctly but skip second-derivative integrability checks, or cannot explain why regularity-aware evaluation changes model ranking. The lesson is that math training data must teach models not only to compute a metric, but to understand what the metric means mathematically.

For scientific computing and PDE settings, this is especially important. Model outputs must be numerically close and function-space reasonable. Otherwise a model may look good on scalar error yet fail on boundary derivatives, physical residuals, flux computation, or downstream simulation.

Together, these five problems show that current model failures are not simply “insufficient knowledge coverage.” More precisely, on complex mathematical tasks models often lack three capabilities:

First, finding the entry point to a solution.

Second, maintaining the validity of a reasoning chain.

Third, locating and correcting process errors.

Why the reasoning process matters more than the final answer

This is not a SoTALab-only claim. Existing research has repeatedly shown that intermediate reasoning steps are crucial for complex reasoning in large models.

Chain-of-Thought shows that generating intermediate reasoning steps significantly improves performance on arithmetic, commonsense, and symbolic reasoning. OpenAI’s process supervision work further argues that feedback on each reasoning step, rather than only the final answer, more reliably improves mathematical reasoning and produces chains that humans find more acceptable. Let's Verify Step by Step also shows on MATH that process supervision outperforms outcome supervision, and releases PRM800K with 800,000 step-level human feedback labels.

The latest ProcessBench pushes further into error localization: 3,400 test examples drawn mainly from competition and Olympiad math, asking models to identify the earliest incorrect step in a solution—or to judge that all steps are correct.

Together, these works suggest that future gains in mathematical model capability cannot come from more problems and more answers alone. What matters is exposing models to high-quality reasoning paths, high-risk error steps, and expert judgments about whether a chain is trustworthy.

What SoTALab is building: from answer datasets to process datasets

This is the starting point for SoTALab’s math training dataset.

We are not simply collecting problems and answers. We are building math examples around real bottlenecks in mathematical reasoning—examples that are trainable, evaluable, traceable, and reviewable. Each item includes not only Problem, Solution, and Answer, but also Hint, Comment, Classification, model error outputs, error locations, error types, step-level rubrics, and human review records.

In other words, we aim to build not a “question bank,” but mathematical reasoning training samples.

In domain coverage, we continue to expand algebra, geometry, analysis, numerical computation, probability and statistics, combinatorics, and number theory. Algebra trains abstract structure and proof rigor; geometry trains spatial modeling, diagram understanding, and local-to-global reasoning; computation trains structure recognition, valid transformation, and self-checking; numerical PDE and scientific-computing problems train models to understand the mathematical meaning behind error metrics.

In evaluation, we insist on multiple models, multiple runs, a unified rubric, and human review. We care not only about total score, but about why points were lost: misreading the problem, misapplying a theorem, overextending a local conclusion, missing denominator conditions, or making a final back-substitution error.

In data production, mathematicians, competition problem authors, math pedagogy experts, and AI evaluation engineers work together. Mathematicians ensure professional quality of problems and solutions; pedagogy experts ensure decomposition granularity and teachability; evaluation engineers ensure rubrics can be executed consistently; human review ensures trustworthiness.

This is the essential difference between high-quality math data and ordinary Q&A data. Ordinary Q&A emphasizes knowledge retrieval; math training emphasizes structured reasoning. Ordinary data tells a model what the answer is; process-oriented math data tells a model why to take each step, where mistakes are likely, and how to check whether a step is valid.

Looking ahead: AI should not replace mathematicians—it must first learn to be checked like one

We believe AI can become a collaborative tool for mathematical researchers: proposing conjectures, searching for examples, organizing proofs, checking boundary conditions, and in some directions helping discover new structure.

But first it must accept the most basic requirement of the mathematical community: every step must be checkable, every conclusion must have a legitimate source, and every failure must be locatable.

That is the problem SoTALab’s math training dataset is designed to solve.

With expert-constructed high-quality examples, we want to move models from “can give an answer” to “can organize reasoning”; from “the result looks right” to “the process withstands scrutiny”; from “occasionally solving a hard problem” to “forming stable, reviewable mathematical judgment.”

Math problems are not answer problems. Valuable math data should not be answer data alone.