An expression can return the right value and still send an optimizer in the wrong direction. That becomes especially important when an automated system rewrites numerical code. A forward test can pass while the derivative rule, branch behavior, or boundary convention has changed.
One direction from my recent research planning is to search over the forward expression and its differentiation rule together. The target is a checked numerical primitive with an explicit domain, rather than a patch that happens to lower one test error.
This remains a proposal. Automatic numerical repair already has substantial prior art, and custom derivative rules are supported by existing systems. The question is whether joint search and validation can solve a useful set of cases more reliably than established tools and a strong coding agent.
A small cancellation problem
Consider f(x) = (exp(x) − 1) / x near zero. In exact arithmetic the singularity is removable and the limit is 1. In floating-point arithmetic, subtracting nearly equal values can destroy useful information. At zero, the written program also needs a convention.
Replacing exp(x) − 1 with a stable expm1 primitive helps the forward calculation, but it does not finish the contract. A quotient-rule derivative can still contain cancellation near zero. A series expansion may be useful in a small region, but the branch threshold, approximation error, and transition behavior all need checking.
The correct derivative limit is 1/2. That does not mean every program returning 1 at zero should be assigned that derivative. The underlying function and its intended extension must be specified first. A custom gradient cannot repair an undefined scientific meaning by itself.
A tangent rule is a mathematical object
For a differentiable scalar function, a directional derivative should be linear in the tangent direction. If a proposed JVP depends nonlinearly on that tangent, it may produce plausible answers for individual tests while violating the transformation’s contract.
A reverse rule should also agree with the forward rule through the appropriate transpose relation. Tests can compare inner products of a forward-propagated tangent and an output cotangent with the reverse-propagated result. Higher precision and directional finite differences can provide additional references away from ill-conditioned points.
None of these checks alone proves correctness over an entire domain. Together they expose more failure modes than value-only regression tests. The allowed domain, excluded singularities, tolerances, and branch policy should travel with the accepted implementation.
Joint search, bounded acceptance
The proposed search space includes algebraic rewrites, precision choices, branching policies, and explicit derivative rules. An optimizer can trade runtime against value and gradient error, but it should not change the intended function merely because a nearby smoother function trains better.
JAX already provides mechanisms for custom JVPs and VJPs. The opportunity would have to come from finding and checking useful paired implementations, not from wrapping those mechanisms in a new interface.
The comparison should include stable library primitives, forward-only rewriting followed by automatic differentiation, expert implementations, and a coding agent given the same tests and compute. Hold out regions of the domain and compositions of primitives, because an isolated expression can behave differently inside a larger calculation.
Two related ways to get fooled
My PINN-to-interpolant experiments show one version: close function values can conceal large derivative residuals. My SPSA work asks another accounting question: does a proposed normalization add curvature information, or does a retuned learning rate explain the benefit?
Both make me suspicious of a single favorable metric after a transformation. The right response is to state the invariants the next computation needs and test those directly. For numerical repair, that includes the derivative. For an optimization method, it includes a baseline allowed to retune the scale.