How would you compare two models without mistaking random variation for an improvement?

Evaluate both models on the same suitable splits and metric, using identical preprocessing boundaries and a meaningful baseline. Examine paired differences, uncertainty, and practical effect size rather than only one average score. Use a resampling or statistical approach appropriate to the data; ordinary independent-sample assumptions can fail for repeated entities or time series. Account for extensive model searching and keep the final test set untouched until selection is complete. Also compare calibration, latency, fai ess-relevant segments, and maintenance cost when those affect deployment.