Loss functions modulate the optimal bias-variance trade-off

Abstract

Prediction problems vary in the extent to which accuracy is rewarded and inaccuracy is penalized—-i.e., in their loss functions. Here, we focus on a particular feature of loss functions that controls how much large errors are penalized relative to how much precise correctness is rewarded: convexity. We show that prediction problems with convex loss functions (i.e., those in which large errors are particularly harmful) favor simpler models that tend to be biased, but exhibit low variability. Conversely, problems with concave loss functions (in which precise correctness is particularly rewarded) favor more complex models that are less biased, but exhibit higher variability. We discuss how this relationship between the bias-variance trade-off and the shape of the loss function may help explain features of human psychology, such as dual-process psychology and fast versus slow learning strategies, and inform statistical inference.

Publication
Proceedings of the Forty-Second Annual Conference of the Cognitive Science Society
Avatar
Adam Bear
Postdoctoral Researcher