Loss functions

This post presents a structured taxonomy of loss functions used in deep learning, organizing them by task type and objective mechanism. The list is not exhaustive, as it focuses on the most widely adopted losses in modern computer vision and machine learning research. Niche or highly specialized variants are omitted, as well as losses specific to sequence-to-sequence tasks.

The section on generative tasks serves as a broad overview of terms rather than a granular list of standalone functions. Conversely, the discriminative section provides a more detailed breakdown of specific formulations.

The goal of this taxonomy is to provide an intuitive yet mathematically rigorous reference for selecting the appropriate loss function based on the geometric and probabilistic requirements of a specific problem. The categorization and definitions presented here are primarily derived from Li et al. (2025).

this will be skipped

Discriminative tasks

Optimize for \(P(Y|X)\). These losses focus on defining decision boundaries or fitting functions that map inputs directly to targets.

D1. Regression losses (continuous)

Regression models aim to predict a continuous dependent variable \(y\) based on independent variables \(x\). Losses in this category are functions of the residuals—the difference between the observed value \(y\) and the predicted value \(\hat{y} = f(x)\).

D1a. Magnitude-based (point-wise)

These losses measure the point-wise error between the prediction and the ground truth. They guide models to approximate the target value by minimizing the magnitude of these errors.

MAE (Mean Absolute Error)

🗒️ Calculates the average of the absolute differences between the predicted and actual values.
💡 "I don't care about the direction of the error, just tell me on average how many units I am off by. Also, I won't freak out over massive outliers."
✅ Robust to outliers (linear penalty).
✅ Provides a physical unit of error that is interpretable.
❌ Gradients are non-differentiable at 0, which can complicate convergence near the optimum.

MSE (Mean Squared Error)

🗒️ Calculates the average of the squared differences. Squaring the error ensures positivity and penalizes larger errors disproportionately more than small ones.
💡 "Small mistakes are okay, but if you make a huge mistake, I am going to punish you severely to make sure you never do it again."
✅ Differentiable everywhere (smooth gradient descent).
✅ Converges faster than MAE when close to the minimum.
❌ Highly sensitive to outliers, one bad data point can skew the entire model.
↔️ Variant RMSE: Converts the error back into the original units of the target variable by taking the square root, making it easier to interpret.
↔️ Variant RMSLE: Makes the loss sensitive to relative errors rather than absolute ones and penalizes underestimation more than overestimation.

Log-Cosh

🗒️ Computes the logarithm of the hyperbolic cosine of the prediction error. It approximates \(\frac{x^2}{2}\) for small \(x\) and \(|x| - \log(2)\) for large \(x\).
💡 "Act like MSE when the error is small to fine-tune gently, but switch to MAE behavior when the error is huge so outliers don't distract you."
✅ Combines the best of both worlds: robust to outliers (like MAE) and differentiable everywhere (like MSE).
❌ Computationally more expensive than simple polynomial losses.

Huber

🗒️ A piecewise function that is quadratic for small errors (below a threshold \(\delta\)) and linear for large errors. It requires a hyperparameter \(\delta\) to define the transition point.
💡 "Don't panic if a data point is way off, just pull it in linearly. But once you get close, curve the loss to land the plane smoothly."
✅ Robust to outliers while maintaining differentiability at 0.
❌ Introduces a hyperparameter (\(\delta\)) that must be tuned.

Quantile

🗒️ An extension of MAE that applies different penalties to overestimation and underestimation based on a chosen quantile \(\tau\) (between 0 and 1). Used for predicting prediction intervals rather than a single mean.
💡 "I don't just want the average outcome, I want to be 90% sure the real value is below my prediction line."
✅ Allows for uncertainty estimation and construction of confidence intervals.
❌ More difficult to train, convergence can be slower than standard MSE/MAE.

D1b. Geometry-aware (bounding boxes)

In object detection tasks, the goal of bounding box regression is to achieve geometric alignment between the predicted box and the ground truth. Unlike magnitude-based losses, geometric losses do not treat coordinates in isolation; instead, they view the box as a unified geometric entity, optimizing the spatial relationship (overlap, distance, and shape) between the prediction and the ground truth.

IoU (Intersection over Union)

🗒️ Measures the overlap area between the predicted box \(B\) and the ground truth box \(B^{gt}\) divided by their union area.
💡 "I don't care where the pixels are exactly, just make sure the two squares overlap as much as possible."
✅ Invariant to the scale of the problem (a small box and large box with same overlap % have same loss).
❌ If boxes do not overlap: IoU is 0, the gradient is 0, and the model stops learning completely.
❌ If boxes overlap completely: IoU is 1, and the gradient becomes 0 again.

GIoU (Generalized IoU)

Where \(C\) is the smallest convex box covering both \(B\) and \(B^{gt}\).
🗒️ Adds a penalty term based on the empty space within the smallest enclosing box \(C\). This ensures gradients exist even when boxes do not overlap.
💡 "If the boxes aren't touching, move the prediction towards the target to minimize the empty space between them."
✅ Solves the vanishing gradient problem of standard IoU for non-overlapping boxes.
❌ Does not solve the vanishing gradient problem of standard IoU for completely-overlapping boxes.
❌ Convergence is slow, it tends to expand the predicted box to cover the target first before shrinking to fit.

DIoU (Distance IoU)

Where \(\rho\) is Euclidean distance, \(b\) and \(b^{gt}\) are the center points, and \(c\) is the diagonal length of the enclosing box.
🗒️ Adds a penalty minimizing the normalized distance between the center points of the two boxes.
💡 "Don't just overlap, aim for the bullseye. Align the centers of the boxes directly."
✅ Converges much faster than GIoU because it minimizes distance directly rather than area.
✅ Completely solves the vanishing gradient problem.
❌ Does not consider the aspect ratio of the boxes.

CIoU (Complete IoU)

Where \(v\) measures aspect ratio consistency and \(\alpha\) is a weighting parameter.
🗒️ Extends DIoU by adding a term to ensure the aspect ratio of the prediction matches the target.
💡 "Overlap, hit the center, and make sure you aren't drawing a tall rectangle when it should be a wide one."
✅ Considers all geometric factors: overlap area, central point distance, and aspect ratio.
❌ The aspect ratio term \(v\) is complex and gradients can sometimes be unstable depending on the implementation.

EIoU (Efficient IoU)

Where \(w,h\) are width/height and \(C_w, C_h\) are the width/height of the enclosing box.
🗒️ Improves CIoU by splitting the aspect ratio term into separate penalties for width and height differences.
💡 "CIoU was a bit messy with the shape math. Let's just strictly measure the width error and the height error separately."
✅ Faster convergence and better localization accuracy than CIoU.
✅ Solves the ambiguity in CIoU where different \(w/h\) pairs could produce the same aspect ratio penalty.

SIoU (Scylla-IoU)

Where \(\Delta\) is the distance cost and \(\Omega\) is the shape cost.
🗒️ Introduces an angle cost to the regression. It considers the vector angle between the center of the predicted box and the ground truth. It prioritizes aligning the box to the nearest axis (X or Y) to minimize freedom of movement.
💡 "Stop wandering around diagonally! Move strictly horizontal or vertical to line up with the target first, then adjust the size."
✅ Converges faster than CIoU and EIoU by reducing the oscillation of the box during training.
❌ Computationally slightly heavier due to the calculation of trigonometric (inverse sine) components.

D2. Classification losses (discrete)

Classification is a subset of supervised learning tasks where the goal is to assign an input \(x\) to one of \(K\) discrete classes.

D2a. Margin-based (decision boundaries)

Margin losses introduce a threshold parameter to enforce a minimal separation between the predicted score and the correct class. They compel the model to not just classify correctly, but to do so with high confidence by maintaining a 'safe distance' from the decision boundary.

Hinge

🗒️ The standard loss for Support Vector Machines (SVMs). It only penalizes the model if the correct class score is not sufficiently higher than the margin. If the prediction is correct and confident (\(y \hat{y} \ge 1\)), the loss is zero.
💡 "I don't just want you to be right, I want you to be right by a wide margin. If you barely squeak past the finish line, I'm still giving you a penalty."
✅ Points that are correctly classified with high confidence have 0 gradients and do not affect the model update, saving computation.
❌ The function is non-differentiable at \(y\hat{y}=1\), requiring sub-gradient optimization methods.
↔️ Variant Squared Hinge: Differentiable but still sensitive to outliers.
↔️ Variant Quadratic Smoothed Hinge: Linear for large errors to keep robustness, and quadratic near the margin boundary to ensure differentiability.
↔️ Variant Ramp: Caps the loss to ignore extreme outliers.

Exponential

🗒️ Primarily used in boosting algorithms like AdaBoost. It applies an exponential penalty to negative margins (incorrect classifications).
💡 "If you get a difficult example wrong, the penalty will be massive. You must obsess over the hardest data points."
✅ Forces the model to focus intensely on the examples it is currently getting wrong.
✅ Differentiable and convex.
❌ Because the penalty grows exponentially, a single mislabeled outlier can dominate the gradient and ruin the training process.

D2b. Probabilistic (distribution divergence)

Let \(q\) be the true probability distribution of the dataset and \(p_{\theta}\) be the predicted distribution generated by the model. Probabilistic loss functions measure the divergence (distance) between \(q\) and \(p_{\theta}\). By minimizing this divergence, the model's output distribution converges toward the ground truth.

CE (Cross-Entropy)

🗒️ Measures the information difference between the predicted distribution and the true distribution. When targets are one-hot encoded, minimizing CE is mathematically equivalent to maximizing the likelihood of the correct class.
💡 "If the image is a cat, I want the probability of 'cat' to be 1.0. Every bit of probability mass assigned to 'dog' or 'bird' increases the penalty."
✅ The default loss for classification, differentiable and rigorous strictly based on Information Theory.
❌ Dominated by majority classes if data is unbalanced.
❌ Dominated by easy examples (background) in dense detection tasks.
↔️ Variant Weighted CE: Multiplies the loss of class \(k\) by a weight \(\alpha_k\) (usually inverse to class frequency or the effective number of samples).
↔️ Variant Label Smoothing: Changes target \(y=1\) to \(y=1-\epsilon\) and \(y=0\) to \(y=\frac{\epsilon}{K-1}\) to prevent overfitting.

Focal

🗒️ Adds a modulating factor \((1 - \hat{p}_i)^\gamma\) to standard Cross-Entropy. If a sample is already well-classified (e.g., \(\hat{p}_i = 0.9\)), the factor approaches 0, effectively silencing the loss for that example.
💡 "I don't care about the background sky that you've already correctly identified 1,000 times. Focus entirely on that one difficult pixel that looks like a pedestrian."
✅ Solves the class imbalance problem without manual oversampling.
✅ The standard for dense object detection.
❌ Requires tuning two hyperparameters (\(\alpha\) and \(\gamma\)) which can be sensitive to the dataset.

GHM (Gradient Harmonized Mechanism)

Where \(g_i\) is the gradient norm and \(GD\) is the gradient density (a measure of how many samples have that same gradient).
🗒️ An advanced alternative to Focal loss. It observes that easy examples and outliers both have distinct gradient behaviors. It harmonizes training by normalizing the loss based on the density of gradients. If a million examples produce the same small gradient (easy background), their contribution is divided by a large density factor.
💡 "If everyone else is shouting the same thing, I'm going to turn down the volume on that group. I only want to hear the unique/rare errors."
✅ Does not require manual tuning of \(\alpha\) or \(\gamma\) like Focal loss, it adapts to the training data dynamics.
❌ The computational cost increases: it requires calculating a histogram of gradients across the batch/dataset during training.

Poly

🗒️ Views Cross-Entropy as a Taylor series expansion and generalizes it, allowing for the adjustment of the leading polynomial coefficients \(\alpha_j\) to structurally change how the loss behaves (instead of having it fixed at \(1/j\)). In practice, usually only the leading coefficient (\(\epsilon_1\)) is modified: \(L_{Poly1} = L_{CE} + \epsilon_1 (1 - \hat{p}_t)\).
💡 "Cross-Entropy is a fixed curve, but we can reshape it if we need to."
✅ Generalized framework that encompasses Cross-Entropy and Focal Loss as special cases.
✅ Poly-1 often outperforms Focal/CE by tuning a single parameter, with negligible additional computational cost.
❌ Introduces a non-standard hyperparameter that must be found via grid search, as there is no universal default.

D2c. Overlap-based (segmentation)

In semantic segmentation, classification occurs at the pixel level. However, pixel-wise losses often struggle when the target object occupies only a small fraction of the image. Overlap-based losses address this by directly optimizing the intersection between the predicted segmentation map and the ground truth, prioritizing global shape alignment over individual pixel accuracy.

Tversky

Where \(y\) is the target, \(\hat{y}\) is the prediction, \(\alpha\) controls penalty for false positives, and \(\beta\) controls penalty for false negatives.
🗒️ A generalization of the Dice coefficient (when \(\alpha = \beta = 0.5\)). Allows the shifting of the balance between precision (avoiding false positives) and recall (avoiding false negatives).
💡 "If finding the tumor is critical and we cannot afford to miss it, set \(\beta\) higher than \(\alpha\) to punish missing pixels more than extra ones."
✅ Much better than Cross-Entropy at imbalance handling for small objects.
✅ Parameters \(\alpha\) and \(\beta\) give flexibility to tune the trade-off based on clinical or business needs.
❌ Can be unstable during the early stages of training compared to pixel-wise CE.
↔️ Variant Focal Tversky: Applies the mechanism of Focal loss to the Tversky index.

Sensitivity-Specificity

🗒️ Explicitly optimizes the weighted sum of the squared errors for the positive class (sensitivity) and the negative class (specificity). It ensures the model does not achieve high accuracy by simply ignoring the background or the foreground.
💡 "I need you to be good at finding the object, but equally good at NOT finding the object where it doesn't exist. Balance your excitement."
✅ Addresses both over-segmentation (too much background included) and under-segmentation (missing parts of the object).
✅ Appropriate for medical contexts where specificity is just as vital as sensitivity.
❌ Highly sensitive to the weight parameter \(w\). If set incorrectly, the model may collapse into predicting only the background or only the foreground.

D3. Metric losses (embedding space)

The goal of metric learning is to learn the relative distances between inputs rather than predicting a specific label or value. Metric loss functions operate on pairs (or triplets) of data instances, extracting an embedded representation for each. A distance metric measures the similarity between these representations. The model is trained to minimize the distance between representations of similar inputs and maximize the distance between dissimilar ones, structuring the embedding space meaningfully.

D3a. Euclidean distance

These losses directly use geometric distance in the embedding space as the optimization target.

Contrastive Loss

Where \(D = ||f(x_1) - f(x_2)||_2\) is the Euclidean distance between the pair of samples, \(Y=1\) implies same class, \(Y=0\) implies different class, and \(m\) is the margin.
🗒️ Takes pairs of samples. If they belong to the same class, it minimizes their distance. If they belong to different classes, it pushes them apart until they are at least margin \(m\) away.
💡 "If you are twins, hug each other. If you are strangers, push away until you have at least 1 meter of personal space between you."
✅ Is the simple foundational approach to metric learning.
❌ Hard to tune the margin. If \(m\) is too small, clusters overlap; if too large, training becomes unstable.

Triplet

Where \(a\) is anchor, \(p\) is positive (same class), \(n\) is negative (different class), and \(m\) is margin.
🗒️ Takes three samples at once: an anchor, a positive, and a negative. It ensures that the anchor is closer to the positive than it is to the negative by at least margin \(m\).
💡 "I don't care exactly where the anchor is located, as long as its friend (positive) is closer to it than its enemy (negative)."
✅ More flexible than Contrastive loss because it relaxes the constraint on absolute distances, only the relative ranking matters.
❌ Requires finding negatives that are currently closer than positives. If random negatives are picked, the loss is usually 0 and the model learns nothing.

InfoNCE (Information Noise-Contrastive Estimation)
🗒️ It treats the task as a classification problem: "Among this batch of \(K\) negatives and 1 positive, identify the positive." It maximizes the mutual information between the query and the positive key.
💡 "Here is one photo of a dog and 1,000 photos of other things. Can you pick the correct dog out of this lineup?"
✅ Learns from one positive and many negatives simultaneously, providing a much richer gradient signal than Triplet.
✅ It is the standard backbone for modern self-supervised representation learning.
❌ Often requires very large batch sizes (to have enough hard negatives) to work effectively.

D3b. Angular margin

Losses based on angular or cosine margins do not directly optimize absolute position and distance on the feature space. Instead, they focus on the angular boundaries between classes by projecting features onto a hypersphere and optimizing the cosine similarity between feature vectors and class centers.

A-Softmax (Angular Softmax / SphereFace)

Where \(\psi(\theta)\) is a monotonic function replacing \(\cos(\theta)\) with \(\cos(m\theta)\).
🗒️ The first major angular loss. It introduces a multiplicative angular margin \(m\), and forces the angle of the correct class to be \(m\) times smaller than the angle of any incorrect class.
💡 "If the angle to your class center is 10 degrees, I will pretend it is actually 40 degrees (\(m=4\)). You have to work 4 times harder to prove you belong there."
✅ Pioneered the concept of angular margins, proving that geometric constraints on the hypersphere significantly improve feature discrimination.
❌ The optimization is difficult and requires complex annealing of the hyperparameter \(\lambda\) to converge.

AM-Softmax (Additive Margin Softmax / CosFace)

Where \(s\) is a scale factor and \(m\) is an additive cosine margin.
🗒️ Simplifies SphereFace by moving the margin \(m\) outside the cosine function. It subtracts a margin \(m\) directly from the cosine similarity value.
💡 "Standard Softmax is too lenient. I'm going to subtract 0.3 from your similarity score. You effectively need a score of 1.3 to get a perfect 1.0. Push harder!"
✅ Much easier to implement and train than SphereFace.
✅ It is more interpretable as it directly optimizes cosine similarity gap.

Additive Angular Margin (ArcFace)

Where \(m\) is an additive angular margin added inside the cosine.
🗒️ It adds the margin \(m\) inside the cosine term, which corresponds to a direct geodesic distance penalty on the hypersphere.
💡 "Imagine the classes are countries on a globe. ArcFace draws strict borders with a 'No Man's Land' buffer zone between every country directly on the surface of the sphere."
✅ The margin has a constant correspondence to arc length on the hypersphere.
✅ Is the state-of-the-art for face recognition.
❌ Requires careful tuning of scale \(s\) and margin \(m\) depending on dataset noise.

Quality Adaptive Margin Softmax (AdaFace)

Where \(g_{angle}\) is a margin function that adapts based on the image quality (feature norm \(||\hat{z}_i||\)).
🗒️ Adapts the margin based on the quality of the input image. It applies a strict margin to high-quality images and a relaxed margin to low-quality images to prevent the model from overfitting to noise.
💡 "If the photo is HD, I expect perfection. If the photo is a blurry CCTV frame, I'll go easy on you so you don't get confused trying to learn noise."
✅ State-of-the-art for unconstrained face recognition (e.g., surveillance, low resolution).
✅ Prevents the model from getting stuck trying to optimize unrecognizably bad samples.
❌ Introduces complexity in implementation and relies on the assumption that feature norm correlates with image quality (which is usually true, but not always).

Generative tasks

Optimize for \(P(X)\) or \(P(X,Y)\). Losses for these tasks focus on learning the underlying data distribution to generate new samples or reconstruct inputs. These objective functions are often composite, blending multiple terms (reconstruction, distribution matching, perceptual quality) to achieve realistic results.

G1. Reconstruction terms (element-wise)

These terms ensure fidelity by measuring the direct difference between the original input \(x\) and the reconstructed/generated output \(\hat{x}\).

MSE (Mean Squared Error)
🗒️ Is the same function used for discriminative tasks. For generative tasks, this acts as the "fidelity" term.
💡 "The generated image must look exactly like the input, pixel by pixel."
✅ Simple to implement and theoretically guarantees the highest PSNR (Peak Signal-to-Noise Ratio).
❌ In generative contexts, pure MSE tends to produce blurry images because it averages out high-frequency details.

G2. Distribution matching terms (divergences)

These terms minimize the statistical discrepancy between the learned generated distribution \(P_g\) and the real data distribution \(P_{data}\). They are central to GANs and Variational Autoencoders (VAEs).

Minimax (GAN loss)
🗒️ A zero-sum game between two networks: a generator (\(G\)) tries to fool the discriminator, and a discriminator (\(D\)) tries to distinguish real from fake.
💡 "Generator: I bet I can trick you. Discriminator: No you can't, I'll spot the fake."
✅ Produces very sharp, realistic details compared to MSE.
❌ Very difficult to train.

Wasserstein Distance (WGAN loss)
🗒️ It calculates the minimum "work" (mass × distance) required to transform one distribution into another. Unlike the standard GAN loss, the discriminator (now called critic) outputs a raw score, not a probability.
💡 "Instead of asking 'True or False?', better ask 'How real is this?' to let the generator exactly know how far it is from the target, even if it's currently failing completely."
✅ Provides meaningful gradients even when the real and fake distributions do not overlap at all, solving the vanishing gradient problem of standard GANs.
✅ The loss value correlates linearly with the visual quality of the generated images, which is not true for standard GAN loss.
❌ Requires enforcing 1-Lipschitz continuity (the gradient cannot change too fast), which is difficult to implement (requires weight clipping or gradient penalties).

KL (Kullback-Leibler Divergence)

🗒️ Measures how much information is lost when distribution \(Q\) is used to approximate \(P\). In VAEs, it forces the learned latent space to follow a standard Gaussian distribution.
💡 "Keep your latent code organized like a standard bell curve so we can sample from it easily later."
✅ Forces the learned latent variables to follow a tractable distribution (usually Unit Gaussian), ensuring the latent space is smooth and continuous.
❌ The strict Gaussian constraint often results in over-regularized and blurry outputs.

Sinkhorn Divergence

Where \(C\) is the cost matrix, \(\pi\) is the transport plan, and \(H\) is regularization entropy.
🗒️ Adds an entropic regularization term to the optimal transport problem. This allows the Wasserstein distance to be computed much faster using the Sinkhorn-Knopp algorithm.
💡 "Calculating the perfect earth-moving plan is hard. If we allow a little bit of randomness in where the dirt goes, we can solve the math 100x faster."
✅ Differentiable and computationally fast enough to be used as a loss function.
❌ If \(\epsilon\) is too large, the metric becomes too blurry and loses the geometric accuracy of the true Wasserstein distance.

G3. Diffusion terms (noise removal)

Used in Diffusion Probabilistic Models (DDPMs). The goal is to reverse a gradual noising process.

Simple Diffusion
🗒️ The model predicts the noise \(\epsilon\) that was added to the image \(x_0\) at timestep \(t\).
💡 "I will show you a noisy TV screen. You tell me exactly which pixels are noise so I can subtract them and reveal the image underneath."
✅ Training is essentially a massive set of regression tasks (MSE on noise), which is a lot more stable compared to GANs.
❌ Inference is slow, as generating a single image requires running the network iteratively (e.g., 50 to 1000 times) to denoise the output step-by-step.

Denoising Score Matching
🗒️ Optimizes the model to estimate the score function (the gradient of the log-density of the data). By moving along the gradient, you move from a noisy data point toward the clean data manifold.
💡 "You are dropped in a foggy forest. You don't know where the mountain peak is, but if you just look at your feet and step where the ground slopes upward, you will eventually get there."
✅ Bypasses the intractable problem of calculating the normalizing constant of the probability distribution.
❌ Technically complex to derive and implement compared to the simplified objective used in practical Simple Diffusion.

G4. Auxiliary guidance terms (feature-based)

Instead of comparing raw pixels, these losses compare high-level representations extracted by a pre-trained network.

Perceptual

Where \(\phi\) is a pre-trained feature extractor.
🗒️ Compares the internal activation maps of a pre-trained network for the real and generated images.
💡 "I don't care if the exact pixel matches. Does the image look like a dog? Do the edges and textures match human perception?"
✅ Correlates much better with human visual judgment than MSE, and provides excellent textures for style transfer and super-resolution.
❌ Relies on pre-trained networks, so it may fail or produce artifacts if the target domain is vastly different.

Style

Where \(G\) computes the Gram Matrix—correlations between features.
🗒️ Measures the correlation between different feature channels. It captures "style" (texture, brushstrokes, color patterns) while discarding spatial structure.
💡 "Capture the vibe of Van Gogh but don't worry about where the trees are actually located."
✅ Explicitly decouples texture from structure, allowing for the synthesis of complex artistic patterns without needing paired training data.
❌ Does not enforce spatial coherence, so texture patches can appear in semantically incorrect locations (e.g., brushstrokes appearing in the sky instead of the trees).