Python cross-entropy code snippets

The cross-entropy loss function is commonly used in machine learning for classification problems. It measures the difference between the true probability distribution and the predicted probability distribution. The cross-entropy loss is minimized during the training process to optimize the model's performance.

Here are some Python code snippets that implement the cross-entropy loss function:

Binary Cross-Entropy Loss

python
import numpy as np

def binary_cross_entropy(t, p):
    t = np.float_(t)
    p = np.float_(p)
    return -np.sum(t * np.log(p) + (1 - t) * np.log(1 - p))

This function takes the true labels `t` and the predicted probabilities `p`, and computes the binary cross-entropy loss. [2]

Categorical Cross-Entropy Loss

python
import numpy as np

def categorical_cross_entropy(t_list, p_list):
    t_list = np.float_(t_list)
    p_list = np.float_(p_list)
    losses = []
    for t, p in zip(t_list, p_list):
        loss = -np.sum(t * np.log(p))
        losses.append(loss)
    return np.sum(losses)

This function takes the true probability distributions `t_list` and the predicted probability distributions `p_list`, and computes the categorical cross-entropy loss. [2][3]

Cross-Entropy Loss with Gradient Descent

python
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def cross_entropy_loss(y_true, y_pred):
    y_pred = np.clip(y_pred, 1e-15, 1 - 1e-15)
    loss = -y_true * np.log(y_pred) - (1 - y_true) * np.log(1 - y_pred)
    return np.mean(loss)

def gradient_descent(X, y_true, w, learning_rate, num_iters):
    for _ in range(num_iters):
        z = np.dot(X, w)
        y_pred = sigmoid(z)
        loss = cross_entropy_loss(y_true, y_pred)
        dz = y_pred - y_true
        dw = np.dot(X.T, dz) / len(y_true)
        w -= learning_rate * dw
    return w

This code implements gradient descent to optimize the cross-entropy loss for a binary classification problem. The `sigmoid` function is used to convert the model outputs to probabilities, and the `cross_entropy_loss` function computes the loss. The `gradient_descent` function updates the model weights iteratively to minimize the loss. [3][4]