Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://www.geeksforgeeks.org/machine-learning/optimization-algorithms-in-machine-learning/ below:

Optimization Algorithms in Machine Learning

Optimization algorithms in machine learning are mathematical techniques used to adjust a model's parameters to minimize errors and improve accuracy. These algorithms help models learn from data by finding the best possible solution through iterative updates.

In this article, we'll explore the most common optimization algorithms, understand how they work, compare their advantages, and learn when to use which one.

First-order optimization algorithms are methods that rely on the first derivative (gradient) of the objective function to find the minimum or maximum. They use gradient information to decide the direction and size of updates for model parameters. These algorithms are widely used in machine learning due to their simplicity and efficiency, especially for large-scale problems. Below are some First-Order Algorithms:

Gradient Descent is an optimization algorithm used for minimizing the objective function by iteratively moving towards the minimum. It is a first-order iterative algorithm for finding a local minimum. The algorithm works by taking repeated steps in the opposite direction of the gradient of the function at the current point because it will be the direction of steepest descent.

Let's assume we want to minimize the function f(x)=x² using gradient descent.

• The main function gradient_descent takes the gradient, a starting point, learning rate, number of iterations and a convergence tolerance.
• In each iteration, it calculates the gradient at the current point and updates the point in the opposite direction of the gradient (descent), scaled by the learning rate.
• The update continues until either the maximum number of iterations is reached or the update magnitude falls below the specified tolerance.
• The final result is printed which should be a value close to the minimum of the function.

import numpy as np

# Define the gradient function for f(x) = x^2
def gradient(x):
    return 2 * x

# Gradient descent optimization function
def gradient_descent(gradient, start, learn_rate, n_iter=50, tolerance=1e-06):
    vector = start
    for _ in range(n_iter):
        diff = -learn_rate * gradient(vector)
        if np.all(np.abs(diff) <= tolerance):
            break
        vector += diff
    return vector

# Initial point
start = 5.0
# Learning rate
learn_rate = 0.1
# Number of iterations
n_iter = 50
# Tolerance for convergence
tolerance = 1e-6

# Gradient descent optimization
result = gradient_descent(gradient, start, learn_rate, n_iter, tolerance)
print(result)

Output:

• Stochastic Gradient Descent (SGD): This variant suggests model update using a single training example at a time which does not require a large amount of computation and therefore is suitable for large datasets.
• Mini-Batch Gradient Descent: This method is designed so that it computes it for every mini-batches of data, a balance between amount of time and precision. It converges faster than SGD and is used widely in practice to train many deep learning models.
• Momentum: Momentum improves SGD by adding information of the previous steps of the algorithm to the next step. By adding a portion of the current update vector to the previous update, it enables the algorithm to go through flat areas and noisy gradients to minimize the time to train and find convergence.

Stochastic optimization techniques introduce randomness to the search process which can be advantageous for tackling complex optimization problems where traditional methods might struggle.

• Simulated Annealing: Similar to the annealing process in metallurgy this technique starts with a high temperature (high randomness) that allows exploration of the search space widely. Over time, the temperature decreases (randomness decreases) which helps the algorithm converge towards better solutions while avoiding local minima.
• Random Search: This simple method randomly chooses points in the search space then evaluates them. Random search is actually quite effective particularly for optimization problems that are high-dimensional. The ease of implementation and its ability to work with complex algorithms makes this approach widely used.

When using stochastic optimization algorithms, we consider the following practical aspects:

• Repeated Evaluations: Stochastic optimization algorithms often need repeated evaluations of the objective function which is time-consuming. Therefore, we have to balance the number of evaluations with the computational resources available.
• Problem Structure: The choice of stochastic optimization algorithm depends on the structure of the problem. For example, simulated annealing is suitable for problems with multiple local optima while random search is effective for high-dimensional optimization landscapes.

In evolutionary algorithms we take inspiration from natural selection and include techniques such as Genetic Algorithms and Differential Evolution. They are often used to solve complex optimization problems that are difficult to solve using traditional methods.

Key Components:

• Population: Set of candidate solutions to the optimization problem.
• Fitness Function: A function that evaluates the quality of each candidate solution.
• Selection: Mechanism for selecting the fittest candidates to reproduce.
• Genetic Operators: Operators that modify the selected candidates to create new offspring such as crossover and mutation.
• Termination: A condition for stopping the algorithm.

These algorithms use crossover and mutation operators to evolve the candidate population. It is commonly used to generate solutions to optimization/search problems by relying on biologically inspired operators such as mutation, crossover and selection. In the code example below we implement a Genetic Algorithm to minimize:

f(x) = \sum_{i=1}^{n} x_i^2 ​

• fitness_func returns the negative sum of squares to convert minimization into maximization.
• generate_population creates random individuals between 0 and 1.
• Each generation, the top 50% (fittest) are selected as parents.
• Offspring are created via single-point crossover between two parents.
• Mutation randomly alters one gene with a small probability.
• The process repeats for a fixed number of generations.
• Outputs the best individual and its minimized objective value.

import numpy as np

# Define the fitness function (negative of the objective function)
def fitness_func(individual):
    return -np.sum(individual**2)

# Generate an initial population
def generate_population(size, dim):
    return np.random.rand(size, dim)

# Genetic algorithm
def genetic_algorithm(population, fitness_func, n_generations=100, mutation_rate=0.01):
    for _ in range(n_generations):
        population = sorted(population, key=fitness_func, reverse=True)
        next_generation = population[:len(population)//2].copy()
        while len(next_generation) < len(population):
            parents_indices = np.random.choice(len(next_generation), 2, replace=False)
            parent1, parent2 = next_generation[parents_indices[0]], next_generation[parents_indices[1]]
            crossover_point = np.random.randint(1, len(parent1))
            child = np.concatenate((parent1[:crossover_point], parent2[crossover_point:]))
            if np.random.rand() < mutation_rate:
                mutate_point = np.random.randint(len(child))
                child[mutate_point] = np.random.rand()
            next_generation.append(child)
        population = np.array(next_generation)
    return population[0]

# Parameters
population_size = 10
dimension = 5
n_generations = 50
mutation_rate = 0.05

# Initialize population
population = generate_population(population_size, dimension)

# Run genetic algorithm
best_individual = genetic_algorithm(population, fitness_func, n_generations, mutation_rate)

# Output the best individual and its fitness
print("Best individual:", best_individual)
print("Best fitness:", -fitness_func(best_individual))  # Convert back to positive for the objective value

Output:

Differential Evolution seeks an optimum of a problem using improvements for a solution. It works by bringing forth new candidate solutions from the population through vector addition. DE is generally performed by mutation and crossover operations to create new vectors and replace low fitting individuals in the population.

This code implements the Differential Evolution (DE) algorithm to minimize our previously demonstrated function f(x) = \sum_{i=1}^{n} x_i^2 :

• The differential_evolution function initializes a population of candidate solutions by sampling uniformly within the specified bounds for each parameter.
• For each individual (target vector) in the population, three distinct individuals a, b and c are selected to generate a mutant vector using the formula mutant = a + F⋅(b−c) where F is a scaling factor which controls differential variation.
• A trial vector is created by mixing the target and mutant vectors based on a crossover rate (CR).
• If the fitness of the trial vector is better than that of the target, it replaces the target in the next generation.
• The process repeats for a specified number of generations (max_generations).
• This example uses the sphere_function as the objective where the goal is to minimize the sum of squares of the vector elements and the bounds define a 10-dimensional search space from −5.12 to 5.12.
• After optimization, the code prints the best solution found and its corresponding fitness value.

import numpy as np

def differential_evolution(objective_func, bounds, pop_size=50, max_generations=100, F=0.5, CR=0.7, seed=None):
    np.random.seed(seed)
    n_params = len(bounds)
    population = np.random.uniform(bounds[:, 0], bounds[:, 1], size=(pop_size, n_params))
    best_solution = None
    best_fitness = np.inf
    
    for generation in range(max_generations):
        for i in range(pop_size):
            target_vector = population[i]
            indices = [idx for idx in range(pop_size) if idx != i]
            a, b, c = population[np.random.choice(indices, 3, replace=False)]
            mutant_vector = np.clip(a + F * (b - c), bounds[:, 0], bounds[:, 1])
            crossover_mask = np.random.rand(n_params) < CR
            trial_vector = np.where(crossover_mask, mutant_vector, target_vector)
            trial_fitness = objective_func(trial_vector)
            if trial_fitness < best_fitness:
                best_fitness = trial_fitness
                best_solution = trial_vector
            if trial_fitness <= objective_func(target_vector):
                population[i] = trial_vector
        
    return best_solution, best_fitness

# Example objective function (minimization)
def sphere_function(x):
    return np.sum(x**2)

# Define the bounds for each parameter
bounds = np.array([[-5.12, 5.12]] * 10)  # Example: 10 parameters in [-5.12, 5.12] range

# Run Differential Evolution
best_solution, best_fitness = differential_evolution(sphere_function, bounds)

# Output the best solution and its fitness
print("Best solution:", best_solution)
print("Best fitness:", best_fitness)

Output:

Metaheuristic optimization algorithms are used to supply strategies at guiding lower level heuristic techniques that are used in the optimization of difficult search spaces. Tabu search and iterated local search are two techniques that are used to enhance the capabilities of local search algorithms.

Tabu Search improves the efficiency of local search by using memory structures that prevent cycling back to recently visited solutions. This helps the algorithm escape local optima and explore new regions of the search space.

Key Components:

• Tabu List: A short-term memory structure that stores recently visited solutions or moves. Any move that results in a solution on this list is considered forbidden(tabu) which helps prevent revisiting the same solutions.
• Aspiration Criteria: An override rule that allows the algorithm to accept a tabu move if it results in a solution better than the best known so far.
• Neighborhood Search: At each iteration, the algorithm explores neighboring solutions and selects the best one that is not in the tabu list. If all potential moves are tabu, the best move is chosen based on the aspiration criteria.

Iterated Local Search is another strategy for enhancing local search, but unlike Tabu Search, it does not use memory structures. It relies on repeated application of local search, combined with random changes to escape local minima and continue the search.

Key Components:

• Local Search: Starts with an initial solution and performs local search to find a local optimum.
• Perturbation: Applies a small random change to the current solution, effectively getting it out of its current local optimum.
• Restart Mechanism: The perturbed solution is used as a new starting point for local search. If the newly found solution is better than the current best, it is accepted. If not the search continues with further perturbations.
• Exploration vs. Exploitation: ILS balances exploration (through perturbation) and exploitation (local search), making it simple yet effective for a wide range of optimization problems.

Swarm intelligence algorithms resemble natural systems by using the collective, decentralized behavior observed in organisms like bird flocks and insect colonies. These systems operate through shared rules and interactions among individual agents, enabling efficient problem-solving through cooperation.

There are two of the widely applied algorithms in swarm intelligence:

Particle Swarm Optimization (PSO) is a population-based optimization algorithm inspired by the social behavior of bird flocks and fish schools. Each individual in the swarm (a particle), represents a potential solution. These particles move through the search space by updating their positions based on experience and knowledge shared by neighboring particles. This cooperative mechanism helps the swarm converge toward optimal or near-optimal solutions.

Below is a simple Python implementation of PSO to minimize the Rastrigin function, a common benchmark in optimization problems:

• Each particle has a position, velocity and remembers its personal best position and value.
• Velocity is updated using: Inertia (current movement), Cognitive component (attraction to personal best) and Social component (attraction to global best).
• Position is updated by adding velocity and clipped within bounds [−5.12,5.12].
• The PSO function initializes particles and updates them over 100 iterations.
• In each iteration, it evaluates fitness, updates personal and global bests and moves particles.
• After all iterations, it returns and prints the best solution and its fitness value.

import numpy as np

def rastrigin(x):
    return 10 * len(x) + sum([(xi ** 2 - 10 * np.cos(2 * np.pi * xi)) for xi in x])

class Particle:
    def __init__(self, bounds):
        self.position = np.random.uniform(bounds[:, 0], bounds[:, 1], len(bounds))
        self.velocity = np.random.uniform(-1, 1, len(bounds))
        self.pbest_position = self.position.copy()
        self.pbest_value = float('inf')

    def update_velocity(self, gbest_position, w=0.5, c1=1.0, c2=1.5):
        r1 = np.random.rand(len(self.position))
        r2 = np.random.rand(len(self.position))
        cognitive_velocity = c1 * r1 * (self.pbest_position - self.position)
        social_velocity = c2 * r2 * (gbest_position - self.position)
        self.velocity = w * self.velocity + cognitive_velocity + social_velocity

    def update_position(self, bounds):
        self.position += self.velocity
        self.position = np.clip(self.position, bounds[:, 0], bounds[:, 1])

def particle_swarm_optimization(objective_func, bounds, n_particles=30, max_iter=100):
    particles = [Particle(bounds) for _ in range(n_particles)]
    gbest_position = np.random.uniform(bounds[:, 0], bounds[:, 1], len(bounds))
    gbest_value = float('inf')

    for _ in range(max_iter):
        for particle in particles:
            fitness = objective_func(particle.position)
            if fitness < particle.pbest_value:
                particle.pbest_value = fitness
                particle.pbest_position = particle.position.copy()

            if fitness < gbest_value:
                gbest_value = fitness
                gbest_position = particle.position.copy()

        for particle in particles:
            particle.update_velocity(gbest_position)
            particle.update_position(bounds)

    return gbest_position, gbest_value

# Define bounds
bounds = np.array([[-5.12, 5.12]] * 10)

# Run PSO
best_solution, best_fitness = particle_swarm_optimization(rastrigin, bounds, n_particles=30, max_iter=100)

# Output the best solution and its fitness
print("Best solution:", best_solution)
print("Best fitness:", best_fitness)

Output:

Ant Colony Optimization is inspired by the behavior of ants. Ants find the shortest path between their colony and food sources by laying down pheromones which guide other ants to the path.

Here’s a basic implementation of ACO for the Traveling Salesman Problem (TSP):

• Each ant constructs a complete tour by selecting unvisited cities based on pheromone intensity and inverse distance.
• The transition probability combines pheromone influence ( \alpha ) and heuristic desirability ( \beta ).
• After each iteration, the best tour is updated if a shorter path is found.
• Pheromone levels are globally evaporated (rate \rho ) and reinforced in proportion to the quality (1/length) of each ant’s tour.
• The algorithm iterates over multiple generations to converge toward an optimal or near-optimal solution.
• Returns the shortest tour and its total length discovered during the search.

import numpy as np

class Ant:
    def __init__(self, n_cities):
        self.path = []
        self.visited = [False] * n_cities
        self.distance = 0.0

    def visit_city(self, city, distance_matrix):
        if len(self.path) > 0:
            self.distance += distance_matrix[self.path[-1]][city]
        self.path.append(city)
        self.visited[city] = True

    def path_length(self, distance_matrix):
        return self.distance + distance_matrix[self.path[-1]][self.path[0]]

def ant_colony_optimization(distance_matrix, n_ants=10, n_iterations=100, alpha=1, beta=5, rho=0.1, Q=10):
    n_cities = len(distance_matrix)
    pheromone = np.ones((n_cities, n_cities)) / n_cities
    best_path = None
    best_length = float('inf')

    for _ in range(n_iterations):
        ants = [Ant(n_cities) for _ in range(n_ants)]
        for ant in ants:
            ant.visit_city(np.random.randint(n_cities), distance_matrix)

            for _ in range(n_cities - 1):
                current_city = ant.path[-1]
                probabilities = []
                for next_city in range(n_cities):
                    if not ant.visited[next_city]:
                        pheromone_level = pheromone[current_city][next_city] ** alpha
                        heuristic_value = (1.0 / distance_matrix[current_city][next_city]) ** beta
                        probabilities.append(pheromone_level * heuristic_value)
                    else:
                        probabilities.append(0)
                probabilities = np.array(probabilities)
                probabilities /= probabilities.sum()
                next_city = np.random.choice(range(n_cities), p=probabilities)
                ant.visit_city(next_city, distance_matrix)
        
        for ant in ants:
            length = ant.path_length(distance_matrix)
            if length < best_length:
                best_length = length
                best_path = ant.path

        pheromone *= (1 - rho)
        for ant in ants:
            contribution = Q / ant.path_length(distance_matrix)
            for i in range(n_cities):
                pheromone[ant.path[i]][ant.path[(i + 1) % n_cities]] += contribution

    return best_path, best_length

# Example distance matrix for a TSP with 5 cities
distance_matrix = np.array([
    [0, 2, 2, 5, 7],
    [2, 0, 4, 8, 2],
    [2, 4, 0, 1, 3],
    [5, 8, 1, 0, 6],
    [7, 2, 3, 6, 0]
])

# Run ACO
best_path, best_length = ant_colony_optimization(distance_matrix)

# Output the best path and its length
print("Best path:", best_path)
print("Best length:", best_length)

Output:

Tuning of model parameters that does not directly adapt to datasets is termed as hyperparameter tuning and is a vital process in machine learning. These parameters referred to as the hyperparameters may influence the performance of a certain model. Tuning them is crucial in order to get the best out of the model, as it will theoretically work at its best.

• Grid Search: It is a hyperparameter optimization technique that systematically evaluates all combinations of predefined values. As it ensures the best parameters within the specified grid, it is computationally expensive and time-consuming, making it suitable only when resources are ample and the search space is relatively small.
• Random Search: It selects hyperparameters randomly from specified distributions. Though it may not always find the absolute best values, it often yields near-optimal results more efficiently, especially in high-dimensional or large parameter spaces.

Deep learning models are usually complex and some contain millions of parameters. These models are dependent on optimization techniques that enable their effective training as well as generalization on unseen data. Different optimizers can effect the speed of convergence and the quality of the result at the output of the model.

Common Techniques are:

• Adam (Adaptive Moment Estimation): It is a widely used optimization technique. At each time step, Adam keeps track of both the gradients and their second moments moving average. It is used to modify the learning rate for each parameter in the process. Most of them are computationally efficient, have small memory requirements and are particularly useful for large data and parameters.
• RMSProp (Root Mean Square Propagation): It is designed to adapt the learning rate for each parameter individually. It maintains a moving average of the squared gradients to adjust the learning rate dynamically, helping to stabilize training. By scaling the learning rate according to the magnitude of recent gradients, RMSProp ensures more balanced and efficient convergence.

Now that we have discussed about first order algorithms lets now learn about Second-order optimization algorithms. They use both the first derivative (gradient) and the second derivative (Hessian) of the objective function. The Hessian provides information about the curvature, helping these methods make more informed and accurate updates. Although they often converge faster and more precisely than first-order methods, they are computationally expensive and less practical for very large datasets or deep learning models.

Below are some Second-order algorithms:

Newton's method and quasi-Newton methods are optimization techniques used to find the minimum or maximum of a function. They are based on the idea of iteratively updating an estimate of the function's Hessian matrix to improve the search direction.

Newton's method is applied on the basis of the second derivative in order to minimize or maximize Quadratic forms. It has faster rate of convergence than the first-order methods such as gradient descent but has calculation of second order derivative or Hessian matrix which is a challenge when dimensions are high.

Let's consider the function f(x)=x³−2x²+2 and find its minimum using Newton's Method:

• f_prime(x) is the first derivative f'(x) = 3x^2 - 4x , used to locate critical points.
• f_double_prime(x) is the second derivative f''(x) = 6x - 4 , used to refine convergence and ensure curvature.
• The newtons_method function iteratively updates the estimate using: x_{\text{new}} = x - \frac{f'(x)}{f''(x)} .
• Iteration stops when the step size is below a small threshold (tol) or max_iter is reached.
• Starts at x_0 =3.0 and returns the value of x where a local minimum is achieved.
• Final output shows the estimated value of x where f(x) is minimized.

# Define the function and its first and second derivatives
def f(x):
    return x**3 - 2*x**2 + 2

def f_prime(x):
    return 3*x**2 - 4*x

def f_double_prime(x):
    return 6*x - 4
  
def newtons_method(f_prime, f_double_prime, x0, tol=1e-6, max_iter=100):
    x = x0
    for _ in range(max_iter):
        step = f_prime(x) / f_double_prime(x)
        if abs(step) < tol:
            break
        x -= step
    return x

# Initial point
x0 = 3.0
# Tolerance for convergence
tol = 1e-6
# Maximum iterations
max_iter = 100

# Apply Newton's Method
result = newtons_method(f_prime, f_double_prime, x0, tol, max_iter)
print("Minimum at x =", result)

Output:

Quasi-Newton methods are optimization algorithms that use gradient and curvature information to find local minima, but avoid computing the Hessian matrix explicitly(which Newton's Method does). It has alternatives such as the BFGS (Broyden-Fletcher-Goldfarb-Shanno) and the L-BFGS (Limited-memory BFGS) suited for large-scale optimization due to the fact that direct computation of the Hessian matrix is more challenging.

• BFGS: A method such as BFGS constructs an estimation of the Hessian matrix from gradients. It uses this approximation in an iterative manner where it can obtain quick rates of convergence comparable to Newton’s Method without the necessity to compute the Hessian form.
• L-BFGS: L-BFGS is a memory efficient version of BFGS and suitable for solving problems in large scale. It maintains only a few iterations' updates which results in greater scalability without sacrificing the properties of BFGS convergence.

• Lagrange Multipliers: Additional variables called Lagrange multipliers are introduced in this method so that a constrained problem can be turned into an unconstrained one. It is designed for problems having equality constraints which allows finding out the points where both the objective function and constraints are satisfied optimally.
• KKT Conditions: These conditions generalize those of Lagrange multipliers to encompass both equality and inequality constraints. They are used to give necessary conditions of optimality for a solution incorporating primal feasibility, dual feasibility as well as complementary slackness thus extending the range of problems under consideration in constrained optimization.

Bayesian optimization is a probabilistic technique for optimizing expensive or complex objective functions. Unlike Grid or Random Search, it uses information from previous evaluations to make informed decisions about which hyperparameter values to test next. This makes it more sample-efficient, often requiring fewer iterations to find optimal solutions. It is useful when function evaluations are costly or computational resources are limited.

Logistic Regression is an algorithm for classification of objects and is widely used in binary classification tasks. It estimates the likelihood of an object being in a class with the help of a logistic function. The optimization goal is the cross-entropy which is a measure of the difference between predicted probabilities and actual class labels.

Define and fit the Model

from sklearn.linear_model import LogisticRegression
model = LogisticRegression()
model.fit(X_train, y_train)

Optimization Details:

• Optimizer: Logistic Regression is optimized using iterative algorithms since it lacks a closed-form solution. Common solvers include Newton’s Method, Gradient Descent and its variants selected based on dataset size and sparsity.
• Loss Function: The cost function of the Logistic Regression is the log loss or cross entropy, the calculations are made in order to optimize it.

Evaluation: After training, evaluate the model's performance using metrics like accuracy, precision, recall or ROC-AUC depending on the classification problem.

Linear Regression is an essential method in the regression, as the purpose of the algorithm involves predicting the target variable. The Common goal of optimization model is generally to minimize the Mean Squared Error which represents the difference between the predicted values and the actual target values.

Define and fit the Model

from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(X_train, y_train)

Optimization Details:

• Optimizer: Linear Regression can be solved analytically using the Normal Equation or by using Gradient Descent. For regularized versions (Ridge), solvers like 'lbfgs', 'sag' and 'saga' are employed for efficient optimization.
• Loss Function: The loss function for Linear Regression is the Mean Squared Error (MSE) which is minimized during training.

Evaluation: After training, evaluate the model's performance using metrics like accuracy, precision, recall or ROC-AUC depending on the classification problem.

• Non-Convexity: Cost functions of many machine learning algorithms are non-convex which implies that they have a number of local minima and saddle points. Traditional optimization methods cannot guarantee to obtain the global optimum in such complex models.
• High Dimensionality: Finding optimal solutions in high-dimensional spaces is challenging, the algorithms and computing resources needed to do so can be expensive.
• Overfitting: Regularization neutralizes overfitting which leads to memorization of training data than the new data. The applied model requirements for optimization should be kept as simple as possible due to the risk of overfitting.

Optimization is a component needed for the success of any machine learning models. Proper application of optimization algorithms enables one to boost performance and the accurateness of most machine learning applications.

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4