|title:||EVOLUTIONARY OPTIMIZATION OF KERNEL MACHINES|
|published in:||August 2007|
Master of Science thesis
Man-machine interaction group
Delft University of Technology
|PDF (2600 KB)|
Machine learning techniques have become increasingly more sophisticated and useful over the last
decades. In particular, the class of Kernel Machines has been shown to outperform many other techniques
in various classification and regression problems. Its performance, however, depends to a large extent on
the kernel function and hyperparameters that are being used. Unfortunately, there is no analytic method to
aid the user in finding a proper kernel function and hyperparameters. This selection procedure normally
comes down to a "trial-and-error" approach, which limits the user in the range of kernel functions that can
be considered. In this report we present an automated approach for finding a good kernel function and
optimal hyperparameters using Evolutionary Computation techniques. Evolutionary Computation is class
of optimization techniques that is inspired by biological evolution. Potential solutions to the problem under
consideration are iteratively generated, mutated, recombined, and evaluated. The search process converges
to good solutions, since it biases reproduction toward the fittest individuals (cf. "survival of the fittest").
Two separate evolutionary models are proposed in this report. The first of these two models uses Evolution Strategies to rapidly find optimal hyperparameters. The second model aims to improve the generalization capacity of the machine by evolving complex kernel functions using Genetic Programming. Empirical studies show that our Evolution Strategies approach is able to find competitive hyperparameters, as compared with traditional methods, in less time for regression problems. Classification problems are problematic, due to the discontinuity of the error surface. Nonetheless, it has to be noted that the approach is still able to find reasonable solutions in a time efficient manner. The Genetic Programming approach, however, is shown to improve the generalization capacity of the machine only marginally. In most practical applications this minor improvement will not justify the high computational requirements of the model.
Lastly, we present a study on the reliability of Kernel Target Alignment as a performance measure for Kernel Machines. The main advantage of Kernel Target Alignment is the computational efficiency, as compared with other performance measures. Nonetheless, our empirical study suggests that Kernel Target Alignment does not reliably approximate the true generalization performance of a Kernel Machine. Therefore, the use of this measure as an objective function should be avoided.