Abstract

Machine learning techniques have become increasingly more sophisticated and useful over the last decades. In particular, the class of Kernel Machines has been shown to outperform many other techniques in various classification and regression problems. Its performance, however, depends to a large extent on the kernel function and hyperparameters that are being used. Unfortunately, there is no analytic method to aid the user in finding a proper kernel function and hyperparameters. This selection procedure normally comes down to a "trial-and-error" approach, which limits the user in the range of kernel functions that can be considered. In this report we present an automated approach for finding a good kernel function and optimal hyperparameters using Evolutionary Computation techniques. Evolutionary Computation is class of optimization techniques that is inspired by biological evolution. Potential solutions to the problem under consideration are iteratively generated, mutated, recombined, and evaluated. The search process converges to good solutions, since it biases reproduction toward the fittest individuals (cf. "survival of the fittest").
Two separate evolutionary models are proposed in this report. The first of these two models uses Evolution Strategies to rapidly find optimal hyperparameters. The second model aims to improve the generalization capacity of the machine by evolving complex kernel functions using Genetic Programming. Empirical studies show that our Evolution Strategies approach is able to find competitive hyperparameters, as compared with traditional methods, in less time for regression problems. Classification problems are problematic, due to the discontinuity of the error surface. Nonetheless, it has to be noted that the approach is still able to find reasonable solutions in a time efficient manner. The Genetic Programming approach, however, is shown to improve the generalization capacity of the machine only marginally. In most practical applications this minor improvement will not justify the high computational requirements of the model.
Lastly, we present a study on the reliability of Kernel Target Alignment as a performance measure for Kernel Machines. The main advantage of Kernel Target Alignment is the computational efficiency, as compared with other performance measures. Nonetheless, our empirical study suggests that Kernel Target Alignment does not reliably approximate the true generalization performance of a Kernel Machine. Therefore, the use of this measure as an objective function should be avoided.