Forward-backward pursuit based sparse extreme learning machine

Abstract

Recently, the Extreme Learning Machine (ELM) becomes an interesting topic in machine learning area. The ELM has been proposed as a new learning algorithm for Single-Hidden Layer Feed forward Networks (SLFNs). The ELM structure has several advantageous such as good generalization performance, extremely fast learning ability and low computational process. Besides this advantageous, the ELM structure has some drawbacks. Firstly, the ELM encounters over-fitting problems because of using a least squares minimization in calculation of the output weights. The other drawback is about accuracy of the ELM. It depends on the number of hidden neurons. This situation is a big challenge in high-dimensional problems for some practical applications. In this paper, we propose a sparse ELM model to overcome the above-mentioned drawbacks. In the sparse ELM model, Forward-Backward Pursuit (FBP) algorithms based on greedy pursuit was used to obtain sparse representation of the output weights. The proposed method which is called FBP-ELM, has several benefits in comparing with the traditional ELM schemes such as avoiding over-fitting, low computational complexity and with adequate number of neurons in hidden layer. FBP-ELM shows its remarkable advantages when it is compared with the empirical studies on commonly used classification benchmarks. Moreover, a comparison with the original ELM and the other regularized ELM schemes such as Least-angle regression (LARS), Least absolute shrinkage and selection operator (LASSO) and Elastic Net is presented to show effectiveness of proposed FBP-ELM method.