Training Algorithms for Supervised Machine Learning: Comparative Study

Supervised machine learning is an important task for learning artificial neural networks; therefore a demand for selected supervised learning algorithms such as back propagation algorithm, decision tree learning algorithm and perceptron algorithm has been arise in order to perform the learning stage of the artificial neural networks. In this paper; a comparative study has been presented for the aforementioned algorithms to evaluate their performance within a range of specific parameters such as speed of learning, overfitting avoidance, and their accuracy. Besides these parameters we have included their benefits and limitations to unveil their hidden features and provide more details regarding their performance. We have found the decision tree algorithm is the best as compared with other algorithms that can solve the complex problems with a remarkable speed


LEARNING
The ability to learn new knowledge is a basically trait of intelligence. For learning network, we need to know two things: first we must have a model of environment in which neural network will be operated that means we must know the learning paradigm and second, we must know how the weight of neural network will update and by which rule the weight will be updated [14]. There are three ways to learn neural network (learning paradigm); supervised learning, unsupervised learning and hybrid learning. Each paradigm is working for different tasks based on one of the rules: Error Correction, Boltzmann, Hebbian and Competitive learning [1] [14]. The details are explained and showing in the Table 1.

Supervised Learning
Supervised learning is a machine learning technique that uses to train the artificial neural networks by the value of input and output by findings the relationship between the input (same time call independent variable) and the output (same time call dependent variable), that means making map between input and output for the artificial system that want to learn. Supervised can predict the value of output for system given new input [1] [12] [10] [16]. Supervised learning is a method for training artificial neural networks when the input and output is known, we use supervised learning when we have a set of training data, this training data is a collection of few input data that connected with few output data, this training data it will use by learning algorithms like back propagation and perceptron, which we have discussed about them in section 3 and 4 [3] [6] [7] [12]. The process of training will be done when the neural network will use the input data to produce its own output data and then compares this with the desired output data or the target output data. If there is no difference between an actual output data and the target output data, that means the learning for this data is completed and no need to continue the training for this data, otherwise, the connection weights will be changed by the algorithm of training to reduce the difference between the input and target output data (see fig 1).

BACK-PROPAGATION
The perceptron that uses in today's artificial neural networks was conceived by Rosenblatt in 1950 and he proposed it in 1962 [3] [15]. The back-propagation or (back error-propagation) is the most commonly adopted Multi Layers Perceptron (MLPs) training algorithm, it was first presented by werbos in 1974 and then it was independently re-invented in 1986 by Rumelhart et al. Neural networks are a mathematical model inspired by biological neural networks that shows the set of connected input/output units in which each connection has a weight associated with it, this algorithm is one of the neural used supervised learning method. The learning of neural network will be done by adjusting the weight of neural network [13] [15] [8]. Back propagation uses the target values to calculate the mean square error of the artificial neural networks and level of individual in a population [3] [4] [6].

Fig 2: Architecture of back propagation (MLP)
The back propagation algorithm is the most used algorithms for learning in neural network especially for Multi Layers Perceptron (MLPs) [8].
The algorithm is summarized as follow [18] [19]: Step 1: Initialize all weights for the neural network and bias.
Step 2: Select the input data to the neural network and apply to input layer of network.
Step 3: Activation of the units of the output layer through the network.
Step 4: Calculate the error for the neuron. We mean by errors are the deference between the desired output and neural output.
Calculate what is the output should have been and a scaling factor, how much lower or higher. The output must be adjusted to match the desired output.
Step 5: Reduce the error by changing the weight let W + AB is the new (training) and WAB is initial weight.
( ) Step 6: Finding the error for the hidden layer. Also change the weights for the hidden layers.
Step 7: Repeat the steps from step 2 to step 6 to train a network of any number of layers.

PERCEPTRON ALGORITHM
Perceptron algorithm has proposed in 1958 by Rosenblatt and its one of the neural networks algorithms that are based on error correction rules. Error correction is using the error signal (d-y) to modify the connection weight to gradually reduce the error. Perceptron consists of a single neuron with adjustable weight, wi,j = 1,2,..,n, and threshold U, as shown in (Figure 3) given an input vector x= (x1, x2,..,xn) the net input to neuron is: The output is +1 if V>0 and 0 otherwise [14]. The benefit of perceptron algorithm is its low computational requirement and its ability to guarantee convergence to a solution for linearly separable problem [11].

Fig 3: Architecture of Perceptron
Perceptron learning algorithm is summarized as follow [14]: 1-Initialize the weight and threshold to small random number. 2-Present a pattern vector (x1, x2, ..., xn) t and evaluate the output of the neuron. 3-Update the weight according to: Where d is the desired output, t is the nitration number, and ῃ (0.0< ῃ<1.0) is the gain (step size).

DECISION TREE
The algorithm of decision tree is one of the supervised learning algorithms used like technique machine that build a decision tree from a set of class labeled training sample during the machine learning process. The main purpose of decision tree is the structural information contained in data [9].

CONCLUSION
In this paper we performed a comparative study on various algorithms of supervised learning and we have found the decision tree algorithm is the best as compared with other algorithms that can solve the complex problems with a remarkable speed. As for the back propagation algorithm is the most used algorithm for learning in neural networks especially for multi layers perceptron's (MLPs) and it is very good algorithm for solving simple problems and able to tolerate noisy data. While perceptron algorithm can deal with noisy or incomplete inputs because of their similarity with pure and complete inputs. The benefit of perceptron algorithm is its low computational requirement and its ability to guarantee convergence to a solution for linearly separable problem and Relearning after damage can be relatively quick. [