Activity 16: Neural Networks

In activity 15, we used Linear discriminant analysis (LDA) as predictor to classify chronic lymphocytic leukemia (CLLs) from lymphocytes using two different sets of features, {area perimeter} and {%red %green}. We already know that using perimeter and area as features would yield 100% classification for LDA and minimum distance algorithm while using %red and %green as features yield 90% success rate for LDA. In this activity we used neural network as predictor to classify CLLs and lymphocytes using the %red and %green as features. Similar to activity 15, we used 5 images of both CLLs and lymphocytes as training set and 5 images from both classes as test set. (see fig 2 and fig 3 of activity 14).

Neural network is a computational model that mimics the behavior of biological neurons, the brain for example[1]. Essentially, the idea of neural network is to assign weights on the input (say features), sums them and pass it to an activation function that processes the weighted input into an output. The process of determining the weights of the input is an iterative process with an a priori knowledge of the output. What neural network does is to compare its output with the desired output and adjust the weights of the input depending on the deviation from the desired output. This process is called the learning process of the network. After the network learns how to classify objects based from their features, classification of the test set is straightforward by feeding the network with the objects' features. For detailed discussion on the subject matter, refer to article in [2]

In order to understand more on how neural networks are modeled, fig 1 shows an artificial neuron as modeled by McCulloch and Pits (1943). A connection of many such neurons comprises a network.

Figure 1: Artificial neuron (left) and many connections of an artificial neuron leading to a neural network (right). It typically consists of an input layer followed by a hidden layer and an output layer.

In this activity, we used the ANN (artificial neural network) toolbox of scilab in performing artificial neural network computation. The process is straightforward and the parameter of interest adjusted are the learning rate and the training cycles. Table 1 shows the result of the computation for learning rate of 0.1, i.e., l=(0.1 0) and training cycle of 10000

Table 1: Neural network classification yielding 100% success rate for learning rate=0.1 and training cycle = 10000

We investigate the effect of learning rate and training cycle on the output of the neural network. We used the lymphocytes as test object, if the network's output is close to 1 the classification is correct otherwise it is wrong. For a constant learning rate (i.e 0.01) and increasing training cycle (1000-10000) we expect the output of the network to approach a limiting value as shown in fig 2. This implies that for increasing training cycle, the neural network is improving and is able to classify the object better.

Figure 2: Effect of different training cycle on the neural network's output for a constant learning rate(0.01). As shown, increasing the training cycle increases the output approaching the value 1 indicating that the network is learning better.

For a constant training cycle (1000) , and increasing learning rate (0.1-1), the result also approaches the limiting value of 1 as shown in figure 3.

Figure 3: Effect of different learning rate on the neural network's output for a constant training cycle (1000). As shown, increasing the learning rate increases the output approaching the value 1 indicating that the network is learning better.

In this activity, I give myself a grade of 10 for performing the classification well.

I would like to acknowledge Luis Buno for very useful discussions and for mentioning the idea of plotting the output for different training cycle and learning rate.

References
[1] http://en.wikipedia.org/wiki/Neural_Network#Learning_paradigms
[2] C. Bishop. Neural Networks and Their Applications. Rev. Sci. Instrum. 65(6),
June 1994.
[3] Applied Physics 186 Activity 16 (c)2008 Maricor Soriano