University Homepage School Homepage School Contacts School Search

 

DME Lab Class 4

In this tutorial we will have a brief look at the Logistic Regression and Support Vector Machine capabilites of Weka. As usual, we will use the Landsatt dataset.

1. First off, we'll use the logistic regression classifier. After loading the training data in Weka ("Open file..." in the preprocess tab), go to the classify tab and choose the "Logistic" classifier under the "functions" group.

   Weka, unfortunately, does not support non-linear basis functions for logistic regression, so we'll only try the simple case of learning a linear decision boundary for our classification task.

   Using the default parameters, train the logistic regression classifier and test it using the supplied test data. Observe the outputted classification performance. How does it compare with any other method we've used so far? Is it at least comparable? What does this tell us regarding the linear separability of our dataset? Do you think this performance will be achievable for more complex data?

   Remember that logistic regression tries to "draw" a separating hyperplane between two classes in order to classify unobserved test data. How would you tackle the classification of more than two classes? How do you think Weka does it?

2. Let's now try building a support vector machine classifier. SVMs try to "draw" a decision boundary between classes, in the same fashion as Logistic regression does. However, they try to maximize the margin between the decision boundary and the closest training instances (called Support Vectors). This way, they achieve more robust classification solutions, aiming at better generalization performance.

   On the Classify tab, choose the SMO classifier, under the functions group (SMO stands for Sequential Minimal Optimization, which is an algorithm for training SVMs).

   There are two main parameters to tune here. The complexity parameter c and the kernel type.

   • The complexity parameter controls how "soft" the class margins are, in practice how many instances are used as 'support vectors' to draw the decision boundary.
   • The kernel parameter changes the basis function used to transform the data into a higher dimensional space. The purpose of this is to make the data linearly separable. In simpler terms, it controls the "shape" of the decision boundary.

   Select PolyKernel as the "kernel" parameter. This way we'll try to fit a polynomial decision boundary to our data. Control the degree of the polynomial using the exponent paremeter (1.0, 2.0, 3.0 should be enough). Also, tune the complexity parameter c (not to be confused with the capital "C" parameter of the PolyKernel which should not be changed). Try values 0.01, 0.1, 1, 10, 100 for c.

   Observe the results. You should get a feel for which parameter combinations work better for this dataset. You'll have to tune the SVMs for your Mini-project in a similar way (ideally by using the Experimenter functionality of Weka ;-) ).

3. Now try the RBFkernel. This will use a Gaussian kernel to try and make the data linearly separable. Try different values for its "gamma" parameter (0.01, 0.1, 1, 10). This controls how "wide" your RBF kernel will be (smaller values -> wider gaussian). Combine these values with the different c values used in step 2. Observe which combination gives the best classification performance.

You have now used a whole range of different classification algorithms on the same dataset. Some of them gave great results, others were less good. Some of them give clear explanations of their decisions, others form a black box. Some are easy to optimise and have fast learning times, others are more difficult and time-consuming. These are all things to take into account when you choose classification algorithms for your dataset. Often particular properties of the dataset and the task will have a large influence on which algorithms are best to use. Don't hesitate in your project to try out different algorithms and to use the best elements of each of them.

That's all for the DME lab sessions. Feel free to come to me with any queries and comments either in person, or via email