这是一篇来自澳洲的关于统计机器学习项目1描述的机器学习代写

Overview

The project consists of four parts.

1. Solving the primal problem with stochastic online updates [5 marks]
2. Solving the dual problem with Kernel-Adtron [7 marks]
3. Incorporating kernels [5 marks]

4. Implementing the sequential minimal optimization (SMO) algorithm [8 marks]

All parts are to be completed in the provided Python Jupyter notebook proj1.ipynb.1 Detailed instructions for each task are included in this document.

SVM algorithms. The project’s tasks require you to implement SVM training algorithms by completing provided skeleton code in proj1.ipynb. All of the SVM training algorithms must be implemented as sub-classes of the provided base SVM class. This ensures all SVM training algorithms inherit the same interface, and your implementations must conform to this interface. You may implement functionality in the base SVM class if you desire—e.g., to avoid duplicating common functionality in each sub-class. Your classes may also use additional private methods to better structure your code. And you may decide how to use class inheritance.

Python environment. You must use the Python environment used in workshops to ensure markers can reproduce your results if required. We assume you are using Python ≥ 3.8, numpy ≥ 1.19.0, scikit-learn ≥ 0.23.0 and matplotlib≥ 3.2.0.

Other constraints. You may not use functionality from external libraries/packages, beyond what is imported in the provided Jupyter notebook highlighted here with margin marking – . You must preserve the structure of the skeleton code—please only insert your own code where specified. You should not add new cells to the notebook. You may discuss the SVM learning slide deck or Python at a high-level with others, including via Piazza, but do not collaborate with anyone on direct solutions. You may consult resources to understand SVM conceptually, but do not make any use of online code whatsoever. (We will run code comparisons against online partial implementations to enforce these rules. See ‘academic misconduct’ statement above.)

Submission Checklist

You must complete all your work in the provided proj1.ipynb Jupyter notebook. When you are ready to submit,

follow these steps. You may submit multiple times. We will mark your last attempt. Hint:

­it is a good idea to submit early as a backup. Try to complete Part 1 in the first week and submit it; it will help you understand other tasks and be a fantastic start!

1. Restart your Jupyter kernel and run all cells consecutively.
2. Ensure outputs are saved in the ipynb file, as we may choose not to run your notebook when grading.
3. Rename your completed notebook from proj1.ipynb to username.ipynb where username is your university central username.2

4. Upload your submission to the Project 1 Canvas page.

Marking

Projects will be marked out of 25. Overall approximately 60%, 20%, 20% of available marks will come from correctness,code structure & style, and experimentation. Markers will perform code reviews of your submissions with indicative focus on the following. We will endeavour to provide (indicative not exhaustive) feedback.

1. Correctness: Faithful implementation of the algorithm as specified in the reference or clarified in the specification with possible updates in the Canvas changelog. It is important that your code performs other basic functions such as: raising errors if the input is incorrect, working for any dataset that meets the requirements(i.e., not hard-coded).

2. Code structure and style: Efficient code (e.g., making use of vectorised functions, avoiding recalculation of expensive results); self-documenting variable names and/or comments; avoiding inappropriate data structures,duplicated code and illegal package imports.

3. Experimentation: Each task you choose to complete directs you to perform some experimentation with your implementation, such as evaluation, tuning, or comparison. You will need to choose a reasonable approach to your experimentation, based on your acquired understanding of the SVM learners.

Late submission policy. Late submissions will be accepted to 4 days at −2.5 penalty per day or part day. Weekends and holidays will also be counted towards the late penalty.

Part Descriptions

Part 1: Primal problem with stochastic gradient update [5 marks total]

Your first task is to implement a soft-margin SVM in its primal formulation, using stochastic gradient descent (SGD) for training. This is an online algorithm which is similar to the perceptron training algorithm (see week 6 tutorial),where training involves an outer loop run several times, and an inner loop over each instance in the training set,where the parameters are updated using the gradient of the loss for a single example. The stopping critierion here for the outer is to finish iterations iterations.

At each step, the algorithm will update the weights w and bias b, with update rules:

wt = wt−1 −η∇wLi(w,b)

bt = bt−1 −η∇bLi(w,b)

where η is the learning rate, and Li(w,b) is the loss function for the i th example. You will first have to figure out the per-instance loss, Li(w,b), based on the total loss of the training set,

L(w,b) = nXi=1 max¡ 0,1− yi(w′xi +b)¢ + 12λ∥w∥2

and then derive the relevant gradients needed for the SGD updates. Your will then need to implement a PrimalSVM class with the following functions:

1. fit, which trains using SGD as described above
2. predict, which classifies a new instance x based on sign of w′x+b
3. __init__, to retain state or precompute values to support the above methods

Experiments. Once your PrimalSVM class is implemented, it is time to perform some basic experimentation.

(a) Include and run an evaluation on the given dataset:

psvm = PrimalSVM(eta = 0.1, lambda0 = 0.1)

psvm.fit(X,y,iterations = 100)

print(f”Accuracy is {round(psvm.evaluate(X,y),4)}”)

psvm.visualize(X,y)

Print the training accuracy and plot the dataset and decision surface.

(b) Now you will need to tune your PrimalSVM’s hyperparameters. Based on your understanding of the λ value, test a range of values (consider equally spaced points in logarithmic space) and run experiments to find the best value.3

You must set eta to 0.1 and iterations to 100. Output the result of this strategy—which could be a graph,number, etc. of your choice.

Part 2: Dual SVM formulation with stochastic online update [7 marks total]

In this part, you are to implement the dual formulation of the soft-margin SVM. Your implementation should be done in the provided DualSVM skeleton code, and be based on SGD training algorithm in Table 7.1 of Chapter 7 of the following book:

Cristianini, N., & Shawe-Taylor, J. (2000). An Introduction to Support Vector Machines and Other Kernelbased Learning Methods (pp. 125-148). Cambridge: Cambridge University Press.4

You will need to implement the following functions:

1. __init__
2. fit with the training algorithm described above. For the training loop the same stopping criterion as Part 1, that is stopping after a given number of iterations. The algorithm listed in the paper assumes a fixed bias, however for in this project, you should implement get_bias function to compute b. The standard method for doing so uses the equation: