Mercurial > hg > machine-learning-hw2

annotate ex2.m @ 6:141d81a2acf5 default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Remove usage of sum function in cost function (thanks Jeroen Willems)
author Jordi Gutiérrez Hermoso <jordigh@octave.org>
date Tue, 08 Nov 2011 03:30:56 -0500
parents 5664e0047b3e
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
1 %% Machine Learning Online Class - Exercise 2: Logistic Regression
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
2 %
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
3 % Instructions
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
4 % ------------
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
5 %
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
6 % This file contains code that helps you get started on the logistic
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
7 % regression exercise. You will need to complete the following functions
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
8 % in this exericse:
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
9 %
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
10 % sigmoid.m
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
11 % costFunction.m
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
12 % predict.m
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
13 % costFunctionReg.m
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
14 %
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
15 % For this exercise, you will not need to change any code in this file,
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
16 % or any other files other than those mentioned above.
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
17 %
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
18
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
19 %% Initialization
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
20 clear ; close all; clc
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
21
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
22 %% Load Data
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
23 % The first two columns contains the exam scores and the third column
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
24 % contains the label.
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
25
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
26 data = load('ex2data1.txt');
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
27 X = data(:, [1, 2]); y = data(:, 3);
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
28
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
29 %% ==================== Part 1: Plotting ====================
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
30 % We start the exercise by first plotting the data to understand the
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
31 % the problem we are working with.
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
32
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
33 fprintf(['Plotting data with + indicating (y = 1) examples and o ' ...
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
34 'indicating (y = 0) examples.\n']);
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
35
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
36 plotData(X, y);
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
37
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
38 % Put some labels
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
39 hold on;
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
40 % Labels and Legend
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
41 xlabel('Exam 1 score')
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
42 ylabel('Exam 2 score')
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
43
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
44 % Specified in plot order
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
45 legend('Admitted', 'Not admitted')
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
46 hold off;
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
47
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
48 fprintf('\nProgram paused. Press enter to continue.\n');
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
49 pause;
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
50
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
51
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
52 %% ============ Part 2: Compute Cost and Gradient ============
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
53 % In this part of the exercise, you will implement the cost and gradient
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
54 % for logistic regression. You neeed to complete the code in
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
55 % costFunction.m
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
56
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
57 % Setup the data matrix appropriately, and add ones for the intercept term
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
58 [m, n] = size(X);
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
59
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
60 % Add intercept term to x and X_test
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
61 X = [ones(m, 1) X];
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
62
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
63 % Initialize fitting parameters
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
64 initial_theta = zeros(n + 1, 1);
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
65
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
66 % Compute and display initial cost and gradient
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
67 [cost, grad] = costFunction(initial_theta, X, y);
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
68
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
69 fprintf('Cost at initial theta (zeros): %f\n', cost);
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
70 fprintf('Gradient at initial theta (zeros): \n');
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
71 fprintf(' %f \n', grad);
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
72
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
73 fprintf('\nProgram paused. Press enter to continue.\n');
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
74 pause;
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
75
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
76
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
77 %% ============= Part 3: Optimizing using fminunc =============
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
78 % In this exercise, you will use a built-in function (fminunc) to find the
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
79 % optimal parameters theta.
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
80
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
81 % Set options for fminunc
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
82 options = optimset('GradObj', 'on', 'MaxIter', 400);
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
83
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
84 % Run fminunc to obtain the optimal theta
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
85 % This function will return theta and the cost
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
86 [theta, cost] = ...
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
87 fminunc(@(t)(costFunction(t, X, y)), initial_theta, options);
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
88
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
89 % Print theta to screen
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
90 fprintf('Cost at theta found by fminunc: %f\n', cost);
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
91 fprintf('theta: \n');
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
92 fprintf(' %f \n', theta);
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
93
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
94 % Plot Boundary
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
95 plotDecisionBoundary(theta, X, y);
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
96
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
97 % Put some labels
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
98 hold on;
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
99 % Labels and Legend
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
100 xlabel('Exam 1 score')
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
101 ylabel('Exam 2 score')
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
102
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
103 % Specified in plot order
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
104 legend('Admitted', 'Not admitted')
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
105 hold off;
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
106
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
107 fprintf('\nProgram paused. Press enter to continue.\n');
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
108 pause;
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
109
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
110 %% ============== Part 4: Predict and Accuracies ==============
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
111 % After learning the parameters, you'll like to use it to predict the outcomes
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
112 % on unseen data. In this part, you will use the logistic regression model
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
113 % to predict the probability that a student with score 20 on exam 1 and
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
114 % score 80 on exam 2 will be admitted.
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
115 %
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
116 % Furthermore, you will compute the training and test set accuracies of
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
117 % our model.
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
118 %
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
119 % Your task is to complete the code in predict.m
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
120
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
121 % Predict probability for a student with score 45 on exam 1
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
122 % and score 85 on exam 2
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
123
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
124 prob = sigmoid([1 45 85] * theta);
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
125 fprintf(['For a student with scores 45 and 85, we predict an admission ' ...
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
126 'probability of %f\n\n'], prob);
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
127
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
128 % Compute accuracy on our training set
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
129 p = predict(theta, X);
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
130
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
131 fprintf('Train Accuracy: %f\n', mean(double(p == y)) * 100);
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
132
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
133 fprintf('\nProgram paused. Press enter to continue.\n');
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
134 pause;
5664e0047b3e Initial commit
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents:
diff changeset
135