Coursera_PracticalMachineLearning/WriteUp.html at master · jbregli/Coursera_PracticalMachineLearning · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
<html>

<head>
<style type="text/css">
.knitr.inline {
  background-color: #f7f7f7;
  border:solid 1px #B0B0B0;
}
.error {
	font-weight: bold;
	color: #FF0000;
},
.warning {
	font-weight: bold;
}
.message {
	font-style: italic;
}
.source, .output, .warning, .error, .message {
	padding: 0em 1em;
  border:solid 1px #F7F7F7;
}
.source {
  background-color: #f5f5f5;
}
.rimage.left {
  text-align: left;
}
.rimage.right {
  text-align: right;
}
.rimage.center {
  text-align: center;
}
.hl.num {
  color: #AF0F91;
}
.hl.str {
  color: #317ECC;
}
.hl.com {
  color: #AD95AF;
  font-style: italic;
}
.hl.opt {
  color: #000000;
}
.hl.std {
  color: #585858;
}
.hl.kwa {
  color: #295F94;
  font-weight: bold;
}
.hl.kwb {
  color: #B05A65;
}
.hl.kwc {
  color: #55aa55;
}
.hl.kwd {
  color: #BC5A65;
  font-weight: bold;
}
</style>
<title>Practical Machine Learning Assignment</title>
</head>

<body>

<p>This document sums up what I have done on the programming assignment for the course "Practical Machine Learning" on Coursera</p>

<h1>Step 0:</h1>

<p>Let's first import the library we will use. </p>
<div class="chunk" id="unnamed-chunk-1"><div class="rcode"><div class="source"><pre class="knitr r"><span class="hl kwd">library</span><span class="hl std">(lattice);</span> <span class="hl kwd">library</span><span class="hl std">(ggplot2);</span> <span class="hl kwd">library</span><span class="hl std">(caret);</span><span class="hl kwd">library</span><span class="hl std">(kernlab);</span> <span class="hl kwd">library</span><span class="hl std">(randomForest)</span>
</pre></div>
<div class="message"><pre class="knitr r">## randomForest 4.6-7
## Type rfNews() to see new features/changes/bug fixes.
</pre></div>
</div></div>

<p>And let's fix the seed to make the experience reproductive. </p>
<div class="chunk" id="unnamed-chunk-2"><div class="rcode"><div class="source"><pre class="knitr r"><span class="hl kwd">set.seed</span><span class="hl std">(</span><span class="hl num">32323</span><span class="hl std">)</span>
</pre></div>
</div></div>

<h1>Step 1: Loading the data</h1>

<p>Let's load the train and test datasets.</p>

<div class="chunk" id="unnamed-chunk-3"><div class="rcode"><div class="source"><pre class="knitr r"><span class="hl com"># Load the training dataset:</span>
<span class="hl std">dataset</span>  <span class="hl kwb">&lt;-</span> <span class="hl kwd">read.csv</span><span class="hl std">(</span><span class="hl str">&quot;pml-training.csv&quot;</span><span class="hl std">,</span> <span class="hl kwc">na.strings</span><span class="hl std">=</span><span class="hl kwd">c</span><span class="hl std">(</span><span class="hl str">&quot;NA&quot;</span><span class="hl std">,</span><span class="hl str">&quot;&quot;</span><span class="hl std">),</span> <span class="hl kwc">strip.white</span><span class="hl std">=</span><span class="hl num">TRUE</span><span class="hl std">)</span>
<span class="hl kwd">dim</span><span class="hl std">(dataset)</span>
</pre></div>
<div class="output"><pre class="knitr r">## [1] 19622   160
</pre></div>
<div class="source"><pre class="knitr r"><span class="hl com"># Load the test dataset</span>
<span class="hl std">dataset_test</span> <span class="hl kwb">&lt;-</span> <span class="hl kwd">read.csv</span><span class="hl std">(</span><span class="hl str">&quot;pml-testing.csv&quot;</span><span class="hl std">,</span> <span class="hl kwc">na.strings</span><span class="hl std">=</span><span class="hl kwd">c</span><span class="hl std">(</span><span class="hl str">&quot;NA&quot;</span><span class="hl std">,</span><span class="hl str">&quot;&quot;</span><span class="hl std">),</span> <span class="hl kwc">strip.white</span><span class="hl std">=T)</span>
<span class="hl kwd">dim</span><span class="hl std">(dataset_test)</span>
</pre></div>
<div class="output"><pre class="knitr r">## [1]  20 160
</pre></div>
</div></div>

<p>We have <b>19622 training examples</b> composed of the recording of <b>160 measures</b>. <br/>
We have <b>20 test examples</b>.</p>

<h1>Step 2: Cleaning the data</h1>

<p>By looking at the data we can notice that there are a lot of empty columns.<br/>
Let's remove them. </p>

<div class="chunk" id="unnamed-chunk-4"><div class="rcode"><div class="source"><pre class="knitr r"><span class="hl com"># Cleaning the data:</span>
<span class="hl std">isNA</span> <span class="hl kwb">&lt;-</span> <span class="hl kwd">apply</span><span class="hl std">(dataset,</span> <span class="hl num">2</span><span class="hl std">,</span> <span class="hl kwa">function</span><span class="hl std">(</span><span class="hl kwc">x</span><span class="hl std">) {</span> <span class="hl kwd">sum</span><span class="hl std">(</span><span class="hl kwd">is.na</span><span class="hl std">(x)) })</span>

<span class="hl com"># For the training dataset:</span>
<span class="hl std">dataset</span> <span class="hl kwb">&lt;-</span> <span class="hl kwd">subset</span><span class="hl std">(dataset[,</span> <span class="hl kwd">which</span><span class="hl std">(isNA</span> <span class="hl opt">==</span> <span class="hl num">0</span><span class="hl std">)],</span>
                    <span class="hl kwc">select</span><span class="hl std">=</span><span class="hl opt">-</span><span class="hl kwd">c</span><span class="hl std">(X, user_name, new_window, num_window,</span>
                              <span class="hl std">raw_timestamp_part_1, raw_timestamp_part_2,</span>
                              <span class="hl std">cvtd_timestamp))</span>
<span class="hl kwd">dim</span><span class="hl std">(dataset)</span>
</pre></div>
<div class="output"><pre class="knitr r">## [1] 19622    53
</pre></div>
<div class="source"><pre class="knitr r"><span class="hl com"># For the test dataset:</span>
<span class="hl std">dataset_test</span> <span class="hl kwb">&lt;-</span> <span class="hl kwd">subset</span><span class="hl std">(dataset_test[,</span> <span class="hl kwd">which</span><span class="hl std">(isNA</span> <span class="hl opt">==</span> <span class="hl num">0</span><span class="hl std">)],</span>
                        <span class="hl kwc">select</span><span class="hl std">=</span><span class="hl opt">-</span><span class="hl kwd">c</span><span class="hl std">(X, user_name, new_window, num_window,</span>
                                  <span class="hl std">raw_timestamp_part_1, raw_timestamp_part_2,</span>
                                  <span class="hl std">cvtd_timestamp))</span>
</pre></div>
</div></div>

<p>We have reduced the training set to <b>19622 training examples</b> composed of the recording of <b>53 usable measures</b>.</p>


<h1>Step 3: Creation of the training set and validation set</h1>

<p>In order to measure the performances of our future predictor we split the dataset into a training set and a validation set. The first one will be used to train the predictor, while the second one will be used to assess the performances.<br/>
The split used is: <b>80% </b> for the training set and <b>20%</b> for the validation set.</p>

<div class="chunk" id="unnamed-chunk-5"><div class="rcode"><div class="source"><pre class="knitr r"><span class="hl com"># Spliting the dataset:</span>
<span class="hl std">inTrain</span> <span class="hl kwb">&lt;-</span> <span class="hl kwd">createDataPartition</span><span class="hl std">(dataset</span><span class="hl opt">$</span><span class="hl std">classe,</span> <span class="hl kwc">p</span><span class="hl std">=</span><span class="hl num">0.8</span><span class="hl std">,</span> <span class="hl kwc">list</span><span class="hl std">=</span><span class="hl num">FALSE</span><span class="hl std">)</span>
<span class="hl std">train_set</span> <span class="hl kwb">&lt;-</span> <span class="hl std">dataset[inTrain,]</span>
<span class="hl std">valid_set</span> <span class="hl kwb">&lt;-</span> <span class="hl std">dataset[</span><span class="hl opt">-</span><span class="hl std">inTrain,]</span>
</pre></div>
</div></div>

<h1>Step 4: Training some predictors</h1>

<h2> Test 1: Random Forest:</h1>
<p>We first try an powerful yet easy to parametrize predictor to have an idea of what performance could be obtained. <br/>
We train a <b>Random Forest Classifier</b>.</p>

<div class="chunk" id="unnamed-chunk-6"><div class="rcode"><div class="source"><pre class="knitr r"><span class="hl com"># Test 1: Random forest classifier using cross validation control:</span>
<span class="hl std">ctrl</span> <span class="hl kwb">&lt;-</span> <span class="hl kwd">trainControl</span><span class="hl std">(</span><span class="hl kwc">allowParallel</span><span class="hl std">=</span><span class="hl num">TRUE</span><span class="hl std">,</span> <span class="hl kwc">method</span><span class="hl std">=</span><span class="hl str">&quot;cv&quot;</span><span class="hl std">,</span> <span class="hl kwc">number</span><span class="hl std">=</span><span class="hl num">4</span><span class="hl std">)</span>
<span class="hl std">model</span> <span class="hl kwb">&lt;-</span> <span class="hl kwd">train</span><span class="hl std">(classe</span> <span class="hl opt">~</span> <span class="hl std">.,</span> <span class="hl kwc">data</span><span class="hl std">=train_set,</span> <span class="hl kwc">model</span><span class="hl std">=</span><span class="hl str">&quot;rf&quot;</span><span class="hl std">,</span> <span class="hl kwc">trControl</span><span class="hl std">=ctrl)</span>
<span class="hl std">predictor</span> <span class="hl kwb">&lt;-</span> <span class="hl kwd">predict</span><span class="hl std">(model,</span> <span class="hl kwc">newdata</span><span class="hl std">=valid_set)</span>
</pre></div>
</div></div>

<p>Let's now assess the generalization performance of this classifier by computing the error made on the validation set.</p>

<div class="chunk" id="unnamed-chunk-7"><div class="rcode"><div class="source"><pre class="knitr r"><span class="hl com"># Error on valid_set:</span>
<span class="hl kwd">sum</span><span class="hl std">(predictor</span> <span class="hl opt">==</span> <span class="hl std">valid_set</span><span class="hl opt">$</span><span class="hl std">classe)</span> <span class="hl opt">/</span> <span class="hl kwd">length</span><span class="hl std">(predictor)</span>
</pre></div>
<div class="output"><pre class="knitr r">## [1] 0.9929
</pre></div>
<div class="source"><pre class="knitr r"><span class="hl kwd">confusionMatrix</span><span class="hl std">(valid_set</span><span class="hl opt">$</span><span class="hl std">classe, predictor)</span><span class="hl opt">$</span><span class="hl std">table</span>
</pre></div>
<div class="output"><pre class="knitr r">##           Reference
## Prediction    A    B    C    D    E
##          A 1114    1    1    0    0
##          B    3  750    2    4    0
##          C    0    2  679    3    0
##          D    0    0   10  633    0
##          E    0    1    1    0  719
</pre></div>
</div></div>

<p>We have a validation error of <b>99.29%</b>.<br/>
We can consider this classifier as good.</p>

<p>Let's see what are the prediction for the test set.</p>

<div class="chunk" id="unnamed-chunk-8"><div class="rcode"><div class="source"><pre class="knitr r"><span class="hl com"># Prediction on the test set:</span>
<span class="hl kwd">predict</span><span class="hl std">(model,</span> <span class="hl kwc">newdata</span><span class="hl std">=dataset_test)</span>
</pre></div>
<div class="output"><pre class="knitr r">##  [1] B A B A A E D B A A B C B A E E A B B B
## Levels: A B C D E
</pre></div>
</div></div>

<p>We have the following prediction: <br/>
<b>  B A B A A E D B A A B C B A E E A B B B </b></p>

<h2> Test 2: SVM on lower dimension training examples</h1>

<p>The random forest appears to be a good predictor but it is quite slow to train on the 53 dimensions vectors of our dataset. <br/>
Let's assess the performance of an SVM trained only on the ten most important parameters. <br/>
We train a <b>SVM Classifier</b>.</p>

<div class="chunk" id="unnamed-chunk-9"><div class="rcode"><div class="source"><pre class="knitr r"><span class="hl com"># Most important variable for the ramdom forest predictor:</span>
<span class="hl kwd">varImp</span><span class="hl std">(model)</span>
</pre></div>
<div class="output"><pre class="knitr r">## rf variable importance
##
##   only 20 most important variables shown (out of 52)
##
##                      Overall
## roll_belt              100.0
## pitch_forearm           58.5
## yaw_belt                55.8
## magnet_dumbbell_z       44.7
## pitch_belt              43.8
## roll_forearm            42.7
## magnet_dumbbell_y       42.3
## accel_dumbbell_y        21.8
## accel_forearm_x         17.5
## roll_dumbbell           16.8
## magnet_dumbbell_x       16.0
## magnet_belt_z           14.2
## magnet_forearm_z        14.1
## total_accel_dumbbell    13.5
## accel_belt_z            13.3
## accel_dumbbell_z        13.2
## magnet_belt_y           12.9
## gyros_belt_z            10.8
## magnet_belt_x           10.3
## yaw_arm                 10.3
</pre></div>
<div class="source"><pre class="knitr r"><span class="hl com"># Let us reduce the training dataset to those ten variables</span>
<span class="hl std">dataset_small</span> <span class="hl kwb">&lt;-</span> <span class="hl kwd">subset</span><span class="hl std">(dataset,</span>
                         <span class="hl kwc">select</span><span class="hl std">=</span><span class="hl kwd">c</span><span class="hl std">(roll_belt, pitch_forearm, yaw_belt,</span>
                                  <span class="hl std">magnet_dumbbell_y, pitch_belt,</span>
                                  <span class="hl std">magnet_dumbbell_z, roll_forearm,</span>
                                  <span class="hl std">accel_dumbbell_y, roll_dumbbell,</span>
                                  <span class="hl std">magnet_dumbbell_x,classe))</span>
<span class="hl kwd">dim</span><span class="hl std">(dataset_small)</span>
</pre></div>
<div class="output"><pre class="knitr r">## [1] 19622    11
</pre></div>
</div></div>

<p>We now have <b>19622 training examples</b> composed of the recording of <b>10 measures</b>. (11th parammeter is the class)</p>

<p>Let's train the SVM on this "small" dataset.</p>

<div class="chunk" id="unnamed-chunk-10"><div class="rcode"><div class="source"><pre class="knitr r"><span class="hl com"># Training an SVM on the small dataset</span>
<span class="hl std">svm</span> <span class="hl kwb">&lt;-</span> <span class="hl kwd">train</span><span class="hl std">(classe</span> <span class="hl opt">~</span> <span class="hl std">.,</span> <span class="hl kwc">data</span><span class="hl std">=dataset_small[inTrain,],</span> <span class="hl kwc">model</span><span class="hl std">=</span><span class="hl str">&quot;svm&quot;</span><span class="hl std">,</span> <span class="hl kwc">trControl</span><span class="hl std">=ctrl)</span>
<span class="hl std">predictor_svm</span> <span class="hl kwb">&lt;-</span> <span class="hl kwd">predict</span><span class="hl std">(svm,</span> <span class="hl kwc">newdata</span><span class="hl std">=valid_set)</span>
</pre></div>
</div></div>

<p>Let's now assess the generalization performance of this classifier by computing the error made on the validation set.</p>

<div class="chunk" id="unnamed-chunk-11"><div class="rcode"><div class="source"><pre class="knitr r"><span class="hl com"># Error on valid_set:</span>
<span class="hl kwd">sum</span><span class="hl std">(predictor_svm</span> <span class="hl opt">==</span> <span class="hl std">valid_set</span><span class="hl opt">$</span><span class="hl std">classe)</span> <span class="hl opt">/</span> <span class="hl kwd">length</span><span class="hl std">(predictor_svm)</span>
</pre></div>
<div class="output"><pre class="knitr r">## [1] 0.9845
</pre></div>
<div class="source"><pre class="knitr r"><span class="hl kwd">confusionMatrix</span><span class="hl std">(valid_set</span><span class="hl opt">$</span><span class="hl std">classe, predictor_svm)</span><span class="hl opt">$</span><span class="hl std">table</span>
</pre></div>
<div class="output"><pre class="knitr r">##           Reference
## Prediction    A    B    C    D    E
##          A 1106    3    6    0    1
##          B    6  733   11    9    0
##          C    1    4  675    4    0
##          D    0    3    8  632    0
##          E    0    2    1    2  716
</pre></div>
</div></div>

<p>We have a validation error of <b>98.45%</b>.<br/>
The performance are slightly lower but the training is way faster. <br/>
Depending on the application the trade-off could be interesting</p>

<p>Let's see what are the prediction for the test set.</p>

<div class="chunk" id="unnamed-chunk-12"><div class="rcode"><div class="source"><pre class="knitr r"><span class="hl com"># Prediction on the test set:</span>
<span class="hl kwd">predict</span><span class="hl std">(model,</span> <span class="hl kwc">newdata</span><span class="hl std">=dataset_test)</span>
</pre></div>
<div class="output"><pre class="knitr r">##  [1] B A B A A E D B A A B C B A E E A B B B
## Levels: A B C D E
</pre></div>
</div></div>

<p>We have the following prediction: <br/>
<b> B A B A A E D B A A B C B A E E A B B B </b> <br/>
We have the same prediction with the random forest trained on the full dataset and with the SVM trained with the reduced dataset. Again the trade-off "speed of training versus "classification performances" could be interesting depending on the application.</p>

</body>
</html>