Identify the role of a loss, activation or update rule before recalling its expression. Formula recall does not establish that a model is appropriate for a data set.
Where this helps
Reading and implementing machine-learning models with less formula lookup.
Before you start
Try the examples to see what you’ll practice. Check the answer form and any stated conditions.
For J(θ) = θ² at θ = 3, the gradient is 6. With step size η = 0.1:
θ next = 3 − 0.1 × 6 = 2.4.
Use θ next = θ − η∇J(θ), where J is a differentiable loss, ∇J is its gradient, and η > 0 is the step size. In one dimension, a positive derivative means decrease θ, and a negative derivative means increase θ. A sufficiently small move that way lowers the loss. A zero derivative alone gives no decrease direction. A large step can overshoot.
Add weighted inputs and the intercept
Let x = (2, 3), w = (4, −1), and b = 5.
ŷ = 4 × 2 + (−1) × 3 + 5 = 10.
The model is ŷ = w · x + b. Pair each input with its weight, add the products, then add the intercept b once. The vectors w and x must have the same number of entries.
Average the squared errors
Observed values are 2 and 5. Predictions are 3 and 3.
MSE = ((2 − 3)² + (5 − 3)²)/2 = (1 + 4)/2 = 2.5.
Use MSE = Σ(yᵢ − ŷᵢ)²/n for n > 0 pairs. Here yᵢ is an observed value, ŷᵢ is its prediction, and Σ means add all n terms. Squaring keeps opposite errors from canceling.
Turn one score into a value between zero and one
At z = 0, σ(z) = 1/(1 + e⁰) = 1/2.
The sigmoid formula is σ(z) = 1/(1 + exp(−z)), where exp(u) means e raised to the power u. For a finite real score z, the result is strictly between 0 and 1. Positive scores give values above 1/2, and negative scores give values below 1/2. Keep the minus sign in the exponent.
Turn several scores into shares
Suppose two scores have exponentials 2 and 3. Their softmax values are:
2/(2 + 3) = 2/5
3/(2 + 3) = 3/5
For K scores, softmax(z)ᵢ = exp(zᵢ)/Σⱼ₌₁ᴷ exp(zⱼ). Here exp(u) means e raised to the power u. Divide the chosen score’s exponential by the sum for every score. The outputs are positive and add to 1.
Try a few examples
Real practice. Real progress.
Example 1 of 4
complete the gradient descent parameter update
J is differentiable. θ is the parameter vector and η > 0 is the step size.