Notes

ML Strategy
1.)Orthogonalisation-in order to achieve something, there must be only one thing to tune. Tuning a hyperparameter must not affect 2 aspects of the model.
2.)Single Evaluation Metric-An optimal model is usually a tradeoff between 2-3 parameters. If we try to optimize all the params and satisfy all the conditions, we might never converge on a single model. If possible combine 2 parameters that can give a weighted average of the optimal parameter(F1 score is a good combination of precision and recall).
3.)Optimizing and satisficing metrics-You optimize a parameter subject to some conditions that just need to be satisfied.
Train and Dev sets-Should come from the distribution.
4.)Any model cannot pass the Bayes error accuracy. Bayes error is close to human error in many cases. If the difference between human level error and train error is not much, it is called an avoidable bias. And the difference between train and test error is variance which can be reduced by regularization etc.
5.)Error Analysis-If the accuracy (or error) on the dev set is less than expected, hand engineering or manually checking the mislabeled examples can prove to be a good idea before deciding the path ahead to deal with the problem. It may turn out after spending months to fix a problem( that you assumed was making a difference), the error comes down only by 1% from 10%. That was clearly not the best investment of your time. Thus error analysis can provide a 'ceiling' for each problem.
6.)DL algorithms are robust to random errors but not to systematic errors.