I decided to take super-brief notes on the takeaways of the papers I read because I realized I would read it and then promptly forget what I even read.
https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf MLPs One of the first neural net papers! It’s really funny to see how nowadays neural nets and feature embeddings are defaults but back then they were revolutionary ideas. Historical value 10/10 Did I learn anything? 4/10
https://arxiv.org/pdf/1502.03167 Batch norm Really cool paper on batch norm. It addresses the problem of “internal covariate shift” where the input distribution to each layer continues to change, which makes the learning problem very difficult (especially for deep models). Batch norm also addresses excessive saturation of neurons which leads to dead neurons and unhealthy gradient flow. Apparently batch norm also has a nice regularization effect because each input has a bit of stochasticity (depending on the other inputs in the batch) so the model is prevented from overfitting. Overall a great read. But (according to Karpathy) batch norm has a bunch of associated bugs in practice so it’s not preferred. Did I learn anything? 9/10
https://arxiv.org/abs/1502.01852 Kaiming init GREAT paper. I love how clean and well-motivated the math is, and it gives a really clean intuition about nonlinearities, gain, and keeping variance under check during initialization. It’s a motif that keeps coming up (like the sqrt(d_k) in transformers is also for controlling variance!). Did I learn anything? 10/10
https://arxiv.org/abs/1609.03499 Wavenet Good paper. I didn’t really go cover to cover, just looked at the architecture for Makemore P5. It’s a really clean, motivated way to increase context length without exploding parameter count. It also gives a clear connection between compute efficiency, convolutions, and neural nets. Did I learn anything? 7/10
https://arxiv.org/pdf/1512.03385 Residual connections Very cool paper on residual connections, which seem to be everywhere. The motivation seems to be two-sided: (1) residual mappings are usually easier to learn and (2) the residual connections seem to allow for healthier gradient flow. A very interesting point is that even when gradient flow is healthy (nothing vanishing / exploding) the model might still not be good; the plain model with batch norm has good gradient flow, but doesn’t work very well. It needs the additional push of residual connections to make the learning problem easier. Also a good review on convolutions and keeping track of dimensions. Did I learn anything? 9/10
https://arxiv.org/abs/1607.06450 Layer norm
Well-written paper. There were some parts that used information geometry (which I didn’t know was a field of study until this paper!) that I didn’t get; I should definitely revisit after learning more Riemannian geometry. There seems to be three different motivations for the LN layer: (1) batch norms have a bunch of issues, one of which is that it you implement BN in a RNN you need separate normalization parameters for each time value for good performance, but because input length is unbounded for RNNs this is very impractical. (2) RNNs have an issue of exploding/vanishing hidden states, which LNs directly resolve. (3) LNs should still be reasonably good at addressing the covariate shift problem laid out in the BN paper.
https://arxiv.org/abs/1207.0580 Dropout Solid paper with clear-cut explanation. The motivations for dropout can be articulated as: (1) During test time, a model trained with dropout acts like the pseudo-geometric average of an ensemble of 2^N networks, which is more robust to overfitting than just one. (2) Dropout forces each individual neuron to learn useful features rather than a fragile, complex co-adaptation of features (“where a feature detector is only helpful in the context of several other feature detectors”), usually a sign of overfitting. The implications of features learned by each neuron should be robust under changes in other features. Did I learn anything? 7/10
https://arxiv.org/abs/1706.03762 Attention is all you need Really like this paper, might be my favorite. It’s famous for a reason; really clear-cut exposition on the math, what regularization techniques they use, etc. Did I learn anything? 10/10
https://arxiv.org/abs/2105.07576 Problems with batch norm