Monday, September 22, 2014

Recurrent Neural Networks + AISTATS 2010

  • Recurrent-Neural Networks [X]
  • Unsupervised + Supervised [JW]
    • Erhan, Courville, Bengio, Vincent. Why does unsupervised pre-training help deep learning?. AISTATS 2010 [PDF]


  1. How exactly is unsupervised pre-training done. I am unable to understand how a unsupervised pre train could possibly help a different supervision task.

    From what I deduce from the paper is that the pre training is done only for the first few layers. So my question is what is exactly is the structure of the unsupervised network used in pre training. Is it a sparse auto encoder; that would make sense given that the first few layers anyway encode low level features.

  2. Hi,
    Could someone guide me to other regularization methods used in deep networks,