Neural Computation

CIS 525 – Spring 2004

 

syllabus

class presentation instructions

Useful Reading:

How to give a bad presentation

Ian Parbery, "How to Present a Paper in Theoretical Computer Science: A Speaker's Guide for Students"

 

Week 1 (Jan 21, 2004):

Course Overview

Math background overview: Calculus, Algebra, Probabilities/Statistics (look also at other links from Kari Torkkola)

Chapter 1 from Haykin book (Reading assignment: Ch 1)

Week 2: (Jan 28, 2004):

Chapter 2 from Haykin book (Reading assignment: Ch 2.1 - 2.10)

Machine learning background

Homework 1: Problems 1.1, 1.2, 1.4, 1.5, 1.6, 1.9, 1.11, 1.12, 1.17, 2.1, 2.2, 2.3, 2.4, 2.7, 2.13. Due on Wed, Feb 04

Homework 1.0 (Matlab exercise, for people without Matlab experience. This will not be graded)

Week 3: (Feb 04, 2004):

Chapter 4 from Haykin book (Reading assignment: Ch 4.1-4.8, 4.12, 4.14-4.17)

Linear and Nonlinear Regression, Backpropagation Algorithm

Homework 2: Problems 3.1, 3.2, 4.1, 4.2, 4.5.. Due on Wed, Feb 11

Week 4: (Feb 11, 2004):

Chapter 4 from Haykin book

Learning process, Issues related to multilayer perceptrons

Homework 3:  Due on Wed, Feb 18

Week 5: (Feb 18, 2004):

Chapter 4 from Haykin book:

Network pruning techniques (complexity regularization; weight elimination; Hessian-based network pruning)

Improved methods for training of multilayer perceptrons (Reading assignment: Section 1.2, p.18-28, from "Nonlinear Programming", D Bertsekas; Newton' methods, Quasi-Newton's method, Conjugate-gradient method)

Chapter 5 from Haykin book (Reading assignment: Ch 5.1-5.4):

Radial Basis Functions (structure, about pattern separability)

Week 6: (Feb 25, 2004):

Chapter 5 from Haykin book (Reading assignment: Ch 5.5, p.267-268, p.273-276, Ch-5.6-5.8, Ch. 5.10-5.14):

Radial Basis Functions

Chapter 6  from Haykin book (Reading assignment: Ch 6):

Support Vector Machines

Homework 4: Due on Wed, Mar 03

Week 7: (Mar 03, 2004):

Chapter 6  from Haykin book. 

Support Vector Machines

Chapter 2.13 from Haykin book. 

Bias/Variance Dilemma 

20-minute presentations:

M. J. Orr, J. Hallman, K. Takezawa, A. Murray, S. Ninomiya, M. Oide, and T. Leonard. Combining Regression Trees and RBFs. Int. J. of Neural Systems, 10(6):453--466, 2000. (Presentation by Vladimir Vacic)

Thorsten Joachims. Text Categorization with Support Vector Machines: Learning with Many Relevant Features. In Proc. of the 10th European Conference on Machine Learning, pages 137--142, Chemnitz, Germany, 1998. (Presentation by Meghneel Gore)

Homework 5: Due on Wed, Mar 17

Week 8: (Mar 17, 2004):

Chapter 7 from Haykin book. Reading assignment: Ch 7.1-7.12)

Committees of Networks (simple average, weighted average)

Boosting (boosting by filtering)

Mixtures of Experts

20-minute presentations:

D. DeCoste and D. Mazzoni, Fast Query-Optimized Kernel Machine Classification Via Incremental Approximate Nearest Support Vectors., 20th International Conference on Machine Learning (ICML), 2003. (Presentation by Despina Kontos)

H. Schwenk and Y. Bengio. Boosting Neural Networks. Neural Computation, 12(8):1869-1887, 2000. (Presentation by Yong Li)

Homework 6: Due on Wed, Mar 24

Week 9: (Mar 24, 2004):

Chapter 8 from Haykin book. Reading Assignment: Ch 8.1-8.6, 8.9-8.10)

Principal Component Analysis (statistical approach; relationship with Hebbian learning)

Nonlinear PCA (autoassociative networks; kernel PCA)

Supplementary reading material: B. Scholkopf, A. Smola, and K.-R. Muller. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10:1299 - 1319, 1998.

20-minute presentation:

Weigend, A. S., M. Mangeas, and A. N. Srivastava. Nonlinear Gated Experts for Time Series: Discovering Regimes and Avoiding Overfitting (1995) International Journal of Neural Systems 6, p. 373-399. (Presentation by Bo Han)

Week 10: (Mar 31, 2004):

Midterm

Chapter 9 from Haykin book. (Reading assignment: Ch. 9.)

Self Organizing Maps

About Course Projects

Instructions for Project Proposal (Proposal is due Apr 07, in class)

Pointers to relevant conferences/journals (useful for topic selection):

  • NIPS - Neural Information Processing Systems (Top quality)

  • ICML - International Conference on Machine Learning (Top quality)

  • IJCNN- International Joint Conference on Neural Networks

  • ANNIE - International Conference on Artificial Neural Networks In Engineering

  • ESANN - European Symposium on Neural Networks

  • ICANN - International Conference on Neural Networks

    • 2002, 2003, (2004 will be combined with IJCNN)

  • NNSP - IEEE Workshop on Neural Networks for Signal Processing

    • 2001 , 2002, 2003, 2004 (will be called Machine Learning for Signal Processing).

Homework 7: Due on Wed, Apr 07

Week 11: (Apr 07, 2004):

Independent Component Analysis (Reading Assignment: A. Hyvärinen and E. Oja. Independent Component Analysis: Algorithms and Applications. Neural Networks, 13(4-5):411-430, 2000.

20-minute presentations:

B. Schoelkopf, J. Weston, E. Eskin, C. Leslie and W.S. Noble. "A kernel approach for learning from almost orthogonal patterns." Proceedings of the 13th European Conference on Machine Learning, August 19-23, 2002. pp. 511-528.  (Presentation by Yilian Qin)

E. Oja, J. Laaksonen, M. Koskela, S. Brandt. Self-Organizing Maps for Content-Based Image Database Retrieval. In Kohonen Maps (eds. E. Oja and S. Kaski). July 1999. (Presentation by Nemanja Petrovic)

Homework 8: Due on Wed, Apr 14

Week 12: (Apr 14, 2004):

Markov Chain Monte Carlo Methods (Reading Assignment: R. M. Neal, Probabilistic inference using Markov chain Monte Carlo methods, Technical Report CRG-TR-93-1, Univ. of Toronto, September 1993.)

20-minute presentations:

P.O. Hoyer and A. Hyvärinen. Independent Component Analysis Applied to Feature Extraction from Colour and Stereo Images. Network: Computation in Neural Systems, 11(3):191-210, 2000. (Presentation by Ajay Yadav)

F. R. Bach and M. I. Jordan, Kernel Independent Component Analysis. International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2003. (Presentation by Nagesh Adluru)

Homework 9: Due on Wed, Apr 21

Week 13: (Apr 21, 2004):

Hidden Markov Models (Reading Assignment: L.R. Rabiner, A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition, Proceedings of the IEEE, 77, no. 2, February 1989, pp. 257-285)

Also look at lectures about HMMs (lectures 16-18) from Jaakkola's Machine Learning course

20-minute presentations:

Cao L.J., Tay F.E.H., Support vector machine with adaptive parameters in financial time series forecasting, IEEE Transactions on Neural Networks, 14:6 , 1506 - 1518, 2003. (Presentation by Pooja Hedge)

Liu, B., Lee, W.S., Yu, P.S. and Li, X., Partially Supervised Classification of Text Documents, Proc. 19th Intl. Conf. on Machine Learning, Sydney, Australia, 387-394, 2002. (Presentation by Swetha Nandyala)

Week 14: (Apr 28, 2004):

Topic: TBD

20-minute presentations:

D. Fradkin, D. Madigan: Experiments with random projections for machine learning. KDD 2003: 517-522. (Presentation by Troy Schrader)

A. Krogh. An introduction to hidden Markov models for biological sequences. In S. L. Salzberg, D. B. Searls, and S. Kasif, editors, Computational Methods in Molecular Biology, chapter 4, pages 45-63. Elsevier, Amsterdam, 1998. (Presentation by Qin Jing)