Hubel and Wiesel (1959, 1962, 1968): Nobel prize 1981 for work on mammalian vision system.   ▷ Results on primary visual cortex (V1).

• V1 is arranged in a spatial map mirroring the structure of the image in the retina.
• V1 has simple cells whose activity is a linear function of the image in a small localized receptive field.
• V1 has complex cells whose activity is invariant to small spatial translations.
• Neurons in V1 respond most strongly to very specific, simple patterns of light, such as oriented bars, but respond hardly to any other patterns.

• A Multilayer perceptron = Feedforward Neural Network is a probabilisitic model: layered matrix-multiplications stacked with non-linear activation functions.
• Optimize log likelihood (classification or regression error) by stochastic gradient descent. Full gradient by backpropagating layer gradients.

• convolution of functions f(t) and w(t) (f∗w)(t)=∫∞−∞dτw(t−τ)f(τ)
• similar to cross correlation of f(t) and w(t) (f⋆w)(t)=∫∞−∞dτw(t+τ)f(τ)

• discrete convolution of functions ft and wt, t∈{1,2,...,D},
  ˜f=∑τwt−τfτ=Wf,˜f,f∈RD where Wtτ=wt−τ, W∈RD×D.

• To obtain ˜f, the same local linear operation is applied to every t and its d neighbors. Here, it's multiplying with w=(−1,1).
• This is meaningful if data has dependencies between degrees of freedom ft that appear independent of the index t and are constraint locally to distance d: data features local patterns.

• Several kernels should learn different local patterns: e.g. edges oriented in different directions.
• So: evidently it's meaningful to use the same kernel for each location t in the input, because patterns appear in the same way across locations t.

1. Read input f.
2. Convolution stage
   ˜f(k):=W(k)f,
   where W(k) is one of K convolution kernels, k=1,...,K.
3. Detector stage ˜f(k)t:=ϕ(˜f(k)t+b) where ϕ is an activation function, b a bias.
4. Pooling stage ˜f(k)t:=maxτ∈[t−d,t+d]˜f(k)t

• Receptive field can grow layer wise.



• Downsampling after pooling layer accounts for reduced information.


• From a Bayesian view, convolutional networks encode our believes about the structure of certain data - as argued up to here - using an infinitely strong prior.

• Why do we put fully connected layers on top of the convolutional layers? For example, if our classification label is translation-invariant, there should be a smarter way?
• Traditionally, CNNs have been used for whole-image classification. Recent work deals with their application to pixelwise classification (object detection, segmentation, tracking), and aims at going beyond and independent treatment of patches.