All a traditional neural network does is a series of matrix operations to transition between an input layer and an output layer – the input layer being a huge vector containing information about all the pixels the image in this case, and the output layer being a binary two-dimensional vector that simply tells us whether we are looking at an image of a face or not.

The matrix operations between layers gradually reduce the size of the input vector until the output vector is reached. But dimensionality is not the only thing that changes. These matrix operations can vary in complexity, and will transform the initial vector multiple times, in multiple different ways, before reaching the output vectors. These consecutive transformations are usually referred to as “hidden layers”. The particular ways in which these matrix operations transform each consecutive vector are decided by the algorithm itself (so to speak) during training. In this phase, the algorithm simply calculates how much its output vector differs from the desired result. Doing so iteratively, it gradually reduces this error by tweaking the matrix operations of each layer.

Convolutional neural networks are not different from this architecture. The only difference is that the matrix operations do not only include dot products and vector additions. We now include a new type of matrix operation: The convolution. To put it simply, the matrix product represents the consecutive application of two transformations, and the matrix convolution represents the transformation of one matrix on the other.

So in summary, convolutional neural networks are just like ordinary neural networks, just that the matrix operations being carried out between each layer are more sophisticated. This of course enhances the performance of these artificial intelligence models.

Convolutional layers in CNNs are designed to mimic the way the human visual cortex processes visual information. The theoretical foundation behind their effectiveness lies in their ability to capture spatial hierarchies and patterns in data. Each convolutional layer applies a set of learnable filters or kernels to the input data, typically an image. These filters perform convolution operations, which involve sliding the filter over the input and computing the dot product between the filter and local regions of the input.

This process allows the network to detect local features such as edges, textures, and shapes in the early layers, and more complex, abstract features in deeper layers. The convolution operation is translation-invariant, meaning it can recognize a feature regardless of its position in the visual field, which is crucial for image recognition tasks. Additionally, convolutional layers use shared weights, significantly reducing the number of parameters compared to fully connected layers. This leads to more efficient training and helps in reducing overfitting.

Pooling layers, often used in conjunction with convolutional layers, further help in making the representation invariant to small translations and reduce the spatial dimensions of the representation, focusing on the most salient features. Overall, convolutional layers work effectively in CNNs by exploiting the spatial structure of the data, enabling the network to learn hierarchically and efficiently from complex visual inputs.

Fully convolutional indicates that the neural network is composed of convolutional layers without any fully-connected layers or MLP usually found at the end of the network. A CNN with fully connected layers is just as end-to-end learnable as a fully convolutional one. The main difference is that the fully convolutional net is learning filters every where. Even the decision-making layers at the end of the network are filters.

A fully convolutional net tries to learn representations and make decisions based on local spatial input. Appending a fully connected layer enables the network to learn something using global information where the spatial arrangement of the input falls away and need not apply.

The pooling layer in a Convolutional Neural Network (CNN) serves several crucial purposes, contributing significantly to the effectiveness and efficiency of the network. Let’s explore the reasons for including pooling layers and why it's not always optimal to directly connect convolutional layers to fully connected layers without pooling.

1. Reduction of Spatial Dimensions

• Decrease in Size: Pooling layers reduce the spatial dimensions (height and width) of the input volume for the next convolutional layer. This downsampling effect reduces the number of parameters and computations in the network, which helps to control overfitting.
• Efficiency: By reducing the number of parameters, pooling layers make the computation more manageable and decrease the computational load, which is essential for training deeper networks.

2. Feature Extraction and Abstraction

• Feature Consolidation: Pooling helps in consolidating the features detected by the convolutional layers. For instance, if a feature is detected in one part of the image, pooling ensures that the spatial variations are less sensitive.
• Abstraction Level: Each pooling step increases the level of abstraction of the features, meaning the network begins to recognize larger patterns instead of focusing on local, fine-grained details.

3. Translation Invariance

• Robustness to Positional Changes: Pooling layers introduce a form of translation invariance, meaning the network becomes less sensitive to the exact location of features in the input. This is crucial for tasks like image classification where the precise location of a feature is less important than its presence.

4. Reduction of Overfitting

• Less Sensitivity to Noise and Variations: By reducing the number of parameters and computations, pooling layers also help in reducing the model's sensitivity to noise and small variations in the input.

5. Improves Learning of Hierarchical Features

• Hierarchical Structure: In CNNs, deeper layers are supposed to learn higher-level features. Pooling helps in this hierarchical learning process by summarizing the presence of features in patches of the input.

Why Not Directly Connect to Fully Connected Layers?

• Too Many Parameters: Without pooling, the size of the feature map remains large, leading to an extremely high number of parameters when connected to fully connected layers. This can cause issues like overfitting and make the network computationally expensive.
• Loss of Spatial Hierarchy: Directly connecting to fully connected layers without pooling can make the network too sensitive to the exact positions of features, reducing the model's ability to generalize from the spatial hierarchy of features.

Conclusion

Pooling layers are therefore integral to the design of CNNs. They help in reducing the computational burden, improving the network's ability to generalize, and facilitating the learning of hierarchical features. While there are CNN architectures that use alternative methods to reduce dimensionality (like strided convolutions), pooling layers remain a simple and effective approach for many applications.

Activation functions in a Convolutional Neural Network (CNN) act like a gateway, deciding what information should go forward into the next layer. Think of them as bouncers at a club, only allowing certain people (or in this case, data) in.

Even in convolutional layers, you do need activation functions. After every convolution operation, the activation function introduces non-linearity into the model, helping it learn from complex data. It's like adding some twists and turns to a straight path, so the model can learn to navigate more complex routes.

Without activation functions, a CNN, no matter how deep, would behave just like a single-layer perceptron because summing these layers would give another linear function. So, yes, you definitely need activation functions in your CNN layers. Keep exploring the realms of machine learning!

First the definition. A fully convolutional CNN (FCN) is one where all the learnable layers are convolutional, so it doesn’t have any fully connected layer.

The key differences between a CNN which has a some convolutional layers followed by a few FC (fully connected) layers and an FCN (Fully Convolutional Network) would be:

• Input image size: If you don’t have any fully connected layer in your network, you can apply the network to images of virtually any size. Because only the fully connected layer expects inputs of a certain size, which is why in architectures like AlexNet, you must provide input images of a certain size (224x224).
• Spatial information: Fully connected layer generally causes loss of spatial information - because its “fully connected”: all output neurons are connected to all input neurons. This kind of architecture can’t be used for segmentation, if you are working in a huge space of possibilities (e.g. unconstrained real images [1]). Although fully connected layers can still do segmentation if you are restricted to a relatively smaller space e.g. a handful of object categories with limited visual variation, such that the FC activations may act as a sufficient statistic for those images [2,3]. In the latter case, the FC activations are enough to encode both the object type and its spatial arrangement. Whether one or the other happens depends upon the capacity of the FC layer as well as the loss function.
• Computational cost and representation power: There is also a distinction in terms of compute vs storage between convolutional layers and fully connected layers that I am a bit confused about. For instance, in AlexNet the convolutional layers comprised of 90% of the weights (~representational capacity) but contributed only to 10% of the computation; and the remaining (10% weights => less representation power, 90% computation) was eaten up by fully connected layers. Thus usually researchers are beginning to favor having a greater number of convolutional layers, tending towards fully convolutional networks for everything.

[1] http://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Long_Fully_Convolutional_Networks_2015_CVPR_paper.pdf

[2] http://papers.nips.cc/paper/5851-deep-convolutional-inverse-graphics-network.pdf

[3] Learning to Generate Chairs, Tables and Cars with Convolutional Networks (PDF) - Semantic Scholar

Shift-Invariant Convolution Neural Network (CNN):

An approach of applying Convolution Neural Networks (CNN) to MNIST may look very similar as below:

Image Source

Consider if you have test image with digit 5 which has been prepossessed using geometric transformation

To make your model generalize better so it can handle such transformation you need shift-invariance CNN. In shift-invariance CNN such prepossessed test images will make no difference in prediction. For a small displacement of object it can generalize it pretty well. For example, cat image below has been displaced but shift invariant model will be able to generalize and predict correctly.

Image Source

Follow up question could be what happens if you rotate an image, for such generalization in network we need Rotational Invariant Convolution Neural Network (CNN). Below is a paper from IEEE CVPR 2017, which solves that problem:

Harmonic Networks: Deep Translation and Rotation Equivariance

Briefly, Harmonic Networks does not use regular Convolution Filters, they replace them with circular harmonic filters which can capture various orientation of a patch.

Website: Harmonic Networks: Deep Translation and Rotation Equivariance
Code: deworrall92/harmonicConvolution
Paper: http://visual.cs.ucl.ac.uk/pubs/harmonicNets/pdfs/worrallEtAl2017.pdf
Video:

Hope that helps
_/\_

Footnotes

Convolutional neural networks work like learnable local filters.

The best example is probably their application to computer vision. The first step in image analysis is often to perform some local filtering of the image, for example, to enhance edges in the image.

You do this by taking the neighborhood of each pixel and convolve it with a certain mask (set of weights). Basically you compute a linear combination of those pixels. For example, if you have a positive weight on the center pixel and negative weights on the surrounding pixels you compute the difference between the center pixel and the surrounding, giving you a crude kind of edge detector.

Now you can either put that filter in there by hand or learn the right filter through a convolutional neural network. If we consider the simplest case, you have an input layer representing all pixels in your image while the output layer representing the filter responses. Each node in the output layer is connected to a pixel and its neighborhood in the input layer. So far, so good. What makes convolutional neural networks special is that the weights are shared, that is, they are the same for different pixels in the image (but different with respect to the position relative to the center pixel). That way you effectively learn a filter, which also turns out to be suited to the problem you are trying to learn.

In a Convolutional Neural Network (CNN), convolutional layers and dense layers play key roles. Think of them as the dynamic duo of a superhero team, each with its own special powers.

A convolutional layer applies a bunch of filters to the input data, detecting patterns like edges, shapes, or textures. It's like a sniffer dog, picking up important features in the data.

On the other hand, the dense layer, or fully connected layer, is where every neuron is connected to every neuron in the next layer. Imagine it as a massive networking event where everyone is connected to everyone else.

The dense layers usually come after the convolutional layers in a CNN. They take all the high-level features learned by the convolutional layers and use them to make a final decision, like determining whether an image is a cat or a dog.

So, from sniffing out patterns to making final decisions, convolutional and dense layers are the heart and soul of a CNN. Dive in and explore them further, they're a fascinating study!

Sure thing! Using different activation functions at each layer of a Convolutional Neural Network (CNN) is like using different tools for different jobs in a workshop.

The advantage is that each activation function can bring its own strengths to the table. It's like how a wrench is good for tightening bolts, while a saw is handy for cutting wood.

Different functions can help the CNN capture different types of patterns in the data. It's like using different tools to create different parts of a piece of furniture.

But the disadvantage is that it can complicate the training of the network. It's like trying to use a dozen tools at once - things can get tricky, and the result may not necessarily be better.

So, like a skilled craftsperson knows when to use which tool, a good data scientist understands when to use which activation function. Keep honing your skills, mate!

Convolutional neural networks (CNN) are a type of deep learning neural networks that are commonly used to classify images. CNNs are known for their ability to reduce computational time and adapt to different variations of images (for example, a well-trained CNN can detect an object from an image even if it is smaller, larger, rotated, translated, etc. from the original image—this is what is known as translation invariance).

As with understanding how any type of neural network works, one needs to understand the theoretical/mathematical side and an application of it to a real-world example.

Below is a basic high-level diagram of a Convolutional Neural Network’s architecture:

The first set in the architecture of a CNN is a set of convolution blocks, which have three components: convolution, ReLU, and pooling.

The first component (convolution) extracts features from the input image (shapes, curves, etc. that can help identify objects in an image). It does this by continuously applying a sliding filter to the image.

On a mathematical level, the convolution feature is derived from multiplying corresponding pairs of values between the current area being processed in the image (highlighted in purple in the animation above) and the kernel/filter (images are represented as a matrix of integer values (colors)). Given an input image [math]f[/math] and a filter/kernel [math]h[/math], any cell’s value with row [math]m[/math] and column [math]n[/math] can be computed by the following formula:

[math]G_{m, n} = (f * h)_{m, n} = \sum_{j} \sum_{k} h_{j, k} f_{m-j, n-k}[/math]

The feature map is then fed to a is ReLU (Rectified Linear Unit). The goal of the ReLU layer is to introduce non-linearity to the network (non-linearity increases the complexity that a neural network can detect). The derivation of the feature map is obviously a linear operation (dot product), so there needs to be some non-linear activation within the CNN. The ReLU’s mathematical function [math]g[/math] given an input [math]z[/math] is:

[math]g(z) = max(0, z)[/math]

In the above picture, the left graph is the result of the ReLU function being applied.

ReLU is a non-linear function, because negative values of z are mapped to 0. However, for positive values of z, ReLU Is a linear function. This is what is known as a piecewise linear function (a hinge function), and this makes the ReLU function ideal:

• It introduces non-linearity ([math]z < 0[/math]), which increases the level of complexity the CNN can detect.
• The ReLU is linear if [math]z > 0[/math], which preserves the speed advantage gradient-based optimization (i.e. gradient descent) has on linear models. I wrote an answer on how gradient descent works here: Quora User's answer to What is an intuitive explanation of gradient descent?

To summarize: ReLU increases the capabilities of the CNN model while still making it fast enough to train.

After ReLU, the model’s data is sent to a pooling layer. The purpose of the pooling layer is to reduce the computational complexity of the CNN by reducing the feature map’s spatial size, as well as to reduce overfitting my selecting the feature map’s most important components. For example: the following 4x4 matrix was reduced to a 2x2 matrix through max pooling:

The most commonly used pooling function is called max pooling. Given a filter size and a stride (how far the filter moves horizontally and vertically) [math](dx, dy)[/math], the max pool function takes the maximum amount from each element in the input (e.x. max(12, 20, 8, 12) = 20, max(30, 0, 2, 0) = 30, max(34, 70, 112, 100) = 112, max(37, 4, 25, 12) = 37)).

The three components (convolution, ReLU, and pooling) are continuously applied to the feature map over and over—CNNs generally have multiple convolution blocks (like a stacked sandwich):

The final pooling layer is flattened into an array or a vector. For example:

[math]\begin{pmatrix} 1 & 2\\ 3 & 4 \end{pmatrix}[/math] will become:
[math] \begin{align} \begin{bmatrix} 1 \\ 2 \\ 3 \\ 4 \end{bmatrix} \end{align}[/math]

The point of the flattening process is to output the probability that a certain feature indicates a certain class. For example, if a CNN was to analyze a car, the probability that a wheel indicates that car should be pretty high. The vector also is in a format that can be fed into a fully connected layer.

The second to last component is the fully connected layers (Fully connected layers in a neural network have all the inputs from the previous layer connected to the activation units of the next layer). The goal of these is to learn non-linear combinations of features derived from the convolution layers. For instance, a car may have many features that define it, such as wheels, a car-like frame, headlights, grill, trunk, etc. These are all individual features with individual probabilities (that the feature belongs to a certain class)—we need to derive a function in that variable space that can detect whether or not a combination is of a certain class.

The final component is the softmax activation: it converts the last layer in the neural network to a probability distribution from 0 to 1. That way, we know exactly how likely an image is of a certain label:

As you can see in the above example, the sum of the outputs of the softmax function add to 1 (0.95 and 0.05—there’s a 95% chance that the image is of a dog and a 5% that the image is of a cat).

The standard softmax function (most commonly used) is the following:

Given an output vector [math]y[/math]: [math]S(y_{i}) = \frac{e^{y_i}}{\sum_j e^{y_j}}[/math]

There’s obviously incredibly complicated mathematics and computational theory that goes behind why all this works, but this is just a basic overview that should be sufficient for practical application/engineering purposes.

Building a Convolutional Neural Network (CNN) without any fully-connected (FC) layers is not only feasible but also practical for certain types of tasks, especially those involving classification and segmentation where the spatial hierarchy of the image is essential. Removing FC layers can lead to a more efficient model in terms of computation and parameter efficiency. Here's how you can design such a CNN:

1. Focus on Convolutional Layers

Start with a series of convolutional layers. These layers will act as the feature extractors, identifying patterns, textures, edges, and other relevant features in the input images. By stacking multiple convolutional layers, the network can learn increasingly complex and abstract features.

2. Utilize Pooling Layers

Incorporate pooling layers (such as max pooling) after some of the convolutional layers to reduce the spatial dimensions of the feature maps. Pooling helps in making the detection of features somewhat invariant to scale and orientation changes, and also reduces the number of parameters, which decreases the computational cost.

3. Apply Global Average Pooling (GAP)

To remove the need for FC layers traditionally used for classification tasks, you can use a Global Average Pooling layer. GAP reduces each feature map to a single number by taking the average of all values in the feature map. If your CNN is aimed at a classification task with N classes, ensure that the last convolutional layer produces N feature maps. Applying GAP will then produce an N-dimensional vector directly corresponding to the class scores.

4. Include Batch Normalization and Activation Functions

Integrate batch normalization layers to help stabilize the learning process and speed up the convergence of the training. After each convolutional layer (and optionally after pooling layers), apply an activation function like ReLU (Rectified Linear Unit) to introduce non-linearity into the model, allowing it to learn more complex patterns.

5. Employ Dropout (Optional)

To prevent overfitting, especially when you have a limited amount of training data, you might consider applying dropout after some of the convolutional or pooling layers. Dropout randomly sets a fraction of input units to 0 at each update during training time, which helps prevent overfitting by making the network's activations more robust.

6. Output Layer

After the Global Average Pooling layer, you might directly output the N-dimensional vector for classification. This vector can be passed through a softmax activation function if you are dealing with a multi-class classification problem to convert the class scores to probabilities.

Architectural Example

Here’s a simplified example architecture for an image classification CNN without fully-connected layers:

• Input Image
• Conv2D + ReLU
• MaxPooling
• Conv2D + ReLU
• MaxPooling
• Conv2D + ReLU
• Global Average Pooling
• Softmax

Advantages

• Parameter Efficiency: This architecture significantly reduces the number of trainable parameters, making the model lighter and faster to train.
• Spatial Information Preservation: Without flattening the feature maps into a vector (as done before FC layers), spatial information is better preserved throughout the network.
• Adaptability: Such models are more adaptable to images of different sizes and are well-suited for tasks like image segmentation and object detection, in addition to classification.

Building a CNN without fully-connected layers is especially beneficial for specific applications where model efficiency and spatial context are crucial. The use of Global Average Pooling to replace FC layers is a powerful strategy to maintain a lean and effective network architecture.

Convolutional neural networks (CNNs) are powerful tools used for a variety of tasks related to computer vision and natural language processing. In identifying an object, CNNs take advantage of convolutional layers which automatically extract features such as texture, color, and edges from the image. This information is then passed through a series of fully-connected layers that help to classify the image according to its content.

Though it can seem like a black box process, let's break down how a CNN identifies an object in more detail:

1. The input layer takes in images or text as input into the network.

2. Next we have convolutional layers which apply different filter sizes with varying strides over each pixel in the image; this allows for feature extraction from otherwise noisy data inputs like pixels on an image. These filters scan across one area of an image at a time to detect certain features; this is known as convolving neural networks (CNNs).

3. As those features pass through several convolutional layers they become increasingly abstract — going from low level (pixel values) all the way up to high level visual representations like textures or objects within your original photo/image/video clip etc..

4. Once these abstract representations have been extracted by CNNs, Max Pooling begins which reduces computational load by reducing dimensions — essentially combining values together and making them more meaningful pieces of data that are easier for our model-building algorithms to use when classifying objects within our images/videos etc..

5. A flattening step then occurs where all extracted abstractions are converted into flattened 1D arrays so they can be fed into our final layer - fully connected deep neural networks which contain many neurons densely interconnected with each other, so patterns between inputted data and desired outputs can be identified in order to accurately classify objects

Here’s another intuition other than the ones already mentioned:

Suppose you have a set of hand-coded rules for a classification task. Then, you can rewrite them in terms of AND and OR operators. For instance, the XOR problem (y = +1 in first and third quadrants, and y = -1 in the second and fourth quadrants) can be written as follows:
- ((x1 > 0) AND (X2 > 0)) OR ((x1 < 0) AND (X2 < 0)) => y=+1
- ((x1 > 0) AND (X2 < 0)) OR ((x1 < 0) AND (X2 > 0)) => y=-1

Now convolutional neural networks have a sequence of alternating convolutional layers and pooling layers.
The convolutional layer acts like an AND operator: the following filter for a grayscale image
+1 -1
0 +1
is analogous to saying that the value of pixel (1,1) is high AND (1,2) is low AND (2,2) is high. An image patch that satisfies these conditions will have a high inner product with this filter, and other patches will have lower inner products.
The pooling layer acts like an OR operator: if the outputs of any of the convolutional filters in the previous layer is high, then max-pooling gives a high output.

A pre-trained convolutional neural network (CNN) is a type of deep learning architecture initially trained on a large dataset, usually on a specific task. This means that the weights and parameters of the network have already been tuned by the data used to train it. Because the network's weights have already been trained, any new data can be quickly classified into classes or clusters identifiable by the pre-trained model.

On the other hand, normal CNNs are shallow networks which require much more training from scratch in order to yield useful results. You need to manually adjust various hyperparameters such as learning rate and layer count before you can begin training your model with your own data set. Another difference between these two types of networks is that normal CNNs do not make use of transfer learning techniques—where an existing pre-trained model is adapted for use with another task—while pre-trained models often rely heavily on these methods.

In conclusion, if you’re looking for quick results without having to spend too much time tuning individual parameters and layers or don't want to start from scratch - then utilizing a pre-trained convolutional neural network (CNN) might be your best option!

A convolution layer in a convolutional neural network (CNN) is the set of neurons (or nodes) that use a mathematical operation called “convolution” to process input data. To understand how it works mathematically, let's begin by defining what a convolution is and how it's used in a CNN.

At its most basic level, a convolution is an operation which takes two functions and produces another function. Mathematically speaking, when two functions f and g are combined through the operation of convolving them they produce another function h: h(x)=(f*g)(x). In the context of CNNs, this means that when one signal or image feature f(x) is applied to one signal or image feature g(x), we can get yet another image feature h(x). This new signal or image feature h(x) will contain information about both f(x) and g(x). These operations form the basis for many machine learning algorithms as well as more sophisticated deep learning models such as ConvNets.

In practice, during training time for Conv Nets, filters are learnt from an initial set of weights. The filter then runs through each layer of neurons calculating dot product at each step within its receptive field; producing different features at each step along with performing down sampling on it – resulting into multiple layers of features being created from small regions from large inputs like images. This whole process helps detect intricate patterns more easily taking into consideration various local structures within data points produced by these filters; thus making them very powerful for detecting objects/faces etc with minimal effort programming wise since AI algorithms themselves can self-learn to create these representations based on visual sense!

You know how in the story, 'Goldilocks and The Three Bears', she tried different bowls of porridge and one was just right? Here, too, it's the same game with CNNs and their input size! Too small, and your CNN won't pick up enough info! Too big, and it becomes overly complex and slow. Just like Goldilocks' perfect porridge, you gotta find the right spot!

The size affects both training speed and recognition accuracy. If the input image is too small, essential features might be lost, hampering the accuracy of the model. Crank up the size, and the CNN sees too much. This leads to longer training times due to increased computation, and your model can also start picking up irrelevant patterns. Essentially, you're overfeeding it, and the CNN loses focus, you get me?

The main idea here is balance. Your goal should be to capture essential features without overwhelming the model or losing critical details. So, start small, then gradually increase the size, observing the CNN's performance at each step. You'll eventually strike gold and find the sweet spot!

Fully Convolutional Networks (FCNs) and traditional Convolutional Neural Networks (CNNs). You see, the old-school CNN - it's a cool cat for image classification. Its deal is to take an input, slap on convolutional layers, then destroy spatial info with densely connected layers, bum-rushing you with a fixed-size vector at the finish line. It's a one-trick pony. Unique images only.

Switch up to the FCN - this babe is all about semantic segmentation. It starts off just like its cousin, going all in on the convolutional layer shebang. Then, plot twist, it heaves the fully connected layers out the window to keep the spatial info intact for output. That means chill, it can handle different image sizes. Really gets the bigger picture, right?

Intermediate layers of Convolutional Neural Networks (CNNs) play a crucial role in the hierarchical learning process essential for image classification tasks. As an image passes through the initial layers of a CNN, these layers tend to learn simple and low-level features, such as edges, colors, and textures. As the image progresses through the network, intermediate layers start to interpret more complex and abstract features by combining the low-level features learned earlier. These layers effectively capture spatial hierarchies and patterns within the image, identifying shapes, structures, or even parts of objects. Through this layered, incremental learning process, intermediate layers help the CNN develop a more nuanced understanding of the various distinctive features within an image, ultimately facilitating efficient and accurate image classification.

Almost all neural networks can be made 'deep'. The distinction between Deep Neural Netowkrs and 'shallow' ones isn't really set in stone. So, we can have a Convolutional Net that is also Deep.

In most cases, deep neural networks can be thought of as having the structure of simple feed-forward networks - it's just that the number of layers is very large.

While convolutional neural networks are also 'feed-forward' (in that they do not have backward connectivity or cycles), they have a special characteristic that sets them apart from other feed-forward nets. Convolutional nets repeat the same set of synpaptic weights over and over again one in a single layer of weights (like a tiling).

For example look at this image:

The two overlapping boxes on the input (the image of the digit '2') will be identical sets of weights that go to distinct neurons in the next layer. This set of weights is tiled over many times, and every tile maps to one neuron in the next layer. This is what we call 'convolution'.

The rationale behind this (especially relevant to image processing) is that the repeated set of weights acts like a repeated feature detector. Repeating that over the entire image allows us to search for the feature in every possible place in the image. It similar to the use of Haar Classifiers in the Viola-Jones Algorithm.

Now, this convolution can be repeated for the next layer, and the one after that, and the one after that...until it becomes a 'deep' neural network. Yann le Cun's first attempt at using CNNs for digit recognition did something similar, and that's why people often get confused between Deep Neural Networks and Convolutional Neural Netowrks:

But in essence, the two are just different approaches - that can be combined together or used exclusively.

The size of the input can have a significant impact on the performance of a convolutional neural network (CNN). The size of the input can affect the network in several ways:

1. Number of parameters: The size of the input directly affects the number of parameters in the network. The larger the input size, the more parameters the network has to learn, which can lead to overfitting if the dataset is not large enough.
2. Computational complexity: The larger the input size, the more computationally expensive it is to perform convolution, pooling, and other operations in the network. This can make the training process slower and can also make it difficult to deploy the network on resource-constrained devices.
3. Feature resolution: The size of the input can affect the resolution of the features that the network is able to learn. A larger input size allows the network to learn finer details in the input, but it also increases the risk of overfitting if the dataset is not large enough. A smaller input size may not capture as much detail in the input, but it can reduce the risk of overfitting.
4. Spatial dimension: In CNNs, the spatial dimension is the height and width of the input image, which affects the number of spatial positions in the input that the network can attend to. A larger input size allows the network to attend to more positions in the input, but it also increases the computational complexity and number of parameters.
5. Data augmentation: The size of the input can also affect the ability to perform data augmentation, which is a technique used to artificially increase the size of the dataset by applying random transformations to the input data. Larger input size allows more flexibility in data augmentation, but it also increases the computational cost.

In summary, the size of the input can have a significant impact on the performance of a CNN. Larger input size allows the network to learn finer details in the input, but it also increases the computational complexity, number of parameters and can lead to overfitting if the dataset is not large enough. The size of the input should be chosen based on the balance between the desired level of detail in the features, the size of the dataset, and the computational resources available.

Thanks for the A2A Ahmed; this is a great question.

Typically CNN-RNN architectures works very well on tasks where:

1. The raw data is well-represented by a (deep) hierarchy of features, which can be modelled using a CNN.
2. And the data we’re working with has temporal properties which we want to model as well — hence the use of a RNN.

One of the most powerful aspects of using a CNN is the ability to effectively model spatial localities using shared-weights for the filters. In the case of images, this means we don’t need to learn, say, an edge detector for every “patch” of the image. Instead, we just have a single (or multiple) edge detector that scans over the entire image using the convolution operator.

And it turns out that the application of this operation is quite general, so we can use it for data besides images. For example, one application is in language understanding at the character-level. If we tried to directly optimize a character-level RNN, we’ll quickly run into problems trying to capture the long-term dependencies in the input sequences. T-h-i-s p-r-o-b-l-e-m s-h-o-u-l-d b-e p-r-e-t-t-y o-b-v-i-o-u-s i-f y-o-u j-u-s-t t-r-y w-o-r-k-i-n-g o-u-t a-n e-x-a-m-p-l-e i-n y-o-u-r h-e-a-d.

Instead, what we want to do is to work with higher-level representations within the RNN — so that the long-term dependencies are easier to capture. In particular, we can interpret a sequence of characters as a 1-D image, which means we can then apply the same convolution technique here as we did for images. Notice that our notions of spatial locality is still preserved — but it's with respect to the location of the characters. Each of these feature detectors could then be used to look for things like common suffixes (e.g. “-ing”, “-ed”) or commonly used connectives (e.g. “as”, “like”, “and”) to “shorten” the length of the dependency across time. This makes the problem of capturing the long-term correlations between characters significantly easier in the RNN.

For a more in-depth treatment on the merits of CNN-RNN models in language understanding, checkout this paper: [1602.02410] Exploring the Limits of Language Modeling

At the end of a CNN, the output of the last Pooling Layer acts as input to the so called Fully Connected Layer. There can be one or more of these layers (“fully connected” means that every node in the first layer is connected to every node in the second layer).

Fully Connected layers perform classification based on the features extracted by the previous layers. Typically, this layer is a traditional ANN containing a softmax activation function, which outputs a probability (a number ranging from 0-1) for each of the classification labels the model is trying to predict.

The figure below shows the end-to-end structure of a simple CNN:

For a more complete and intuitive explanation of all the basic building blocks/end-to-end architecture of a CNN, you can read this blogpost:

Deep Learning Series, P2: Understanding Convolutional Neural Networks

Hope this helps!

The standard reference for CNNs is from 1998/9 by LeCun et al., “Object Recognition with Gradient Based Learning”:

http://yann.lecun.com/exdb/publis/pdf/lecun-99.pdf

Note that Yoshua Bengio is the final author on that paper. Since that time, there have been many improvements and extensions — things like max pooling & batch normalization.

Prior to that time, there were convolutional neural networks by a different name. They were introduced by Kunihiko Fukushima in 1980:

K. Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4): 93-202, 1980.

The neocognitron was based on the idea of simple and complex cells. If you look closely, you will see that the simple cells basically perform a convolution and the complex cells perform average pooling. The neocognitron didn’t catch on for several reasons, including mainly slow performance (at the time), the lack of a “killer app”, and the lack of a community of researchers promoting it. It does not seem that LeCun knew about the neocognitron when he did his work with convolutions.

Jürgen Schmidhüber wrote a historical review of deep learning that is very thorough:

[1404.7828] Deep Learning in Neural Networks: An Overview

But be aware that Schmidhüber’s goal is in that paper is to “correctly” attribute discoveries within deep learning, because he feels that the credit for various contributions has not been allocated correctly before. That is to say, he prefers to emphasize individuals who have been overlooked in the recent popularization of deep learning.

Convolutional Neural Networks (CNNs) and Recursive Neural Networks are quite different.

Convolutional Layer is simply the convolutional operator (some call it filter) applied over a 2d/3d layer.

[math]⋆ [/math]operator is called ‘sliding dot-product’ or ‘cross-correlation’.

This is a nice way to see how these filters work

Please read this page if you want to know how padding, strides and dilation work.

Apart from this there is the max pooling layer. Mathematically, the term "pooling" refers to dimensionality reduction in the context of Convolutional Neural Networks?

Recursive Neural Networks is a tree based architecture. Since it is mostly used to process the sequences for words, it’s best understood in the context of text processing. Let’s say you already have the parse trees for your sentences.

( ( the rat ) ( ate ( cheese ) ) )

In the above example, a simple Tree Long Short-term Memory (LSTM) can take word vectors for individual words and combine them using shared weights (shared across the network) to generate parent nodes. Eventual combined vector can be used to do classification.

Hope this helps. These are technical questions and I always find it hard to put it in words.