An encoder is a network (FC, CNN, RNN, etc) that takes the input, and output a feature map/vector/tensor. These feature vector hold the information, the features, that represents the input. The decoder is again a network (usually the same network structure as encoder but in opposite orientation) that takes the feature vector from the encoder, and gives the best closest match to the actual input or intended output.

The encoders are trained with the decoders. There are no labels (hence unsupervised). The loss function is based on computing the delta between the actual and reconstructed input. The optimizer will try to train both encoder and decoder to lower this reconstruction loss.

Once trained, the encoder will gives feature vector for input that can be use by decoder to construct the input with the features that matter the most to make the reconstructed input recognizable as the actual input.

The same technique is being used in various different applications like in translation, generative models, etc.

It is important to know that in actual application, people donot try to reconstruct the actual input, but rather want to map/translate/associate inputs to certain outputs. For example translating french to english sentences, etc.

Encoders and decoders are 2 major techniques in Natural Language Processing using Deep Learning.

Simple terms , ENCODER folds the data to retain imp information and DECODER does the final task.

Won’t get too technical , Let’s take an example of GOOGLE TRANSLATE

Whenever you type a text in english and want it to convert to hindi , Your input will undergo folds ( encoding ) to retain only important information which in-turn is passed to a DECODER.

DECODER helps translate from english to hindi with the help of input from ENCODER.

Note : There is a lot of math involved in this process of encoding & decoding which itself is a major/separate subject in Deep Learning. All these comes under advanced DL.

This was just an overview in layman terms to give an idea how it works.

Thanks fo reading !

The overall structure of sequence to sequence model(encoder-decoder) which is commonly used is as shown below-

It consists of 3 parts: encoder, intermediate vector and decoder.

Encoder-It accepts a single element of the input sequence at each time step, process it, collects information for that element and propagates it forward.

Intermediate vector- This is the final internal state produced from the encoder part of the model. It contains information about the entire input sequence to help the decoder make accurate predictions.

Decoder- given the entire sentence, it predicts an output at each time step.

An encoder/Decoder in deep learning is a technique used mainly in text generation. Mostly nowadays in text translation models. The global method is called “sequence to sequence”

I will now quote a good description I found in a github

“Generally, the encoder encodes the input sequence to an internal representation called 'context vector' which is used by the decoder to generate the output sequence. The lengths of input and output sequences can be different, as there is no explicit one on one relation between the input and output sequences.”

I hope you now understand the concept. An Encoder/Decoder is not an autoencoder, as the other answers stated. These are two different concepts.

Footnotes

Some really helpful answers here, but I have two questions:

Given that encoders can be used by themselves (i.e., without decoders) or combined with various decoders, and decoders can be used by themselves, or combined with various encoders, what is the particular significance of encoder-decoders (a term you see frequently in descriptions of ML systems)? Is a system that includes an encoder-decoder something special beyond just the sum, or union, of what you’d know about that same system just being told it contains an encoder, serving purpose E, and a decoder, serving purpose D? In other words, is an encoder-decoder more than just the sum of its parts?

Regarding encoding, the stuff I’ve seen about encoding talks about the original input as if it has some sort of exalted, handed-down-from-on-high status, but isn’t every representation an “encoding”? For instance, aren’t pixels with 8-bit RGB values just one of an infinite number of possible representations (encodings) for a color image? And a linear sequence of ASCII characters is just one representation for English (even the choice of “text” is a representation [encoding] choice — there’s nothing fundamental about it). When we start talking about “learning a representation”, it seems like we’re really talking learning another, presumably more “suitable” for some purpose, representation of some data that’s already in some other (“initial”) representation.

Usually in a conventional neural network, one tries to predict a target vector y from input vectors x. In an autoencoder network, one tries to predict x from x.

You may think it sounds trivial to learn x from x ...

Sometimes it is and the neural network simply learns to duplicate the training data instead of learning general concepts from the training data. To avoid this problem, often the training set is augmented by creating replicas of observations with added random noise or “jitter”. A single hidden layer autoencoder network with noise added to the training data is known as a denoising autoencoder.

You may also wonder why autoencoder networks are useful ...

Autoencoder networks have been shown to be surprisingly adept at feature extraction. You can use more than one hidden layer to construct a stacked, denoising autoencoder where each consecutive layer in the network can be an optimally weighted, nonlinear, lower dimensional projection of the training data. You can stop there and use autoencoders for feature extraction, or they are also used as a pre-training mechanism for deep supervised networks.

Neural network is nothing but a composite function which can approximate any function that maps inputs to outputs through iterative process which is technically called “Training”.

So,nothing but a basic neural networks gives a output for an input.

But the same when it comes to Autoencoder, we set the output to be same as input, we can call this neural network called as Autoencoder.

Basically,it goes inside the more dense representation of the data. The most commonly used autoencoder is Variational Autoencoder which learns the hidden or latent variable of the input data by going inside the dense part of the input data.

In other words, Autoencoder is used for Dimensionality reduction.

Once we learn the hidden part of the input data for a given problem,it is more sufficient for finding the output.

Source-Google Images

Both GANs and Autoencoders are generative models, which means they learn a given data distribution rather than its density. The key difference is how they do it.

Autoencoders learn a given distribution comparing its input to its output, this is good for learning hidden representations of data, but is pretty bad for generating new data. Mainly because we learn an averaged representation of the data thus the output becomes pretty blurry.

Generative Adversarial Networks take an entirely different approach. They use another network (so-called Discriminator) to measure the distance between the generated and the real data. Basically what it does is distinguishing the real data from the generated. It receives some data as an input and returns a number between 0 and 1. 0 meaning the data is fake and 1 meaning it is real. The generators goal then is learning to convince the Discriminator into believing it is generating real data.

The main advantage of GANs over Autoencoders in generating data is that they can be conditioned by different inputs. For example, you can learn the mapping between two domains: satellite images to google maps

Because of these differences, they can be used for different tasks. Autoencoders are more suitable for compressing data to lower dimensions or generating semantic vectors from it. Where GANs are more suitable for generating data.

Hope I answered your question, in case not, feel free to ask me one :)

Footnotes

I presume that you are well aware of what Machine Learning is, and you're probably thinking Deep Learning is a subset of ML, and pondering what's new in Deep Learning.

Well, Yes and No.

Machine Learning: (Recap)
In a nutshell, ML is field of Computer science that uses statistical (or mathematical) techniques to construct a model (or system) from observed data rather than have user enter specific set of instructions that define the model for that data.

Though the name seems fancy, sometimes it is as simple as linear regression (in very rudimentary form). A little more complex example is spam detector in your mailbox that "learns" which emails are spam, since you never gave instructions for each and every type of email.

Loosely speaking, most often, these algorithms work on precise set of features extracted from your raw data. Features could be very simple, such as pixel values for images, temporal values for a signal, or complex features such as Bag of Words feature representation for text. Most known ML algorithms only work as good as the features represent the data. Correct feature identification is that is close representative of your all states of your data is a crucial step.

What's the big deal about feature extractor ?
Making correct feature extractor is great deal of science in it self. Most of these features extractors (from data) are very specific in function and utility. For ex: for face detection one needs a feature extractor which correctly represents parts of face, resistant to spatial aberrations etc. Each and every type of data and task may have its own class of feature extraction. ( Ex: Speech Recognition, Image Recognition)

These feature extractors can then be used to extract correct data features for a given sample, and pass this information to a classifier/ predictor.

How is Deep Learning different ?
Deep Learning is broader family of Machine Learning methods that tries to learn high level features from the given data. Thus, the problem it solves is reducing task of making new feature extractor for each and every type of data (speech, image etc.)

For last example, Deep Learning algo's will try to learn features such as distance between eyes, length of nose etc. features when image recognition task is presented to them. They may use this info for classification, prediction etc tasks. Thus, this is a major step away from previous "Shallow Learning Algorithms."

Prof. Andrew Ng remarks that Deep Learning focuses on original aim of One Learning, an ideal Algorithm envisioned for an AI.

Long Story Short:
If you write this:
F(1,2,3.......,100) = 5050
and give it to a ML Algo, then that Algo is like the child which immediately understands that RHS is sum of all numbers in LHS. Given new set of numbers F(1,2,3.......,500) , the childwill then sum up all the numbers upto 500.

Then, Deep Learning Algo's are like Carl Fredrich Gauss, which will realize that sum of ith element from forward and backward is always fixed, and can be used to find the sum.

Autoencoders are a type of deep neural networks that map the data to itself through a process of (non-linear) dimensionality reduction, followed by dimensionality expansion. Given X as an m by n matrix (m samples and n dimensions), M_i an n_i by n_{i+1} dimensional matrix, f_i a function R^{n_{i+1}} \to R^{n_{i+1}}, an autoencoder maps the samples by f_p(…f_3(f_2(f_1(XM_1)M_2)M_3)…M_p) etc., where p being the size of the last matrix, and it is equal to n. Usually, some middle layers have smaller number of dimensions than n (i.e., there exists an z where n_z<<n). The set of layers before z encode the information into n_i dimensions, while the layers after that decode the information back. The aim is to find the best values in M_is so that the, for each sample, the encoded and then decoded is the same. That means each sample is mapped to itself. Once optimized, the layer z embeds information in the n dimensions.

Autoencoders can be used for dimensionality reduction (detach the decoder, all n-dimensional samples are encoded into z dimensions) and anomaly detection (normal samples are mapped back to themselves with less error comparing to abnormal instances).

I hope it helps.

Often these terms are used interchangeably which makes it hard to understand the true meaning. I’ll explain here with a real-life scenario.

Scenario :

Let’s say you are in 2199 A.D. and your mom told you to go to market and fetch one kilogram of Alphonso mango. Now you are too exhausted after playing a FIFA99 match and don’t want to go anywhere. So you told your robot to do the work for you and deliver the mangoes to you. So you have to train your robot.

Machine Learning :

Machine Learning is a set of learning algorithm that is being used in today's world as state-of-the-art ways to build an intelligent machine. This set of algorithms are frequently used because they require less data and are easily comprehensible than Deep Learning.

Now coming to the scenario, your robot has reached fruit shop which has apple, mango and orange. Now differentiating between mango from orange, apple etc., is machine learning. For this you can train your robot with size(weight) and color(0 for orange, 1 for red and 2 for yellow), and your robot can fetch mangoes for you.

Deep Learning:

Deep Learning is a sub-type of learning, but here we take the learning to a step deeper. So you’ll need more data and data of more attributes than machine learning.

In this scenario, detecting Alphonso mango from other mango varieties(which looks almost same in size and color) is Deep Learning. Your mom said to bring Alphonso and you can’t bring any mango to complete your job.

AI :

AI in it’s true sense means to mimic human intelligence. And learning is one of the steps of intelligence. What to do with the insights gained from learning is AI.

Now after your robot picked one kilogram of Alphonso it should be able to realize that he has completed the job and he need to return to home with out damaging the mangoes. After reaching home he should deliver it to you, not to your mom so that you can get appraisal and a good mango shake from your mom.

So in one picture you can draw these three domain as follow:

Convolutional Neural networks use a series of hierarchical pooling operations. As Geoff Hinton pointed out, pooling can result in a lot of information about the position of features being thrown away. This means that CNNs are unable to learn the Transformation matrix which relates the position of one part of an image (say a nose), to the position of another part of the image (say, eyes). Obviously, you can't do things like accurate face recognition when such positional information is wasted.

Transforming autoencoders differ from CNNs in that they are designed to explicitly capture the exact position of each feature, so it can learn the overall transformation matrix. The feature detectors are now no longer simple binary activation neurons, but are complex structures which are capable of representing the precise POS (Position, Orientation, and Scale) of each feature. This complex structure is what Hinton calls a 'capsule' . It allows the autoencoder to maintain translational invariance without throwing away important positional information.

There's a lot of evidence that Human Brains work in a similar way, where Cortical columns seem to do complex representation tasks like capsules.

Hmm at this point in time, I don’t think its appropriate anymore (if it ever was) to describe deep autoencoders in such a way. People typically think of deep autoencoders as a superset of deep belief networks (DBNs). I think the source of the confusion here is that back in 2006, Hinton et al. published a highly influential paper on a fast learning algorithm for Restricted Boltzmann Machines (a.k.a. deep belief networks). Basically, this was one of the first step towards being able to effectively train deep neural networks. You would set up a DBN, do unsupervised pre-training to get a decent set of feature representations for the inputs, then fine-tune on the training set to actually get predictions from the network. Same went for autoencoder models. If you wanted to train a deep autoencoder, you would set it up as a DBN and apply the same unsupervised pre-training algorithm. This was during the time before training with GPUs on large datasets was popular, so training a deep network without pretraining tended to be painfully slow.

Obviously, nowadays, we no longer need to formulate deep autoencoders in terms of RBMs in order to be able to effectively train them. We can use more general network topologies for the encoder/decoder such as those based on a CNN or a RNN (take a look at seq2seq) to train a deep autoencoder. So that definition for deep autoencoders is outdated, although the underlying principles for deep autoencoders/DBNs still hold.

An autoencoder compresses its input down to a vector - with much fewer dimensions than its input data, and then transforms it back into a tensor with the same shape as its input over several neural net layers. They’re trained to reproduce their input, so it’s kind of like learning a compression algorithm for that specific dataset.

A GAN looks kind of like an inside out autoencoder - instead of compressing high dimensional data, it has low dimensional vectors as the inputs, high dimensional data in the middle.

Instead of being given a bit of data as input, it’s given a small vector of random numbers. The generator network tries to transform this little vector into a realistic sample from the training data. The discriminator network then takes this generated sample(and some real samples from the dataset) and learns to guess whether the samples are real or fake. Well, more precisely it’s trained to minimize cross entropy between the probability it outputs and a vector of *0s* for fake images and “1”s for real images. The generator learns to make more convincing samples,(or minimize the cross entropy between the discriminators guess about it’s creations and “1”)

Another difference: while they both fall under the umbrella of unsupervised learning, they are different approaches to the problem. A GAN is a generative model - it’s supposed to learn to generate realistic *new* samples of a dataset. Variational autoencoders are generative models, but normal “vanilla” autoencoders just reconstruct their inputs and can’t generate realistic new samples.

The main difference between these two types of neural networks is that autoencoder is an unsupervised type of NN and LSTM is a supervised type of NN( at least for those notions that I met in my experience :) )

Another difference is the kind of tasks they are used for.

Autoencoders are usually used to reduce noise in data or reduce its dimensionality. Me personally did not use that much.

LSTM in its turn main advantage is that it ‘has memory inside’ which is good to use for regression type of solutions when you need to get output in ‘real time’ but with consideration of some historical data of ‘recent’ past.

b.r.

Consider the following example to understand Deep learning and transfer learning.

In your childhood you would have learned to ride a bi-cycle, at that time you learned

1. how to balance the cycle

2. how to pedal it and move forward

(pic courtesy: mentitude )

Fast forward, a few years later when you learned to drive a bike (say harley davidson) you didn’t want to learn how to balance, since your neurons in your brain already knew how to balance. So you had to learn

1. How to accelerate and apply brakes when needed

In next few years you bought a car for yourself and now you had to learn

1. how to steer and accelerate (but not about braking, since you have already know how much braking is necessary)

In the above story when you first learned about “how to ride a bicycle”, you were trying and failing and then making your neurons know how to balance and move forward a bicycle. This is what deep learning is, using the (artificial) neurons to learn something

In the second part of the story, when you wanted to learn “ how to ride a harley davidson”, you didn’t start from the scratch of how to balance a bike since you knew how to balance one, you transferred that learning from your previous experience. Now this is what we call transfer learning. We already tuned our neurons to balance a two-wheeler, now we just need to tune it to balance a harley-davidson

If you understand the above paragraphs then you don’t need an explanation for the third one (of driving a car), you can interpret it by yourself.

“Deep learning is a technique of tuning the artificial neuron to perform a task”

“Transfer learning is a technique of sharing the already learned things, to execute a similar task”

Further reading:

Transfer learning and fine-tuning | TensorFlow Core

Autoassociative neural networks and autoencoders are closely related concepts in the field of neural networks and unsupervised learning. They share similarities but also have distinct characteristics:

Autoassociative Neural Network:

An autoassociative neural network is a general term for a type of neural network that is trained to map input data to itself. In other words, it learns to reproduce its input as closely as possible at the output layer. These networks are typically used for tasks like pattern recognition, data compression, and noise reduction.

Autoassociative networks do not necessarily have a specific architecture; they can be implemented using various neural network architectures, including feedforward, recurrent, or even convolutional neural networks. The primary goal of an autoassociative network is to capture the underlying structure or features of the input data so that it can accurately reconstruct the input when presented with noisy or incomplete versions of it.

Autoencoder:

An autoencoder is a specific type of autoassociative neural network with a well-defined architecture. It consists of an encoder and a decoder:

1. Encoder: The encoder takes the input data and maps it to a lower-dimensional representation called the "encoding" or "latent space." This process involves compressing the input data into a more compact and informative representation.
2. Decoder: The decoder then takes this lower-dimensional representation and attempts to reconstruct the original input data from it.

Autoencoders are often used for dimensionality reduction, feature learning, and data denoising. They have a symmetric architecture, with the encoder and decoder typically having the same or similar structures.

Key Differences:

1. Architecture: The most significant difference is in their architecture. Autoencoders have a specific encoder-decoder structure, whereas autoassociative networks refer to a broader category of networks that can have various architectures.
2. Dimensionality Reduction: Autoencoders are often employed for dimensionality reduction and feature learning, as they learn a compact representation of the data in the latent space. Autoassociative networks may or may not focus on dimensionality reduction as their primary goal.
3. Structured Encoding: Autoencoders aim to learn structured and meaningful representations in the encoding layer. This structured encoding is useful for tasks like generating new data samples or transferring learned features to other tasks.

Join my Quora group where every day I publish my top trading signals (that I personally use), based on technical and sentiment models! A weekly track record is kept to evaluate the progress. Subscribers also get a free copy of my book (the subscription ranges from free to $0.83/month).

Data compression is a big topic that’s used in computer vision, computer networks, computer architecture, and many other fields. The point of data compression is to convert our input into a smaller representation that we recreate, to a degree of quality. This smaller representation is what would be passed around, and, when anyone needed the original, they would reconstruct it from the smaller representation. Autoencoders are unsupervised neural networks that use machine learning to do this compression for us. the aim of an autoencoder is to learn a compressed, distributed representation for the given data, typically for the purpose of dimensionality reduction.

Now we already have principal component analysis for that then why do we need autoencoders?

An autoencoder can learn non-linear transformations, unlike PCA, with a non-linear activation function and multiple layers.

• An autoencoder doesn’t have to learn dense layers, it can use convolutional layers to learn too, which could be better for video, image and series data.
• It may be more efficient, in terms of model parameters, to learn several layers with an autoencoder rather than learn one huge transformation with PCA.
• An autoencoder gives a representation as the output of each layer, and maybe having multiple representations of different dimensions is useful.
• An autoencoder could let you make use of pretrained layers from another model, to apply transfer learning to prime the encoder/decoder.

Despite the fact, the practical applications of autoencoders were pretty rare some time back, today data denoising and dimensionality reduction for data visualization are considered as two main interesting practical applications of autoencoders. With appropriate dimensionality and sparsity constraints, autoencoders can learn data projections that are more interesting than PCA or other basic techniques. It also provides a more accurate output when compared to PCA.

Autoencoders are simple learning networks that aim to transform inputs into outputs with the minimum possible error. This means that we want the output to be as close to input as possible.

You can also refer to the following video to know more about Autoencoders. Hope you like it!

An autoencoder neural network is an unsupervised Machine learning algorithm that applies backpropagation, setting the target values to be equal to the inputs.

Autoencoders although is quite similar to PCA but its Autoencoders are much more flexible than PCA. Autoencoders can represent both liners and non-linar transformation in encoding but PCA can only perform linear transformation. Autoencoders can be layered to form deep learning network due to it’s Network representation.

Now lets have a look at the components of autoencoders.

There are three main layers : encoder, code and decoder.

Encoder- This is the part of the network that compresses the input into a latent space representation. The encoder layer encodes the input image as a compressed representation in a reduced dimension. The compressed image typically looks garbled, nothing like the original image

Code- The next component represents the latent space. Code is the part of the network that represents the compressed input fed to the decoder.

Decoder- This layer decodes the encoded image back to the original dimension. The decoded image is a lossy reconstruction of the original image. It reconstructs the input from the latent space representation.

Some of the properties of autoencoders are:

1. Data-specific : Autoencoders are only able to compress data similar to what they have been trained on. An autoencoder which has been trained on human faces would not be performing well with images of modern buildings. This improvises the difference between autoencoders and MP3 kind of compression algorithms which only hold assumptions about sound in general, but not about specific types of sounds.

2. Lossy : This means that the decompressed outputs will be degraded compared to the original inputs. Just like what you see in JPEG or MP3.

3. Learned automatically from examples : If you have appropriate training data, it is easy to train specialized instances of the algorithm that will perform well on a specific type of input. It doesn’t require any new engineering.

Applications of Autoencoders:

Image Coloring - Autoencoders are used for converting any black and white picture into a colored image. Depending on what is in the picture, it is possible to tell what the color should be. E.g. The leaves of trees are generally green, the sky is blue, clouds are white, etc. All that is needed to be done is to make a computer be able to do this and that is where autoencoders step in.

Feature variation- it extracts only the required features of an image and generates the output by removing any noise or unnecessary interruption.

Dimensionality Reduction - Using autoencoders, we receive the same image as the input but with reduced dimensions. It helps in providing the similar image along with a reduced pixel value.

Denoising Image - During the training, the autoencoder learns to extract important features from input images and ignores the image noises because the labels have no noises. the input seen by the autoencoder is not the raw input but a stochastically corrupted version. A denoising autoencoder is thus trained to reconstruct the original input from the noisy version.

Watermark Removal - It is also used for removing watermarks from images or to remove any object while filming a video or a movie.

Autoencoders are unsupervised deep learning neural network algorithms that reduce the number of dimensions in the data to encode it.

The clue is in the name really, autoencoders encode data.

Once the data has been encoded through the algorithm, it is then decoded on the other side. The system is completed when the data on both sides of the encoding matches.

Three examples of applications of AutoEncoders are given below:

• Data Storage: The encoding processes are able to compress down large quantities of data, compressing it. This process, as you can imagine, has big benefits for data storage at scale.
• Feature detection: The process used to encode the data identifies features of the data that can be used to identify it. This list of features is used in multiple systems to understand the data. (Convolutional Neural Networks also do feature detection in images)
• Recommendation systems: One application of autoencoders is in recommendation systems. These are the systems that identify films or TV series you are likely to enjoy on your favorite streaming services.

You can use the pytorch libraries to implement these algorithms with python.

And neither is implementing algorithms!

The process is similar to implementing Boltzmann Machines. This is because you have to create a class that will then be used to implement the functions required to train your autoencoder.

Before we jump into programming an AutoEncoder step by step, let’s first take a look at the theory.

A GENTLE OVERVIEW OF THE MATHEMATICS AND THEORY OF AUTOENCODERS

You can think of an AutoEncoder as a bottleneck system.

For example, you can take a dataset with 20 input variables. These variables are encoded into, let’s say, eight features. Then decoded on the other side back to 20 variables.

What you need to remember about autoencoders is that they are trained on the basis that they are trying to replicate the input data with the output.

You take an input of data. This data is encoded via the neural network.

When the data is encoded, the input is transformed into a set of features. These features are identified by the encoder to act as markers for decoding the output at the next stage.

Next, the data is decoded. Then the difference between input and output calculated. Finally, the error is back-propagated through the system.

The autoencoder updates the algorithm and tries again.

Forward and back until a solution is found

Once the variables on each side match, the cycle ends.

The structure of the autoencoder becomes important when you are looking to leverage the encoded middle of the neural network to solve a problem.

Say for example you are trying to understand specific features that are present in your data set. These features are important for allowing you to better understand what the data is telling you.

This is where the bottleneck idea becomes relevant.

To learn more about autoencoders and how they work, try this article.

I’ll start with this image I found on the internet:

As you can see, deep learning is a subset of machine learning, which itself is a subset of Artificial Intelligence.

Artificial Intelligence refers to the concepts involved in making machines being able to act intelligent on their own, just as humans do.

Machine learning comprises of the actual techniques or algorithms which enable machines to do so.

Some of the famous algorithms in Machine Learning are: Linear/Logistic Regression, Support Vector Machines, Classification and Regression Trees, Artificial Neural Networks, etc.

Don’t worry if you don’t know the above algorithms yet.

One of these techniques is Deep Learning.

Before understanding what that is, you need to know what an Artificial Neural Network (ANN) looks like (You don’t need to know it’s actual working mechanism for now).

An ANN has an input layer, some hidden layers and an output layer.

When an ANN has a lot of these hidden layers, it is called a Deep Neural Network.

The large number of intersecting lines that you see in the above diagram connect each of the circular units from one layer to the next layer.

And the process of ‘training’ a Deep Neural Network is called Deep Learning.

There you go.

Hope it helped!

A deep autoencoder is composed of two, symmetrical deep-belief networks that typically have four or five shallow layers representing the encoding half of the net, and second set of four or five layers that make up the decoding half.

At the core of machine learning sit two essential concepts, deep learning, and transfer learning. Each holds a unique role, working in concert to advance the field.

Journeying Through the Layers of Deep Learning

Taking its place as a specialized subset of machine learning, deep learning is known for its utilization of intricate neural networks. These networks boast multiple layers (therein laying the 'deep' aspect) nestled between input and output tiers. Their purpose? To master the art of learning data representations at varying levels of abstraction, evolving from simplicity to complexity. By employing this kind of hierarchical learning, deep learning models emerge as champions in arenas such as image and speech recognition, and natural language processing.

Unleashing Potential with Transfer Learning

While deep learning is a form of machine learning, transfer learning is more of a technique applied within machine learning, inclusive of deep learning. It thrives on the concept of applying knowledge garnered from one task to enhance performance on another related task, especially when the data for the latter task is limited.

To paint a clearer picture, consider a deep learning model, well-trained in distinguishing various dog breeds - a task requiring a hefty data load. This model could then be fine-tuned through transfer learning to discern specific Labrador varieties - a more niche, data-scarce task, yet related.

Deep Learning and Transfer Learning: A Partnership with Purpose

Deep learning and transfer learning play off each other, creating a vital dynamic in contemporary AI. Deep learning signifies a certain model design and learning mechanism, while transfer learning acts as an efficient tactic to exploit these models across diverse tasks. Think of it as a comparison between a car's engine (deep learning) and a driving technique (transfer learning). Both elements interact harmoniously, each serving its unique purpose. In this mutually beneficial relationship, deep learning brings robust complexity to the table, while transfer learning delivers an effective means to channel that complexity across a spectrum of tasks. AI practitioners push the boundaries of the LLM world through cooperation in communities by using TL and DL to grapple with the enigma of mathematical creativity.

Interesting question. Omer has shown some differences between these two models. I would like to add some connections/similarities between them.

Let's focus on skip-gram here. word2vec takes a word as input, and predicts another word (usually contexts, but can also be other features). RNN encoder-decoder takes a sentence as input, and predicts another sentence.

As Omer mentioned, RNN encoder-decoder is usually trained on bilingual data, e.g. input an English sentence and output a French sentence. However, this condition is not required/assumed. We can also feed the encoder and decoder with monolingual sentences, e.g. given a sentence within a document, predict the sentences before and after, which results in the so-called skip-thought vectors ([1506.06726] Skip-Thought Vectors).
The skip-thought vectors can be simply viewed as a natural extension of word2vec to RNN encoder-decoder, which uses RNN for intra-sentence modeling and word2vec for inter-sentence modeling.

word2vec is by itself an encoder-decoder, targeted on words. It can be readily extended to processing more complicated units, with the help of some specific composition strategies such as RNN, which we call encoder and decoders.

I think there are lots of Pros & Cons for autoencoder(AE) over restricted Boltzman Machine (RBM). (BTW, they might be just a different views of essentially the same thing as happened with Matrix Mechanics and Wave Mechanics)

But I personally prefer AE much for the following reasons.
They are not very much the 'disadvantages' of RBM in Computer Science sense, but I think are important in engineering context.

* You usually use Deep Belief Net (that use RBM for layer-wise training) as the pre-training method for a deep neural network, that is empirically successful but lose implication from the original DBN. But AE (and Stacked AE) can be seen as just a layer-wise training method of a DNN, which I think has no disagreement.

* Some variants of AEs, such as Denoising Autoencoder, Contractive Autoencoder, and the combination of the them are reportedly at least as powerful as RBM. So the use of AE do not comes with penalty.

* AE is very compatible with other Artificial Neural Network studies such as Convolutional Neural Net, and you can easily extend the idea of AE for those studies like J.Masci did with Stacked Convolutional Autoencoder (Page on www.idsia.ch).

* AE has ordinary Loss Function whereas RBM has Energy Potential in that place. I think average software engineers (like me) are more familiar with the former and knows how to modify Loss Function to be desirable for a specific problem.

* AE can easily treat all the real number inputs, probabilistic distribution, and binary inputs, by simply changing the activation function and loss function. But RBM only have clear implication for binary inputs.

* AE can be trained by Stochastic Gradient Descent (SGD) and Peudo-Newton algorithms, both of which have lots of studies in the context of convex optimization, whereas RBM uses Contrastive Divergence which I think is more specific and has less reports.

* Stacked AE can be fine-tuned by itself using ordinary back-propagation method to minimize total reconstruction loss, whereas fine tuning of stacked RBM (Deep Boltman Machine) seems to be more difficult.

There must be a lost more both in engineering and computer science sense, and computer scientists must have lots to argue. They are just my ideas.

The distinction between «deep learning» and «shallow learning» is a bit simplistic view. In fact, the term «deep learning» is somewhat unfortunate, it's more a buzzword or a marketing word. Since deep learning involves multiple levels of representation and multiple layers of non-linear processing units (or neurons), it has seemed appropriate to describe them as deep. Then media (and social-media) have decided to run with the «deep learning» term and we are now stucked with it. A name that inspires many to dream...

By contrast, all non deep learning approaches can be qualified of shallow learning. Moreover, it happens that the very first neural networks, which had not the learning capacity of today's deep neural networks, had only a couple of hidden layers, more often just one. So, these were shallow networks, thereby strengthening the use of the «shallow learning» term. But as I wrote, it’s a simplistic view.

A more accurate name should be «representation learning» or «hierarchical learning». Remind you, the ICLR Conference (International Conference on Learning Representations) which is all about deep learning stuff and was named just at the beginning of the «deep learning» wave. There is also a notion of distributed representation. But for now, I'll pass...

Furthermore, deep learning architecture can learn representations and features directly from the input with little to no prior knowledge. It’s why some people have proposed the term «features learning». That's the promise of «no more features engineering», which contrasts with other machine learning approaches. In fact, that's not totally true, there is always a tradeoff between prior knowledge, amount of training data, the capacity of generalization of the learned models, and the difficulty to train the models. For example, there is many research works in deep learning to build very general priors.