Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://www.deeplearningbook.org/contents/autoencoders.html below:

Chapter 14

Autoencoders

An

autoencoder

is a neural network that is trained to attempt to copy its input

to its output. Internally, it has a hidden layer

h

that describes a

code

used to

represent the input. The network may be viewed as consisting of two parts: an

encoder function

h

=

f

(

x

) and a decoder that produces a reconstruction

r

=

g

(

h

).

This architecture is presented in figure 14.1. If an autoencoder succeeds in simply

learning to set

g

(

f

(

x

)) =

x

everywhere, then it is not especially useful. Instead,

autoencoders are designed to be unable to learn to copy perfectly. Usually they are

restricted in ways that allow them to copy only approximately, and to copy only

input that resembles the training data. Because the model is forced to prioritize

which aspects of the input should be copied, it often learns useful properties of the

data.

Modern autoencoders have generalized the idea of an encoder and a de-

coder beyond deterministic functions to stochastic mappings

p

encoder

(

h | x

) and

p

decoder

(x | h).

The idea of autoencoders has been part of the historical landscape of neural

networks for decades (LeCun, 1987; Bourlard and Kamp, 1988; Hinton and Zemel,

1994). Traditionally, autoencoders were used for dimensionality reduction or

feature learning. Recently, theoretical connections between autoencoders and

latent variable models have brought autoencoders to the forefront of generative

modeling, as we will see in chapter 20. Autoencoders may be thought of as being

a special case of feedforward networks and may be trained with all the same

techniques, typically minibatch gradient descent following gradients computed

by back-propagation. Unlike general feedforward networks, autoencoders may

also be trained using

recirculation

(Hinton and McClelland, 1988), a learning

algorithm based on comparing the activations of the network on the original input

499

CHAPTER 14. AUTOENCODERS

Figure 14.1: The general structure of an autoencoder, mapping an input x to an output

(called reconstruction)

r

through an internal representation or code

h

. The autoencoder

has two components: the encoder

f

(mapping

x

to

h

) and the decoder

g

(mapping

h

to r).

to the activations on the reconstructed input. Recirculation is regarded as more

biologically plausible than back-propagation but is rarely used for machine learning

applications.

14.1 Undercomplete Autoencoders

Copying the input to the output may sound useless, but we are typically not

interested in the output of the decoder. Instead, we hope that training the

autoencoder to perform the input copying task will result in

h

taking on useful

properties.

One way to obtain useful features from the autoencoder is to constrain

h

to

have a smaller dimension than

x

. An autoencoder whose code dimension is less

than the input dimension is called

undercomplete

. Learning an undercomplete

representation forces the autoencoder to capture the most salient features of the

training data.

The learning process is described simply as minimizing a loss function

L(x, g(f(x))), (14.1)

where

L

is a loss function penalizing

g

(

f

(

x

)) for being dissimilar from

x

, such as

the mean squared error.

When the decoder is linear and

L

is the mean squared error, an undercomplete

autoencoder learns to span the same subspace as PCA. In this case, an autoencoder

trained to perform the copying task has learned the principal subspace of the

training data as a side effect.

Autoencoders with nonlinear encoder functions

f

and nonlinear decoder func-

tions

g

can thus learn a more powerful nonlinear generalization of PCA. Unfortu-

500

CHAPTER 14. AUTOENCODERS

nately, if the encoder and decoder are allowed too much capacity, the autoencoder

can learn to perform the copying task without extracting useful information about

the distribution of the data. Theoretically, one could imagine that an autoencoder

with a one-dimensional code but a very powerful nonlinear encoder could learn to

represent each training example

x

(i)

with the code

i

. The decoder could learn to

map these integer indices back to the values of specific training examples. This

specific scenario does not occur in practice, but it illustrates clearly that an autoen-

coder trained to perform the copying task can fail to learn anything useful about

the dataset if the capacity of the autoencoder is allowed to become too great.

14.2 Regularized Autoencoders

Undercomplete autoencoders, with code dimension less than the input dimension,

can learn the most salient features of the data distribution. We have seen that

these autoencoders fail to learn anything useful if the encoder and decoder are

given too much capacity.

A similar problem occurs if the hidden code is allowed to have dimension

equal to the input, and in the

overcomplete

case in which the hidden code has

dimension greater than the input. In these cases, even a linear encoder and a linear

decoder can learn to copy the input to the output without learning anything useful

about the data distribution.

Ideally, one could train any architecture of autoencoder successfully, choosing

the code dimension and the capacity of the encoder and decoder based on the

complexity of distribution to be modeled. Regularized autoencoders provide the

ability to do so. Rather than limiting the model capacity by keeping the encoder

and decoder shallow and the code size small, regularized autoencoders use a loss

function that encourages the model to have other properties besides the ability

to copy its input to its output. These other properties include sparsity of the

representation, smallness of the derivative of the representation, and robustness

to noise or to missing inputs. A regularized autoencoder can be nonlinear and

overcomplete but still learn something useful about the data distribution, even if

the model capacity is great enough to learn a trivial identity function.

In addition to the methods described here, which are most naturally interpreted

as regularized autoencoders, nearly any generative model with latent variables

and equipped with an inference procedure (for computing latent representations

given input) may be viewed as a particular form of autoencoder. Two generative

modeling approaches that emphasize this connection with autoencoders are the

descendants of the Helmholtz machine (Hinton et al., 1995b), such as the variational

501

CHAPTER 14. AUTOENCODERS

autoencoder (section 20.10.3) and the generative stochastic networks (section 20.12).

These models naturally learn high-capacity, overcomplete encodings of the input

and do not require regularization for these encodings to be useful. Their encodings

are naturally useful because the models were trained to approximately maximize

the probability of the training data rather than to copy the input to the output.

14.2.1 Sparse Autoencoders

A sparse autoencoder is simply an autoencoder whose training criterion involves a

sparsity penalty Ω(

h

) on the code layer

h

, in addition to the reconstruction error:

L(x, g(f(x))) + Ω(h), (14.2)

where

g

(

h

) is the decoder output, and typically we have

h

=

f

(

x

), the encoder

output.

Sparse autoencoders are typically used to learn features for another task, such

as classification. An autoencoder that has been regularized to be sparse must

respond to unique statistical features of the dataset it has been trained on, rather

than simply acting as an identity function. In this way, training to perform the

copying task with a sparsity penalty can yield a model that has learned useful

features as a byproduct.

We can think of the penalty Ω(

h

) simply as a regularizer term added to

a feedforward network whose primary task is to copy the input to the output

(unsupervised learning objective) and possibly also perform some supervised task

(with a supervised learning objective) that depends on these sparse features.

Unlike other regularizers, such as weight decay, there is not a straightforward

Bayesian interpretation to this regularizer. As described in section 5.6.1, training

with weight decay and other regularization penalties can be interpreted as a

MAP approximation to Bayesian inference, with the added regularizing penalty

corresponding to a prior probability distribution over the model parameters. In

this view, regularized maximum likelihood corresponds to maximizing

p

(

θ | x

),

which is equivalent to maximizing

log p

(

x | θ

) +

log p

(

θ

). The

log p

(

x | θ

) term

is the usual data log-likelihood term, and the

log p

(

θ

) term, the log-prior over

parameters, incorporates the preference over particular values of

θ

. This view is

described in section 5.6. Regularized autoencoders defy such an interpretation

because the regularizer depends on the data and is therefore by definition not a

prior in the formal sense of the word. We can still think of these regularization

terms as implicitly expressing a preference over functions.

Rather than thinking of the sparsity penalty as a regularizer for the copying

task, we can think of the entire sparse autoencoder framework as approximating

502

CHAPTER 14. AUTOENCODERS

maximum likelihood training of a generative model that has latent variables.

Suppose we have a model with visible variables

x

and latent variables

h

, with

an explicit joint distribution

p

model

(

x, h

) =

p

model

(

h

)

p

model

(

x | h

). We refer to

p

model

(

h

) as the model’s prior distribution over the latent variables, representing

the model’s beliefs prior to seeing

x

. This is different from the way we have

previously used the word “prior,” to refer to the distribution

p

(

θ

) encoding our

beliefs about the model’s parameters before we have seen the training data. The

log-likelihood can be decomposed as

log p

model

(x) = log



h

p

model

(h, x). (14.3)

We can think of the autoencoder as approximating this sum with a point estimate

for just one highly likely value for

h

. This is similar to the sparse coding generative

model (section 13.4), but with

h

being the output of the parametric encoder rather

than the result of an optimization that infers the most likely

h

. From this point of

view, with this chosen h, we are maximizing

log p

model

(h, x) = log p

model

(h) + log p

model

(x | h). (14.4)

The log p

model

(h) term can be sparsity inducing. For example, the Laplace prior,

p

model

(h

i

) =

λ

2

e

−λ|h

i

|

, (14.5)

corresponds to an absolute value sparsity penalty. Expressing the log-prior as an

absolute value penalty, we obtain

Ω(h) = λ



i

|h

i

|, (14.6)

−log p

model

(h) =



i



λ|h

i

| − log

λ

2



= Ω(h) + const, (14.7)

where the constant term depends only on

λ

and not

h

. We typically treat

λ

as a

hyperparameter and discard the constant term since it does not affect the parameter

learning. Other priors, such as the Student

t

prior, can also induce sparsity. From

this point of view of sparsity as resulting from the effect of

p

model

(

h

) on approximate

maximum likelihood learning, the sparsity penalty is not a regularization term at

all. It is just a consequence of the model’s distribution over its latent variables.

This view provides a different motivation for training an autoencoder: it is a way

of approximately training a generative model. It also provides a different reason for

503

CHAPTER 14. AUTOENCODERS

why the features learned by the autoencoder are useful: they describe the latent

variables that explain the input.

Early work on sparse autoencoders (Ranzato et al., 2007a, 2008) explored

various forms of sparsity and proposed a connection between the sparsity penalty

and the

log Z

term that arises when applying maximum likelihood to an undirected

probabilistic model

p

(

x

) =

1

Z

˜p

(

x

). The idea is that minimizing

log Z

prevents a

probabilistic model from having high probability everywhere, and imposing sparsity

on an autoencoder prevents the autoencoder from having low reconstruction

error everywhere. In this case, the connection is on the level of an intuitive

understanding of a general mechanism rather than a mathematical correspondence.

The interpretation of the sparsity penalty as corresponding to

log p

model

(

h

) in a

directed model p

model

(h)p

model

(x | h) is more mathematically straightforward.

One way to achieve actual zeros in

h

for sparse (and denoising) autoencoders

was introduced in Glorot et al. (2011b). The idea is to use rectified linear units to

produce the code layer. With a prior that actually pushes the representations to

zero (like the absolute value penalty), one can thus indirectly control the average

number of zeros in the representation.

14.2.2 Denoising Autoencoders

Rather than adding a penalty Ω to the cost function, we can obtain an autoencoder

that learns something useful by changing the reconstruction error term of the cost

function.

Traditionally, autoencoders minimize some function

L(x, g(f(x))), (14.8)

where

L

is a loss function penalizing

g

(

f

(

x

)) for being dissimilar from

x

, such as

the

L

2

norm of their difference. This encourages

g ◦ f

to learn to be merely an

identity function if they have the capacity to do so.

A denoising autoencoder (DAE) instead minimizes

L(x, g(f(

˜

x))), (14.9)

where

˜

x

is a copy of

x

that has been corrupted by some form of noise. Denoising

autoencoders must therefore undo this corruption rather than simply copying their

input.

Denoising training forces

f

and

g

to implicitly learn the structure of

p

data

(

x

),

as shown by Alain and Bengio (2013) and Bengio et al. (2013c). Denoising

504

CHAPTER 14. AUTOENCODERS

autoencoders thus provide yet another example of how useful properties can emerge

as a byproduct of minimizing reconstruction error. They are also an example of

how overcomplete, high-capacity models may be used as autoencoders as long

as care is taken to prevent them from learning the identity function. Denoising

autoencoders are presented in more detail in section 14.5.

14.2.3 Regularizing by Penalizing Derivatives

Another strategy for regularizing an autoencoder is to use a penalty Ω, as in sparse

autoencoders,

L(x, g(f(x))) + Ω(h, x), (14.10)

but with a different form of Ω:

Ω(h, x) = λ



i

||∇

x

h

i

||

2

. (14.11)

This forces the model to learn a function that does not change much when

x

changes slightly. Because this penalty is applied only at training examples, it forces

the autoencoder to learn features that capture information about the training

distribution.

An autoencoder regularized in this way is called a

contractive autoencoder

,

or CAE. This approach has theoretical connections to denoising autoencoders,

manifold learning, and probabilistic modeling. The CAE is described in more

detail in section 14.7.

14.3 Representational Power, Layer Size and Depth

Autoencoders are often trained with only a single layer encoder and a single layer

decoder. However, this is not a requirement. In fact, using deep encoders and

decoders offers many advantages.

Recall from section 6.4.1 that there are many advantages to depth in a feedfor-

ward network. Because autoencoders are feedforward networks, these advantages

also apply to autoencoders. Moreover, the encoder is itself a feedforward network,

as is the decoder, so each of these components of the autoencoder can individually

benefit from depth.

One major advantage of nontrivial depth is that the universal approximator

theorem guarantees that a feedforward neural network with at least one hidden

layer can represent an approximation of any function (within a broad class) to an

505

CHAPTER 14. AUTOENCODERS

arbitrary degree of accuracy, provided that it has enough hidden units. This means

that an autoencoder with a single hidden layer is able to represent the identity

function along the domain of the data arbitrarily well. However, the mapping from

input to code is shallow. This means that we are not able to enforce arbitrary

constraints, such as that the code should be sparse. A deep autoencoder, with at

least one additional hidden layer inside the encoder itself, can approximate any

mapping from input to code arbitrarily well, given enough hidden units.

Depth can exponentially reduce the computational cost of representing some

functions. Depth can also exponentially decrease the amount of training data

needed to learn some functions. See section 6.4.1 for a review of the advantages of

depth in feedforward networks.

Experimentally, deep autoencoders yield much better compression than corre-

sponding shallow or linear autoencoders (Hinton and Salakhutdinov, 2006).

A common strategy for training a deep autoencoder is to greedily pretrain

the deep architecture by training a stack of shallow autoencoders, so we often

encounter shallow autoencoders, even when the ultimate goal is to train a deep

autoencoder.

14.4 Stochastic Encoders and Decoders

Autoencoders are just feedforward networks. The same loss functions and output

unit types that can be used for traditional feedforward networks are also used for

autoencoders.

As described in section 6.2.2.4, a general strategy for designing the output units

and the loss function of a feedforward network is to define an output distribution

p

(

y | x

) and minimize the negative log-likelihood

−log p

(

y | x

). In that setting,

y

is a vector of targets, such as class labels.

In an autoencoder,

x

is now the target as well as the input. Yet we can still

apply the same machinery as before. Given a hidden code

h

, we may think of

the decoder as providing a conditional distribution

p

decoder

(

x | h

). We may then

train the autoencoder by minimizing

−log p

decoder

(x | h)

. The exact form of this

loss function will change depending on the form of

p

decoder

. As with traditional

feedforward networks, we usually use linear output units to parametrize the mean of

a Gaussian distribution if

x

is real valued. In that case, the negative log-likelihood

yields a mean squared error criterion. Similarly, binary

x

values correspond to

a Bernoulli distribution whose parameters are given by a sigmoid output unit,

discrete

x

values correspond to a softmax distribution, and so on. Typically, the

506

CHAPTER 14. AUTOENCODERS

Figure 14.2: The structure of a stochastic autoencoder, in which both the encoder and the

decoder are not simple functions but instead involve some noise injection, meaning that

their output can be seen as sampled from a distribution,

p

encoder

(

h | x

) for the encoder

and p

decoder

(x | h) for the decoder.

output variables are treated as being conditionally independent given

h

so that

this probability distribution is inexpensive to evaluate, but some techniques, such

as mixture density outputs, allow tractable modeling of outputs with correlations.

To make a more radical departure from the feedforward networks we have seen

previously, we can also generalize the notion of an

encoding function f

(

x

) to

an encoding distribution p

encoder

(h | x), as illustrated in figure 14.2.

Any latent variable model p

model

(h, x) defines a stochastic encoder

p

encoder

(h | x) = p

model

(h | x) (14.12)

and a stochastic decoder

p

decoder

(x | h) = p

model

(x | h). (14.13)

In general, the encoder and decoder distributions are not necessarily conditional

distributions compatible with a unique joint distribution

p

model

(

x, h

). Alain et al.

(2015) showed that training the encoder and decoder as a denoising autoencoder

will tend to make them compatible asymptotically (with enough capacity and

examples).

14.5 Denoising Autoencoders

The

denoising autoencoder

(DAE) is an autoencoder that receives a corrupted

data point as input and is trained to predict the original, uncorrupted data point

as its output.

The DAE training procedure is illustrated in figure 14.3. We introduce a

corruption process

C

(

˜

x | x

), which represents a conditional distribution over

507

CHAPTER 14. AUTOENCODERS

Figure 14.3: The computational graph of the cost function for a denoising autoencoder,

which is trained to reconstruct the clean data point

x

from its corrupted version

˜

x

.

This is accomplished by minimizing the loss

L

=

−log p

decoder

(

x | h

=

f

(

˜

x

)), where

˜

x

is a corrupted version of the data example

x

, obtained through a given corruption

process

C

(

˜

x | x

). Typically the distribution

p

decoder

is a factorial distribution whose mean

parameters are emitted by a feedforward network g.

corrupted samples

˜

x

, given a data sample

x

. The autoencoder then learns a

reconstruction distribution p

reconstruct

(

x |

˜

x

) estimated from training pairs

(x,

˜

x) as follows:

1. Sample a training example x from the training data.

2. Sample a corrupted version

˜

x from C(

˜

x | x = x).

3.

Use (

x,

˜

x

) as a training example for estimating the autoencoder reconstruction

distribution

p

reconstruct

(

x |

˜

x

) =

p

decoder

(

x | h

) with

h

the output of encoder

f(

˜

x) and p

decoder

typically defined by a decoder g(h).

Typically we can simply perform gradient-based approximate minimization (such

as minibatch gradient descent) on the negative log-likelihood

−log p

decoder

(

x | h

).

As long as the encoder is deterministic, the denoising autoencoder is a feedforward

network and may be trained with exactly the same techniques as any other

feedforward network.

We can therefore view the DAE as performing stochastic gradient descent on

the following expectation:

− E

x∼ˆp

data

(x)

E

˜

x∼C(

˜

x|x)

log p

decoder

(x | h = f(

˜

x)), (14.14)

where ˆp

data

(x) is the training distribution.

508

CHAPTER 14. AUTOENCODERS

14.5.1 Estimating the Score

Score matching (Hyvärinen, 2005) is an alternative to maximum likelihood. It

provides a consistent estimator of probability distributions based on encouraging

the model to have the same

score

as the data distribution at every training point

x. In this context, the score is a particular gradient field:

∇

x

log p(x). (14.15)

Score matching is discussed further in section 18.4. For the present discussion,

regarding autoencoders, it is sufficient to understand that learning the gradient

field of log p

data

is one way to learn the structure of p

data

itself.

A very important property of DAEs is that their training criterion (with

conditionally Gaussian

p

(

x | h

)) makes the autoencoder learn a vector field

(

g

(

f

(

x

))

− x

) that estimates the score of the data distribution. This is illustrated

in figure 14.4.

Denoising training of a specific kind of autoencoder (sigmoidal hidden units,

linear reconstruction units) using Gaussian noise and mean squared error as the

reconstruction cost is equivalent (Vincent, 2011) to training a specific kind of

undirected probabilistic model called an RBM with Gaussian visible units. This

kind of model is described in detail in section 20.5.1; for the present discussion, it

suffices to know that it is a model that provides an explicit

p

model

(

x

;

θ

). When the

RBM is trained using

denoising score matching

(Kingma and LeCun, 2010),

its learning algorithm is equivalent to denoising training in the corresponding

autoencoder. With a fixed noise level, regularized score matching is not a consistent

estimator; it instead recovers a blurred version of the distribution. If the noise

level is chosen to approach 0 when the number of examples approaches infinity,

however, then consistency is recovered. Denoising score matching is discussed in

more detail in section 18.5.

Other connections between autoencoders and RBMs exist. Score matching

applied to RBMs yields a cost function that is identical to reconstruction error

combined with a regularization term similar to the contractive penalty of the

CAE (Swersky et al., 2011). Bengio and Delalleau (2009) showed that an autoen-

coder gradient provides an approximation to contrastive divergence training of

RBMs.

For continuous-valued

x

, the denoising criterion with Gaussian corruption and

reconstruction distribution yields an estimator of the score that is applicable to

general encoder and decoder parametrizations (Alain and Bengio, 2013). This

means a generic encoder-decoder architecture may be made to estimate the score

509

CHAPTER 14. AUTOENCODERS

Figure 14.4: A denoising autoencoder is trained to map a corrupted data point

˜

x

back to

the original data point

x

. We illustrate training examples

x

as red crosses lying near a

low-dimensional manifold, illustrated with the bold black line. We illustrate the corruption

process

C

(

˜

x | x

) with a gray circle of equiprobable corruptions. A gray arrow demonstrates

how one training example is transformed into one sample from this corruption process.

When the denoising autoencoder is trained to minimize the average of squared errors

||g

(

f

(

˜

x

))

−x||

2

, the reconstruction

g

(

f

(

˜

x

)) estimates

E

x,

˜

x∼p

data

(x)C(

˜

x|x)

[

x |

˜

x

]. The vector

g

(

f

(

˜

x

))

−

˜

x

points approximately toward the nearest point on the manifold, since

g

(

f

(

˜

x

))

estimates the center of mass of the clean points

x

that could have given rise to

˜

x

. The

autoencoder thus learns a vector field

g

(

f

(

x

))

− x

indicated by the green arrows. This

vector field estimates the score

∇

x

log p

data

(

x

) up to a multiplicative factor that is the

average root mean square reconstruction error.

510

CHAPTER 14. AUTOENCODERS

by training with the squared error criterion

||g(f(

˜

x)) − x||

2

(14.16)

and corruption

C(

˜

x =

˜

x|x) = N(

˜

x; µ = x, Σ = σ

2

I) (14.17)

with noise variance σ

2

. See figure 14.5 for an illustration of how this works.

In general, there is no guarantee that the reconstruction

g

(

f

(

x

)) minus the

input

x

corresponds to the gradient of any function, let alone to the score. That is

why the early results (Vincent, 2011) are specialized to particular parametrizations,

where

g

(

f

(

x

))

− x

may be obtained by taking the derivative of another function.

Kamyshanska and Memisevic (2015) generalized the results of Vincent (2011) by

identifying a family of shallow autoencoders such that

g

(

f

(

x

))

− x

corresponds to

a score for all members of the family.

So far we have described only how the denoising autoencoder learns to represent

a probability distribution. More generally, one may want to use the autoencoder

as a generative model and draw samples from this distribution. This is described

in section 20.11.

14.5.1.1 Historical Perspective

The idea of using MLPs for denoising dates back to the work of LeCun (1987)

and Gallinari et al. (1987). Behnke (2001) also used recurrent networks to denoise

images. Denoising autoencoders are, in some sense, just MLPs trained to denoise.

The name “denoising autoencoder,” however, refers to a model that is intended not

merely to learn to denoise its input but to learn a good internal representation

as a side effect of learning to denoise. This idea came much later (Vincent

et al., 2008, 2010). The learned representation may then be used to pretrain a

deeper unsupervised network or a supervised network. Like sparse autoencoders,

sparse coding, contractive autoencoders, and other regularized autoencoders, the

motivation for DAEs was to allow the learning of a very high-capacity encoder

while preventing the encoder and decoder from learning a useless identity function.

Prior to the introduction of the modern DAE, Inayoshi and Kurita (2005)

explored some of the same goals with some of the same methods. Their approach

minimizes reconstruction error in addition to a supervised objective while injecting

noise in the hidden layer of a supervised MLP, with the objective to improve

generalization by introducing the reconstruction error and the injected noise. Their

method was based on a linear encoder, however, and could not learn function

families as powerful as the modern DAE can.

511

CHAPTER 14. AUTOENCODERS

Figure 14.5: Vector field learned by a denoising autoencoder around a 1-D curved

manifold near which the data concentrate in a 2-D space. Each arrow is proportional

to the reconstruction minus input vector of the autoencoder and points towards higher

probability according to the implicitly estimated probability distribution. The vector field

has zeros at both maxima of the estimated density function (on the data manifolds) and

at minima of that density function. For example, the spiral arm forms a 1-D manifold of

local maxima that are connected to each other. Local minima appear near the middle of

the gap between two arms. When the norm of reconstruction error (shown by the length of

the arrows) is large, probability can be significantly increased by moving in the direction

of the arrow, and that is mostly the case in places of low probability. The autoencoder

maps these low probability points to higher probability reconstructions. Where probability

is maximal, the arrows shrink because the reconstruction becomes more accurate. Figure

reproduced with permission from Alain and Bengio (2013).

512

CHAPTER 14. AUTOENCODERS

14.6 Learning Manifolds with Autoencoders

Like many other machine learning algorithms, autoencoders exploit the idea

that data concentrates around a low-dimensional manifold or a small set of such

manifolds, as described in section 5.11.3. Some machine learning algorithms exploit

this idea only insofar as they learn a function that behaves correctly on the manifold

but that may have unusual behavior if given an input that is off the manifold.

Autoencoders take this idea further and aim to learn the structure of the manifold.

To understand how autoencoders do this, we must present some important

characteristics of manifolds.

An important characterization of a manifold is the set of its

tangent planes

.

At a point

x

on a

d

-dimensional manifold, the tangent plane is given by

d

basis

vectors that span the local directions of variation allowed on the manifold. As

illustrated in figure 14.6, these local directions specify how one can change

x

infinitesimally while staying on the manifold.

All autoencoder training procedures involve a compromise between two forces:

1.

Learning a representation

h

of a training example

x

such that

x

can be

approximately recovered from

h

through a decoder. The fact that

x

is

drawn from the training data is crucial, because it means the autoencoder

need not successfully reconstruct inputs that are not probable under the

data-generating distribution.

2. Satisfying the constraint or regularization penalty. This can be an architec-

tural constraint that limits the capacity of the autoencoder, or it can be

a regularization term added to the reconstruction cost. These techniques

generally prefer solutions that are less sensitive to the input.

Clearly, neither force alone would be useful—copying the input to the output

is not useful on its own, nor is ignoring the input. Instead, the two forces together

are useful because they force the hidden representation to capture information

about the structure of the data-generating distribution. The important principle

is that the autoencoder can afford to represent only the variations that are needed

to reconstruct training examples. If the data-generating distribution concentrates

near a low-dimensional manifold, this yields representations that implicitly capture

a local coordinate system for this manifold: only the variations tangent to the

manifold around

x

need to correspond to changes in

h

=

f

(

x

). Hence the encoder

learns a mapping from the input space

x

to a representation space, a mapping that

is only sensitive to changes along the manifold directions, but that is insensitive to

changes orthogonal to the manifold.

513

CHAPTER 14. AUTOENCODERS

Figure 14.6: An illustration of the concept of a tangent hyperplane. Here we create a 1-D

manifold in 784-D space. We take an MNIST image with 784 pixels and transform it by

translating it vertically. The amount of vertical translation defines a coordinate along

a 1-D manifold that traces out a curved path through image space. This plot shows a

few points along this manifold. For visualization, we have projected the manifold into

2-D space using PCA. An

n

-dimensional manifold has an

n

-dimensional tangent plane

at every point. This tangent plane touches the manifold exactly at that point and is

oriented parallel to the surface at that point. It defines the space of directions in which

it is possible to move while remaining on the manifold. This 1-D manifold has a single

tangent line. We indicate an example tangent line at one point, with an image showing

how this tangent direction appears in image space. Gray pixels indicate pixels that do not

change as we move along the tangent line, white pixels indicate pixels that brighten, and

black pixels indicate pixels that darken.

514

CHAPTER 14. AUTOENCODERS

A one-dimensional example is illustrated in figure 14.7, showing that, by making

the reconstruction function insensitive to perturbations of the input around the

data points, we cause the autoencoder to recover the manifold structure.

To understand why autoencoders are useful for manifold learning, it is in-

structive to compare them to other approaches. What is most commonly learned

to characterize a manifold is a

representation

of the data points on (or near)

the manifold. Such a representation for a particular example is also called its

embedding. It is typically given by a low-dimensional vector, with fewer dimensions

than the “ambient” space of which the manifold is a low-dimensional subset. Some

algorithms (nonparametric manifold learning algorithms, discussed below) directly

learn an embedding for each training example, while others learn a more general

mapping, sometimes called an encoder, or representation function, that maps any

point in the ambient space (the input space) to its embedding.

x

0

x

1

x

2

x

0.0

0.2

0.4

0.6

0.8

1.0

r(x)

Identity

Optimal reconstruction

Figure 14.7: If the autoencoder learns a reconstruction function that is invariant to small

perturbations near the data points, it captures the manifold structure of the data. Here

the manifold structure is a collection of 0-dimensional manifolds. The dashed diagonal

line indicates the identity function target for reconstruction. The optimal reconstruction

function crosses the identity function wherever there is a data point. The horizontal

arrows at the bottom of the plot indicate the

r

(

x

)

− x

reconstruction direction vector at

the base of the arrow, in input space, always pointing toward the nearest “manifold” (a

single data point in the 1-D case). The denoising autoencoder explicitly tries to make

the derivative of the reconstruction function

r

(

x

) small around the data points. The

contractive autoencoder does the same for the encoder. Although the derivative of

r

(

x

) is

asked to be small around the data points, it can be large between the data points. The

space between the data points corresponds to the region between the manifolds, where

the reconstruction function must have a large derivative to map corrupted points back

onto the manifold.

515

CHAPTER 14. AUTOENCODERS

Figure 14.8: Nonparametric manifold learning procedures build a nearest neighbor graph

in which nodes represent training examples and directed edges indicate nearest neighbor

relationships. Various procedures can thus obtain the tangent plane associated with a

neighborhood of the graph as well as a coordinate system that associates each training

example with a real-valued vector position, or

embedding

. It is possible to generalize

such a representation to new examples by a form of interpolation. As long as the number

of examples is large enough to cover the curvature and twists of the manifold, these

approaches work well. Images from the QMUL Multiview Face Dataset (Gong et al.,

2000).

Manifold learning has mostly focused on unsupervised learning procedures that

attempt to capture these manifolds. Most of the initial machine learning research

on learning nonlinear manifolds has focused on

nonparametric

methods based

on the

nearest neighbor graph

. This graph has one node per training example

and edges connecting near neighbors to each other. These methods (Schölkopf

et al., 1998; Roweis and Saul, 2000; Tenenbaum et al., 2000; Brand, 2003; Belkin

and Niyogi, 2003; Donoho and Grimes, 2003; Weinberger and Saul, 2004; Hinton

and Roweis, 2003; van der Maaten and Hinton, 2008) associate each node with a

tangent plane that spans the directions of variations associated with the difference

vectors between the example and its neighbors, as illustrated in figure 14.8.

A global coordinate system can then be obtained through an optimization or

by solving a linear system. Figure 14.9 illustrates how a manifold can be tiled by

516

CHAPTER 14. AUTOENCODERS

Figure 14.9: If the tangent planes (see figure 14.6) at each location are known, then they

can be tiled to form a global coordinate system or a density function. Each local patch

can be thought of as a local Euclidean coordinate system or as a locally flat Gaussian, or

“pancake,” with a very small variance in the directions orthogonal to the pancake and a

very large variance in the directions defining the coordinate system on the pancake. A

mixture of these Gaussians provides an estimated density function, as in the manifold

Parzen window algorithm (Vincent and Bengio, 2003) or in its nonlocal neural-net-based

variant (Bengio et al., 2006c).

517

CHAPTER 14. AUTOENCODERS

a large number of locally linear Gaussian-like patches (or “pancakes,” because the

Gaussians are flat in the tangent directions).

A fundamental difficulty with such local nonparametric approaches to manifold

learning is raised in Bengio and Monperrus (2005): if the manifolds are not very

smooth (they have many peaks and troughs and twists), one may need a very

large number of training examples to cover each one of these variations, with

no chance to generalize to unseen variations. Indeed, these methods can only