Notes of Advanced Machine Learning(I)

1. Two different ways represent a distribution over several random variables: (1) product of conditional probabilities: \(p(x_1,x_2,x_3,x_4)=p(x_4)p(x_3|x_4)p(x_2|x_3,x_4)p(x_1|x_2,x_3,x_4)\) and (2) global energy function: \(p(x_1,x_2,x_3,x_4)=\frac{1}{Z}e^{-E(x_1,x_2,x_3,x_4)}\), where \(Z\) is the partition function.
2. Directed graphical models use conditional probabilities, which undirected graphical models use energy functions that are a sum of several terms. Deep belief net(DBN) is a hybrid model.

3. 1. Probabilistic Model

   Two different ways represent a distribution over several random variables:

   • product of conditional probabilities: p(x1,x2,x3,x4)=p(x4)p(x3|x4)p(x2|x3,x4)p(x1|x2,x3,x4)

   • global energy function:

   p(x1,x2,x3,x4)=1Ze{-E(x1,x2,x3,x4)},

   where Zis the partition function.

   Directed graphical models use conditional probabilities(Bayesian networks), while undirected graphical models(Markov random fields, Boltzmann machines) use energy functions that are a sum of several terms. Deep belief net(DBN) is a hybrid model.

   Directed Graphs

   Directed graphs are useful for expressing causal relationships between random variables.

   • The joint distribution defined by the graph is given by the product of a conditional distribution for each node conditioned on its parents.

   • For example, the joint distribution over x1,,x7 factorizes:

   p(x)=p(x1)p(x2)p(x3)p(x4|x1,x2,x3)p(x5|x1,x3)p(x6|x4)p(x7|x4,x5)

   Markov Random Fields

   p(x)=1Zcc(xc)

   • Each potential function is a mapping from joint configurations of random variables in a clique to non-negative real numbers.

   • The choice of potential functions is not restricted to having specific probabilistic interpretations.

   • Potential functions are often represented as exponentials:

   p(x)=1Zcc(xc)=1Z(-cE(xc))=1Z(-E(x)) (Boltzmann distribution)

   • Computing Z is very hard, which represents a major limitation of undirected models.

   ---

    

   2. Singular Value Decomposition

   Singular Value Decomposition(SVD) is a factorization of a real or complex matrix. Formally, the singular value decomposition of an mn matrix M is a factorization of the form

   M=UV*

   where U is a mm unitary matrix,  is an mn rectangular diagonal matrix with nonnegative real numbers on the diagonal, and V*(the conjugate transpose of V: (V*)ij=Vji, for real matrix, it equals the transpose) is an nn unitary matrix.

   A complex square matrix U is unitary if U*U=UU*=I.

   The diagonal entries ij of  are known as the singular values of M, which means they are the square roots of the eigenvalues of matrix MM*. The m columns of U and n columns of V are called the left-singular vectors and right-singular vectors of M, respectively.

   The SVD and the eigendecomposition are closely related:

   • The left-singular vectors of M(columns of U) are eigenvectors of MM*.

   • The right-singular vectors of M(columns of V) are eigenvectors of M*M.

   • The non-zero singular values of M(diagonal entries of ) are the square roots of the non-zero eigenvalues of both M*M and MM*.

References:

1. U Toronto CSC2535: http://www.cs.toronto.edu/~hinton/csc2535/lectures.html