OPTIMIZATION

Consider an optimization problem in the standard form (we call this a primal problem): We denote the optimal value of this as $p^\star$. We don’t assume the problem is convex. The Lagrange dual function We define the Lagrangian $L$ associated with the problem as $$ L(x,\lambda, v) = f_0(x) + \sum^m_{i=1}\lambda_if_i(x) + \sum^p_{i=1}v_ih_i(x)$$ We call vectors $\lambda$ and $v$ the dual variables or Lagrange multiplier vectors associated with the problem (1)....

Optimization problem All optimization problems can be written as: Optimization Categories convex v.s. non-convex Deep Neural Network is non-convex continuous v.s.discrete Most are continuous variable; tree structure is discrete constrained v.s. non-constrained We add prior to make it a constrained problem smooth v.s.non-smooth Most are smooth optimization Different initialization brings different optimum (if not convex) Idea: Give up global optimal and find a good local optimal....