Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://link.springer.com/10.1007/s43069-023-00215-6 below:

Derivation of Coordinate Descent Algorithms from Optimal Control Theory

Recently, it was posited that disparate optimization algorithms may be coalesced in terms of a central source emanating from optimal control theory. Here we further this proposition by showing how coordinate descent algorithms may be derived from this emerging new principle. In particular, we show that basic coordinate descent algorithms can be derived using a maximum principle and a collection of max functions as “control” Lyapunov functions. The convergence of the resulting coordinate descent algorithms is thus connected to the controlled dissipation of their corresponding Lyapunov functions. The operational metric for the search vector in all cases is given by the Hessian of the convex objective function.

This is a preview of subscription content, log in via an institution to check access.

Springer+ Basic

€34.99 /Month

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

My manuscript has no associated data.

1. Ross IM (2019) An optimal control theory for nonlinear optimization. J Comp Appl Math 354:39–51

   Article  Google Scholar 

2. Vinter RB (2000) Optimal Control. Birkhäuser, Boston, MA

   Google Scholar 

3. Bazaraa MS, Sherali HD, Shetty CM (2006) Nonlinear Programming: Theory and Algorithms, 3rd edn. Wiley-Interscience, Hoboken, NJ, pp 367–369

   Book  Google Scholar 

4. Sontag ED, Sussmann HJ (1996) General classes of control-Lyapunov functions. ISNM International Series of Numerical Mathematics, vol 121. Birkhäuser Basel, pp 87–97

5. Polyak B, Shcherbakov P (2017) Lyapunov functions: an optimization theory perspective. IFAC PapersOnLine 50–1:7456–7461

   Article  Google Scholar 

6. Wright SJ (2015) Coordinate descent algorithms. Math Program 151(1):3–34

   Article  Google Scholar 

7. Bottou L, Curtis FE, Nocedal J (2018) Optimization methods for large-scale machine learning. SIAM Rev 60(2):223–311

   Article  Google Scholar 

8. Nesterov Yu (2012) Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM J. Opt. 22(2):341–362

   Article  Google Scholar 

9. Bengio Y, LeCun Y, Hinton G (2015) Deep Learning. Nature 521:436–444

   Article  Google Scholar 

10. Thibault J-B, Sauer KD, Bouman CA, Hsieh J (2007) A three-dimensional statistical approach to improved image quality for multislice helical CT. Med Phys 34(11):4526–4544

    Article  Google Scholar 

11. Wang H et al (2022) A comprehensive survey on training acceleration for large machine learning models in IoT. IEEE Internet Things J 9(2):939–963

    Article  Google Scholar 

12. Ifrim G, Carsten W (2011) Bounded coordinate-descent for biological sequence classification in high dimensional predictor space. Proceedings in 17th ACM SIGKDD Intl. Conf. on knowledge discovery and data mining, KDD, pp 708–716

13. Ross IM (2015) A Primer on Pontryagin’s Principle in Optimal Control, 2nd edn. Collegiate Publishers, San Francisco, CA

    Google Scholar 

14. Goh BS (1997) Algorithms for unconstrained optimization via control theory. J Optim Theory Appl 92(3):581–604

    Article  Google Scholar 

15. Lee MS, Harno HG, Goh BS, Lim KH (2021) On the bang-bang control approach via a component-wise line search strategy for unconstrained optimization. Numerical Algebra, Control and Optimization 11(1):45–61

    Article  Google Scholar 

16. Clarke F (2004) Lyapunov functions and feedback in nonlinear control. In: de Queiroz MS, Malisoff M, Wolenski P (eds) Optimal control, stabilization and nonsmooth analysis. Lecture Notes in Control and Information Science, vol 301. Springer, Berlin, Heidelberg, pp 267–282

17. Bhaya A, Kaszkurewicz E (2006) Control Perspectives on Numerical Algorithms and Matrix Problems. SIAM, Philadelphia, PA

    Book  Google Scholar 

18. Nocedal J, Wright S (2006) Numerical Optimization, 2nd edn. Springer, New York, NY

    Google Scholar 

19. Hiriart-Urruty J-B, Lemaréchal C (1996) Convex Analysis and Minimization Algorithms I. Springer-Verlag, Berlin Heidelberg GmbH

    Google Scholar 

20. Borwein J, Lewis A (2006) Convex Analysis and Nonlinear Optimization: Theory and Examples, 2nd edn. Springer, New York, N.Y

    Book  Google Scholar 

21. Boyd S, Vandenberghe L (2009) Convex Optimization. Cambridge University Press, Cambridge, UK

    Google Scholar 

22. Davidon WC (1991) Variable metric method for minimization, SIAM J. optimization, 1/1, pp. 1–17 (originally published as Argonne National Laboratory Research and Development Report 5990, May 1959; revised November 1959)

Download references

Funding for this research was provided by the Air Force Office of Scientific Research.

1. Control & Optimization Laboratories, Naval Postgraduate School, 1 University Circle, Monterey, CA, 93943, USA

   Isaac M. Ross

Correspondence to Isaac M. Ross.

The author declares no competing interests.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Ross, I. Derivation of Coordinate Descent Algorithms from Optimal Control Theory. Oper. Res. Forum 4, 31 (2023). https://doi.org/10.1007/s43069-023-00215-6

Download citation

• Received: 26 January 2022

• Accepted: 02 March 2023

• Published: 31 March 2023

• DOI: https://doi.org/10.1007/s43069-023-00215-6

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4