| Title | Automatic, dynamic, and nearly optimal learning rate specification via local quadratic approximation |
| Authors | Zhu, Yingqiu Huang, Danyang Gao, Yuan Wu, Rui Chen, Yu Zhang, Bo Wang, Hansheng |
| Affiliation | Renmin Univ China, Sch Stat, Beijing, Peoples R China Renmin Univ China, Ctr Appl Stat, Beijing, Peoples R China East China Normal Univ, Sch Stat, Shanghai, Peoples R China Xian Eurasia Univ, Sch Finance, Xian, Peoples R China Peking Univ, Guanghua Sch Management, Beijing, Peoples R China |
| Issue Date | Sep-2021 |
| Publisher | NEURAL NETWORKS |
| Abstract | In deep learning tasks, the update step size determined by the learning rate at each iteration plays a critical role in gradient-based optimization. However, determining the appropriate learning rate in practice typically relies on subjective judgment. In this work, we propose a novel optimization method based on local quadratic approximation (LQA). In each update step, we locally approximate the loss function along the gradient direction by using a standard quadratic function of the learning rate. Subsequently, we propose an approximation step to obtain a nearly optimal learning rate in a computationally efficient manner. The proposed LQA method has three important features. First, the learning rate is automatically determined in each update step. Second, it is dynamically adjusted according to the current loss function value and parameter estimates. Third, with the gradient direction fixed, the proposed method attains a nearly maximum reduction in the loss function. Extensive experiments were conducted to prove the effectiveness of the proposed LQA method. (C) 2021 Elsevier Ltd. All rights reserved. |
| URI | http://hdl.handle.net/20.500.11897/619477 |
| ISSN | 0893-6080 |
| DOI | 10.1016/j.neunet.2021.03.025 |
| Indexed | SCI(E) |
| Appears in Collections: | 光华管理学院 |
