Bilinear Classes: A Structural Framework for Provable Generalization in RL

Simon Du, Sham Kakade, Jason Lee, Shachar Lovett, Gaurav Mahajan, Wen Sun, Ruosong Wang
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:2826-2836, 2021.

Abstract

This work introduces Bilinear Classes, a new structural framework, which permit generalization in reinforcement learning in a wide variety of settings through the use of function approximation. The framework incorporates nearly all existing models in which a polynomial sample complexity is achievable, and, notably, also includes new models, such as the Linear Q*/V* model in which both the optimal Q-function and the optimal V-function are linear in some known feature space. Our main result provides an RL algorithm which has polynomial sample complexity for Bilinear Classes; notably, this sample complexity is stated in terms of a reduction to the generalization error of an underlying supervised learning sub-problem. These bounds nearly match the best known sample complexity bounds for existing models. Furthermore, this framework also extends to the infinite dimensional (RKHS) setting: for the the Linear Q*/V* model, linear MDPs, and linear mixture MDPs, we provide sample complexities that have no explicit dependence on the explicit feature dimension (which could be infinite), but instead depends only on information theoretic quantities.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-du21a, title = {Bilinear Classes: A Structural Framework for Provable Generalization in RL}, author = {Du, Simon and Kakade, Sham and Lee, Jason and Lovett, Shachar and Mahajan, Gaurav and Sun, Wen and Wang, Ruosong}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {2826--2836}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/du21a/du21a.pdf}, url = {https://proceedings.mlr.press/v139/du21a.html}, abstract = {This work introduces Bilinear Classes, a new structural framework, which permit generalization in reinforcement learning in a wide variety of settings through the use of function approximation. The framework incorporates nearly all existing models in which a polynomial sample complexity is achievable, and, notably, also includes new models, such as the Linear Q*/V* model in which both the optimal Q-function and the optimal V-function are linear in some known feature space. Our main result provides an RL algorithm which has polynomial sample complexity for Bilinear Classes; notably, this sample complexity is stated in terms of a reduction to the generalization error of an underlying supervised learning sub-problem. These bounds nearly match the best known sample complexity bounds for existing models. Furthermore, this framework also extends to the infinite dimensional (RKHS) setting: for the the Linear Q*/V* model, linear MDPs, and linear mixture MDPs, we provide sample complexities that have no explicit dependence on the explicit feature dimension (which could be infinite), but instead depends only on information theoretic quantities.} }
Endnote
%0 Conference Paper %T Bilinear Classes: A Structural Framework for Provable Generalization in RL %A Simon Du %A Sham Kakade %A Jason Lee %A Shachar Lovett %A Gaurav Mahajan %A Wen Sun %A Ruosong Wang %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-du21a %I PMLR %P 2826--2836 %U https://proceedings.mlr.press/v139/du21a.html %V 139 %X This work introduces Bilinear Classes, a new structural framework, which permit generalization in reinforcement learning in a wide variety of settings through the use of function approximation. The framework incorporates nearly all existing models in which a polynomial sample complexity is achievable, and, notably, also includes new models, such as the Linear Q*/V* model in which both the optimal Q-function and the optimal V-function are linear in some known feature space. Our main result provides an RL algorithm which has polynomial sample complexity for Bilinear Classes; notably, this sample complexity is stated in terms of a reduction to the generalization error of an underlying supervised learning sub-problem. These bounds nearly match the best known sample complexity bounds for existing models. Furthermore, this framework also extends to the infinite dimensional (RKHS) setting: for the the Linear Q*/V* model, linear MDPs, and linear mixture MDPs, we provide sample complexities that have no explicit dependence on the explicit feature dimension (which could be infinite), but instead depends only on information theoretic quantities.
APA
Du, S., Kakade, S., Lee, J., Lovett, S., Mahajan, G., Sun, W. & Wang, R.. (2021). Bilinear Classes: A Structural Framework for Provable Generalization in RL. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:2826-2836 Available from https://proceedings.mlr.press/v139/du21a.html.

Related Material