Redundancy Resolution as Action Bias in Policy Search for Robotic Manipulation

Firas Al-Hafez, Jochen J. Steil
Proceedings of the 5th Conference on Robot Learning, PMLR 164:981-990, 2022.

Abstract

We propose a novel approach that biases actions during policy search by lifting the concept of redundancy resolution from multi-DoF robot kinematics to the level of the reward in deep reinforcement learning and evolution strategies. The key idea is to bias the distribution of executed actions in the sense that the immediate reward remains unchanged. The resulting biased actions favor secondary objectives yielding policies that are safer to apply on the real robot. We demonstrate the feasibility of our method, considered as policy search with redundant action bias (PSRAB), in a reaching and a pick-and-lift task with a 7-DoF Franka robot arm trained in RLBench - a recently introduced benchmark for robotic manipulation - using state-of-the-art TD3 deep reinforcement learning and OpenAI’s evolutionary strategy. We show that it is a flexible approach without the need of significant fine-tuning and interference with the main objective even across different policy search methods and tasks of different complexity. We evaluate our approach in simulation and on the real robot. Our project website with videos and further results can be found at: https://sites.google.com/view/redundant-action-bias

Cite this Paper

BibTeX
@InProceedings{pmlr-v164-al-hafez22a, title = {Redundancy Resolution as Action Bias in Policy Search for Robotic Manipulation}, author = {Al-Hafez, Firas and Steil, Jochen J.}, booktitle = {Proceedings of the 5th Conference on Robot Learning}, pages = {981--990}, year = {2022}, editor = {Faust, Aleksandra and Hsu, David and Neumann, Gerhard}, volume = {164}, series = {Proceedings of Machine Learning Research}, month = {08--11 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v164/al-hafez22a/al-hafez22a.pdf}, url = {https://proceedings.mlr.press/v164/al-hafez22a.html}, abstract = {We propose a novel approach that biases actions during policy search by lifting the concept of redundancy resolution from multi-DoF robot kinematics to the level of the reward in deep reinforcement learning and evolution strategies. The key idea is to bias the distribution of executed actions in the sense that the immediate reward remains unchanged. The resulting biased actions favor secondary objectives yielding policies that are safer to apply on the real robot. We demonstrate the feasibility of our method, considered as policy search with redundant action bias (PSRAB), in a reaching and a pick-and-lift task with a 7-DoF Franka robot arm trained in RLBench - a recently introduced benchmark for robotic manipulation - using state-of-the-art TD3 deep reinforcement learning and OpenAI’s evolutionary strategy. We show that it is a flexible approach without the need of significant fine-tuning and interference with the main objective even across different policy search methods and tasks of different complexity. We evaluate our approach in simulation and on the real robot. Our project website with videos and further results can be found at: https://sites.google.com/view/redundant-action-bias} }
Endnote
%0 Conference Paper %T Redundancy Resolution as Action Bias in Policy Search for Robotic Manipulation %A Firas Al-Hafez %A Jochen J. Steil %B Proceedings of the 5th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2022 %E Aleksandra Faust %E David Hsu %E Gerhard Neumann %F pmlr-v164-al-hafez22a %I PMLR %P 981--990 %U https://proceedings.mlr.press/v164/al-hafez22a.html %V 164 %X We propose a novel approach that biases actions during policy search by lifting the concept of redundancy resolution from multi-DoF robot kinematics to the level of the reward in deep reinforcement learning and evolution strategies. The key idea is to bias the distribution of executed actions in the sense that the immediate reward remains unchanged. The resulting biased actions favor secondary objectives yielding policies that are safer to apply on the real robot. We demonstrate the feasibility of our method, considered as policy search with redundant action bias (PSRAB), in a reaching and a pick-and-lift task with a 7-DoF Franka robot arm trained in RLBench - a recently introduced benchmark for robotic manipulation - using state-of-the-art TD3 deep reinforcement learning and OpenAI’s evolutionary strategy. We show that it is a flexible approach without the need of significant fine-tuning and interference with the main objective even across different policy search methods and tasks of different complexity. We evaluate our approach in simulation and on the real robot. Our project website with videos and further results can be found at: https://sites.google.com/view/redundant-action-bias
APA
Al-Hafez, F. & Steil, J.J.. (2022). Redundancy Resolution as Action Bias in Policy Search for Robotic Manipulation. Proceedings of the 5th Conference on Robot Learning, in Proceedings of Machine Learning Research 164:981-990 Available from https://proceedings.mlr.press/v164/al-hafez22a.html.

Related Material