Neuro-symbolic approach to reinforcement learning in robotics

Master's Thesis

Bc. Tomáš Bisták

Supervisor: prof. Ing. Igor Farkaš, Dr.

Consultant: doc. RNDr. Martin Homola, PhD.

Annotation

Reinforcement learning (RL) has become a standard machine learning approach to solving sequential decision tasks, including robotics. Yet, purely experience-driven RL is a computationally intensive black-box with low level of explainability. Neuro-symbolic approaches offer to meet these challenges, also thanks to transparency of the symbolic level.

Aim

  1. Become familiar with RL literature and with selected neuro-symbolic approaches applicable to sequential tasks, with focus on robotic manipulation.
  2. Extend Delfosse et al.'s (2023) model to robotics domain and implement the reasoning component for predicting the consequences of robot's actions.
  3. Using a task with objects on a table, compare the hybrid approach with pure RL approach in terms of explainability and accuracy of predictions.

Plan

February - Early March 2024

Complete

  • Study the literature on the current, state-of-the-art neuro-symbolic approaches to RL.

Late March - Early April 2024

Complete

  • Implement a neuro-symbolic meta-controller for the Block's World problem.

Late April - Early May 2024

Backlog

  • Propose or find a method for generalizing observations and reasoning in order to transfer knowledge gained while learning in a specific environment to more general instances of the same problem.

September - November 2024

Complete

  • Prepare the environment for training the hierarchical model on the Block's World problem.
  • Implement the hierarchical model for the Block's World problem.

November - December 2024

In Progress

  • Adapt the neuro-symbolic meta-controller so that it can be effectively used in the hierarchical model.
  • Fine-tune the sub-symbolic controller.

Literature

  1. Delfosse, Q., Shindo, H., Dhami, D., & Kersting, K. (2023). Interpretable and Explainable Logical Policies via Neurally Guided Symbolic Abstraction. Advances in Neural Information Processing Systems, 36, 50838-50858. https://proceedings.neurips.cc/paper_files/paper/2023/file/9f42f06a54ce3b709ad78d34c73e4363-Paper-Conference.pdf
  2. Jiang, Z. & Luo, S. (2019). Neural Logic Reinforcement Learning. Proceedings of the 36th International Conference on Machine Learning, PMLR, 97, 3110-3119. https://proceedings.mlr.press/v97/jiang19a.html
  3. Kimura, D., Ono, M., Chaudhury, S., Kohita, R., Wachi, A., Agravante, D. J., Tatsubori, M., Munawar, A., & Gray, A. (2021). Neuro-Symbolic Reinforcement Learning with First-Order Logic. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 3505-3511. https://doi.org/10.18653/v1/2021.emnlp-main.283
  4. Riegel, R., Gray, A., Luus, F., Khan, N., Makondo, N., Akhalwaya, I. Y., Qian, H., Fagin, R., Barahona, F., Sharma, U., Ikbal, S., Karanam, H., Neelam, S., Likhyani, A., & Srivastava, S. (2020). Logical Neural Networks. arXiv. https://doi.org/10.48550/arXiv.2006.13155
  5. Verma, A., Murali, V., Singh, R., Kohli, P., & Chaudhuri, S. (2018). Programmatically Interpretable Reinforcement Learning. Proceedings of the 35th International Conference on Machine Learning, PMLR, 80, 5045-5054. https://proceedings.mlr.press/v80/verma18a.html
  6. Dong, H., Mao, J., Lin, T., Wang, Ch., Li, L., & Zhou, D. (2019). Neural Logic Machines. International Conference on Learning Representations (ICLR 2019), 4, 2574-2595. https://openreview.net/forum?id=B1xY-hRctX
  7. Glanois, C., Weng, P., Zimmer, M., Li, D., Yang, T., Hao, J., & Liu, W. (2022). A Survey on Interpretable Reinforcement Learning. arXiv. https://doi.org/10.48550/arXiv.2112.13112
  8. Lyu, D., Yang, F., Liu, B., & Gustafson, S. (2019). SDRL: Interpretable and Data-Efficient Deep Reinforcement Learning Leveraging Symbolic Planning. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 2970-2977. https://doi.org/10.1609/aaai.v33i01.33012970
  9. Xu, D. & Fekri, F. (2021). Interpretable Model-based Hierarchical Reinforcement Learning using Inductive Logic Programming. arXiv. https://doi.org/10.48550/arXiv.2106.11417
  10. Payani, A. & Fekri, F. (2019). Inductive Logic Programming via Differentiable Deep Neural Logic Networks. arXiv. https://doi.org/10.48550/arXiv.1906.03523
  11. Lamanna, L., Serafini, L., Saetti, A., and Gerevini, A., & and Traverso, P. (2022). Online Grounding of Symbolic Planning Domains in Unknown Environments. Proceedings of the 19th International Conference on Principles of Knowledge Representation and Reasoning, 511-521. https://doi.org/10.24963/kr.2022/53
  12. Ciravegna, G., Barbiero, P., Giannini, F., Gori, M., Liò, P., Maggini, M., & Melacci, S. (2023). Logic Explained Networks. Artificial Intelligence, 314, 103822. https://doi.org/10.1016/j.artint.2022.103822
  13. Bekkemoen, Y. (2024). Explainable reinforcement learning (XRL): a systematic literature review and taxonomy. Machine Learning, 113, 355-441. https://doi.org/10.1007/s10994-023-06479-7
  14. Quing, Y., Liu, S., Song, J., Wang, H., & Song, M. (2022). A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges. arXiv. https://doi.org/10.48550/arXiv.2211.06665
  15. Dierckx, L., Veroneze, R., & Nijssen, S. (2023). RL-Net: Interpretable Rule Learning with Neural Networks. Advances in Knowledge Discovery and Data Mining: 27th Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD 2023, Osaka, Japan, May 25-28, 2023, Proceedings, Part I, 95-107. https://doi.org/10.1007/978-3-031-33374-3_8