Chris Nota, Philip S. Thomas: Is the Policy Gradient a Gradient? AAMAS 2020: 939-947