You might enjoy this paper[0] which shows that recurrent position encodings recover grid cell representations and maps to path integration found in a popular model of the hippocampus. This isn't terribly surprising since RNNs have shown this before[1, 2] but its an interesting connection.

[0] https://arxiv.org/abs/2112.04035

[1] https://arxiv.org/abs/1803.07770

[2] https://www.nature.com/articles/s41586-018-0102-6