What lies outside the "regular" embeddings space of an LLM?

By definition an llm is just a manifold in a space with (whatever dimension of a single token)* times (context length) dimensions.

human text is naturally going to cluster over certain regions and since neural networks are defined over the entire space this means that there are regions where the LLM is extrapolating into something completely outside any human text it has seen.

Now my question, is there any research that investigates this? look at the boundaries of an LLM? or really anything on the topology of an LLM?

My guess is that most of it is going to be gibberish input tokens producing a gibberish output token, but there has to be somethings of interest.