Eliezer Yudkowsky often mentions that "we don’t really know what’s going on inside the AI systems". What does it mean?

I don't know much about inner workings of AI but I know that key components are neural networks, backpropagation, gradient descent and transformers. And apparently all that we figured out throughout the years and now we just using it on massive scale thanks to finally having computing power with all the GPUs available. So in that sense we know what's going on. But Eliezer talks like these systems are some kind of black box? How should we understand that exactly?