Quick tech segway, skip to the heading below if you already know compiler theory.
Most people think of compilers as things that takes source code and produce a program. But a compiler is more general than that. A compiler is just a system that takes one set of symbols in and outputs a corresponding set of symbols.
Compilers operate on three levels: token, syntax and semantics. In a programming language like Python, a token might be "while" or "=". Syntax might be "variable = value". A semantics might be "a variable that is assigned to is automatically recorded in the lookup table for local variables."
Once you have identified the tokens, syntax and semantics of your input symbols then you just need to built the mapping between the input and output. In most compilers, this is the "emitter" which takes the internal representation of the input and walks through it, transforming it to the output symbols as it goes, applying various optimizations as it goes.
This is a gross over-simplification and misses many nuances, technologies, approaches, etc. But it gets us where we need to be for this conversation.
AI Model Compiling
It seems to me that we have obvious need for optimizing compilers in the model space. The problem is identifying the syntax and semantics of a neural network. This seems like something that AI could tackle, however, without having to achieve AGI.
Given a functional understanding of compiler optimization and of a model's structure, it should be possible to have an AI identify semantic patterns and re-arrange (even reduce) the syntax to get there.
Immediately obvious problems:
- Training the compiler model seems challenging at best.
- Only very small models could be compiled (number of tokens in).
- Proving that the compiler does not damage the model is probably impossible other than empirically.
Does anyone know if this is an area of active research?
[link] [comments]