For a while we’ve been were hearing rumors GPT-4 is a trillion parameter model. Well in the last week some insiders have shed light on this. It appear the model is actually a Mixture of Experts (MoE), where each of the eight experts has 220B para…