Google’s new Gemini Pro is worse than GPT-3

Gemini is the latest addition to Google DeepMind's series of large language models (LLMs). It stands out as it marks the first instance where reported results rival the performance of the OpenAI GPT model series across a diverse range of tasks. Specifically, Gemini's "Ultra" version is said to outshine GPT-4 on various tasks, while the "Pro" version is deemed comparable to GPT-3.5. However, the lack of detailed evaluation information and model predictions being made public hinders the ability to replicate, scrutinize, and thoroughly analyze the implications of these impactful findings.

So, let's see how Gemini actually compares to the GPT Models:

​

Table showing the performance difference between GPT-3.5, GPT-4 Turbo and Gemini Pro

Despite long development, Gemini Pro’s capability lags behind not only GPT-3.5 but also OpenAI’s advanced GPT-4 models. Gemini Pro, however, excels in translation tasks across certain languages but shows a strong content moderation tendency, blocking responses in several language pairs. The findings suggest that while Google is a significant player in AI, its latest generative AI offering still trails behind OpenAI’s established models.

P.S If you love this AI stuff just as much as I do then consider checking out my newsletter