🟢 Gemini 1.5 Pro

What is Gemini 1.5 Pro?

Gemini 1.5 Pro is Google's paid Gemini product, and it's designed strictly to optimize reasoning capabilities along with a larger context window than before.

Its robust performance on benchmarks solidifies its position as one of the best models out there for reasoning, in particular over long-form content, and its trademark native multimodal capabilities continue Google's stronghold in that aspect of LLMs.

Advantages of Gemini 1.5 Pro

• Strong reasoning: Gemini scored very well on several benchmarks, including MMLU, which it scored an 86% on, and the GPQA diamond, which it scored a 61% on, placing it second out of all other LLMs, behind Chat-GPT o1-preview.

• Large context window: Gemini 1.5 Pro boasts a 2 million token context window, which is almost eight times the window of the closest non-Gemini model. This means it can do Q and A over entire codebases and video series.

• Multimodal: 1.5 Pro can natively process multiple data types, including text, images, audio, and video, within a single conversation and/or prompt.

• Cheap: Despite its incredible reasoning capabilities, Gemini 1.5 Pro is actually cheaper than average, at 2.2 USD per million tokens.

So 1.5 Pro has the largest context window of any other LLM, is natively multimodal, and performs extremely well on reasoning benchmarks despite having a very low price in comparison to other models with similar reasoning abilities.

Comparisons to Other Models

• Gemini 1.5 Flash: Gemini 1.5 Flash is Google's free version of Gemini, and it's faster and cheaper than 1.5 Pro, at 190 output tokens per second and 0.1 USD per 1M tokens compared to Pro's 59 output tokens per second and 2.2 USD per 1M tokens. Pro is more capable than Flash at reasoning, however, with a score of 86% on the MMLU compared to Flash's 81% and a score of 61% on the GPQA Diamond compared to Flash's 45%.

• Gemma 2 (27B): Gemma 2 has a vastly smaller context window size than Pro, at 8.5k tokens compared to Pro's 2 million. Additionally, Pro is better at reasoning; Gemma 2 only scored a 77% on the MMLU and a 39% on the GPQA. At $0.30 per 1M tokens and 50 output tokens per second, Gemma 2 is cheaper but surprisingly slower than 1.5 Pro.

Conclusion

Gemini 1.5 Pro is a well-rounded Google model that sets itself apart with uniquely high scores on reasoning benchmarks, especially the GPQA Diamond. While its speed may leave much to be desired, it's abnormally low price in comparison to its capabilities, along with its multimodal functionality and extremely large context window, really help establish it as one of the better language models in the current market.