Another issue: Gemini can’t do tool calling and (forced) json output at the same time

If you want to use application/json as the specified output in the request, you can’t use tools

So if you need both, you either hope it gives you correct json when using tools (which many times it doesn’t). Or you have to do two requests, one for the tool calling, another for formatting

At least, even if annoying, this issue is pretty straightforward to get around

Unfortunately Gemini isn't the only culprit here. I've had major problems with ChatGPT reliability myself.

Small things like this or the fact that AI studio still has issues with simple scrolling confuse me. How does such a brilliant tool still lack such basic things?

This is my perception as well.

Gemini 2.5 Pro is _amazing_ for software architecture, but I just get tired of poking it along. Sonnet does well enough.

chatgpt also has lots of reliability issues

I wonder if this is because a memory cap was reached at that output token. Perhaps they route conversations to different hardware depending on how long they expect it to be.

When this happened to me it was because, I can only guess, it was the Gemini servers were overloaded. Symptoms: Gemini model, Opaque API wrapper error, truncated responses. To be fair the Anthropic servers are overloaded a lot too but they have a clear error. I gave Gemini a few days on the bench and it fixed itself without any client side changes. YMMV.

Half my requests get retried because they fail, I've contributed to a ticket in June, with no fix yet.

That used to happen a lot in ChatGPT too.

The latest comment on that issue is someone saying there's a fix available for you to try.

Yes agree, it was totally broken when I tested the API two months ago. Lots of failed to connect and very slow response time. Hoping the update fixes these issues.

> I've been running into it consistently, responses that just stop mid-sentence

I’ve seen that behavior when LLMs of any make or model aren’t given enough time or allowed enough tokens.

FWIW, I think GLM-4.5 or Kimi K2 0905 fit the bill pretty well in terms of complete and consistent.

(Disclosure: I'm the founder of Synthetic.new, a company that runs open-source LLMs for monthly subscriptions.)

I wonder if [good examples of] SVGs of pelicans on bikes are "being introduced" into training sets. Some of the engineers who work on this stuff are the kind to hang out here.

Who wins in the end? the frogs? the ducks? or the pelicans?

That's why people called the second version of Sonnet v3.5 simply v3.6, and Anthropic acknowledged that by naming the next version v3.7

It's pretty common to refer to models by the month and year they were released.

For example, the latest Gemini 2.5 Flash is known as "google/gemini-2.5-flash-preview-09-2025" [1].

[1]: https://openrouter.ai/google/gemini-2.5-flash-preview-09-202...

2.5 is not the version number, it's the generation of the underlying model architecture. Think of it like the trim level on a Mazda 3 hatchback. Mazda already has the Mazda 3 Sport in their lineup, then later they release the Mazda 3 Turbo which is much faster. When they release this new version of the vehicle its not called the Mazda 4... that would be an entirely different vehicle based on a new platform and powertrain etc (if it existed). The new vehicle is just a new trim level / visual refresh of the existing Mazda 3.

That's why Google names it like this, but I agree its dumb. Semver would be easier.

That would be even more confusing because then it is unclear whether 2.6 Flash is better than 2.5 Pro.

Maybe they’re signalling it’s more of a bug fix?

I would be surprised if this dichotomy you're painting holds up to scrutiny.

My understanding is Gemini is not far behind on "intelligence", certainly not in a way that leaves obvious doubt over where they will be over the next iteration/model cycles, where I would expect them to at least continue closing the gap. I'd be curious if you have some benchmarks to share that suggest otherwise.

Meanwhile, afaik something Google has done, and perhaps relates back to your point re "latency/TPS/cost dimensions" that other providers aren't doing as much is integrating their model into interesting products beyond chat, at a pace that seems surprising given how much criticism they had been taking for being "slow" to react to the LLM trend.

Besides the Google Workspace surface and Google search, which now seem obvious - there are other interesting places where Gemini will surface - https://jules.google/ for one, to say nothing of their experiments/betas in the creative space - https://labs.google/flow/about

Another I noticed today: https://www.google.com/finance/beta

I would have thought putting Gemini on a finance dashboard like this would be inviting all sorts of regulatory (and other) scrutiny... and wouldn't be in keeping with a "slow" incumbent. But given the current climate, it seems Google is plowing ahead just as much as anyone else - with a lot more resources and surface to bring to bear. Imagine Gemini integration on Youtube. At this point it just seems like counting down the days...

  2025-09-26T14:32:10Z
  2025-09-26T14:32:10Z200s
  2025-09-26T14:32:10Z200s600s
  2025-09-26T14:32:10Z200s600s300s

    text = re.sub(r'(\*|_)(.+?)\1', replace_italic, text, flags=re.DOTALL)

  import re
  
  def markdown_to_ansi(markdown_string: str) -> str:
      ansi_string = markdown_string

      ANSI_BOLD = "\033[1m"
      ANSI_ITALIC = "\033[3m"
      ANSI_UNDERLINE = "\033[4m"
      ANSI_RESET = "\033[0m"
  
      ansi_string = re.sub(r'\*\*(.*?)\*\*', f'{ANSI_BOLD}\\1{ANSI_RESET}', ansi_string)
      ansi_string = re.sub(r'__(.*?)__', f'{ANSI_BOLD}\\1{ANSI_RESET}', ansi_string)
  
      ansi_string = re.sub(r'\*(.*?)\*', f'{ANSI_ITALIC}\\1{ANSI_RESET}', ansi_string)
      ansi_string = re.sub(r'\_(.*?)\_', f'{ANSI_ITALIC}\\1{ANSI_RESET}', ansi_string)
  
      ansi_string = re.sub(r'\+\+(.*?)\+\+', f'{ANSI_UNDERLINE}\\1{ANSI_RESET}', ansi_string)
  
      # A more robust solution for nesting would require a parsing library or a more complex state machine.
      # However, for simple cases, applying in order can layer them correctly.
  
      return ansi_string

    def markdown_ansi_code_subst(mdstr: str, src_pattern: str, replacement_start: str, replacement_end: str) -> str:
        while src_pattern in mdstr:
            mdstr = mdstr.replace(src_pattern, replacement_start, 1)
            mdstr = mdstr.replace(src_pattern, replacement_end, 1)
        return mdstr

Can't agree with that. Gemini doesn't lead just on price/performance - ironically it's the best "normie" model most of the time, despite it's lack of popularity with them until very recent.

It's bad at agentic stuff, especially coding. Incomparably so compared to Claude and now GPT-5. But if it's just about asking it random stuff, and especially going on for very long in the same conversation - which non-tech users have a tendency to do - Gemini wins. It's still the best at long context, noticing things said long ago.

Earlier this week I was doing some debugging. For debugging especially I like to run sonnet/gpt5/2.5-pro in parallel with the same prompt/convo. Gemini was the only one that, 4 or so messages in, pointed out something very relevant in the middle of the logs in the very first message. GPT and Sonnet both failed to notice, leading them to give wrong sample code. I would've wasted more time if I hadn't used Gemini.

It's also still the best at a good number of low-resource languages. It doesn't glaze too much (Sonnet, ChatGPT) without being overly stubborn (raw GPT-5 API). It's by far the best at OCR and image recognition, which a lot of average users use quite a bit.

Google's ridiculously bad at marketing and AI UX, but they'll get there. They're already much more than just a "bang for the buck" player.

FWIW I use all 3 above mentioned on a daily basis for a wide variety of tasks, often side-by-side in parallel to compare performance.

    ===============================
    Got it — *compliment on the info you've shared*, *informal summary of task*. *Another compliment*, but *downside of question*.
    ----------
    (relevant emoji) Bla bla bla
    1. Aspect 1
    2. Aspect 2
    ----------

    *Actual answer*

    -----------
    (checkmark emoji) *Reassuring you about its answer because:*

    * Summary point 1
    * Summary point 2
    * Summary point 3

    Would you like me to *verb* a ready-made *noun* that will *something that's helpful to you 40% of the time*?
    ===============================

> because these Gemini models sometimes feel downright lobotomized compared to claude or gpt-5.

I'm using Gemini (2.5-pro) less and less these days. I used to be really impressived with its deep research capabilities and ability to cite sources reliably.

The last few weeks, it's increasingly argumentative and incapable of recognizing hallucinations around sourcing. I'm tired of arguing with it on basics like RFCs and sources it fabricates, won't validate, and refuses to budge on.

Example prompt I was arguing with it on last night:

> within a github actions workflow, is it possible to get access to the entire secrets map, or enumerate keys in this object?

As recent supply-chain attacks have shown, exfiltrating all the secrets from a Github workflow is as simple as `${{ toJSON(secrets) }}` or `echo ${{ toJSON(secrets) }} | base64` at worse. [1]

Give this prompt a shot! Gemini won't do anything except be obstinately ignorant. With me, it provided a test case workflow, and refused to believe the results. When challenged, expect it to cite unrelated community posts. Chatgpt had no problem with it.

[1] https://github.com/orgs/community/discussions/174045 https://github.com/orgs/community/discussions/47165

IMO the race for Latency/TPS/cost is entirely between grok and gemini flash. No model can touch them (especially for image to text related tasks), openai/anthropic seem entirely uninterested in competing for this.

Agree, Gemini is soooooo freaking fast, but I rarely use it personally because Anthropic/OpenAI model have such a better output

10 years ago: "before you marry someone, put the person in front of a really slow internet connection"

today: "before you marry someone, put the person in front of a slow AI model"

;-)

We had to drop Gemini api cause it was so unreliable in production, no matter how long you waited.

The other day I heard gpt-5 was really an efficiency update

2.5 Flash is the first time I've felt AI has become truly useful to me. I was #1 AI hater but now find myself going to the Gemini app instead of Google search. It's just better in every way and no ads. The info it provides is usually always right and it feels like I have the whole generalized and accurate knowledge of the internet at my fingertips in the app. It's more intimate, less distractions. Just me and the Gemini app alone talking about kale's ideal germination temperature, instead of a bunch of mommy bloggers, bots, and SEO spam.

Now how long can Google keep this going and cannibalizing how they make money is another question...

I think “Non-AI summary” is going to become a thing. I already enjoyed reading it more because I knew someone had thought about the content.

I'm stealing "Non-AI Summary"

Any idea what "output token efficiency" refers to? Gemini Flash is billed by number of input/output tokens, which I assume is fixed for the same output, so I'm struggling to understand how it could result in lower cost. Unless of course they have changed tokenization in the new version?

Thank you for this, seems like an iterative improvement.

This is also my beef...

Anthropic kind of did the same thing [1] except it back-fired recently with the cries of "nerfing".

We buy these tokens, which are very hard to do in limited tiers, they expire after only a year, and we don't even know how often the responses are changing in the background. Even a 1% improvement or reduction I would want disclosed.

Really scary foundation AI companies are building on IMO. Transparency and access is important.

[1] https://status.claude.com/incidents/h26lykctfnsz

I wouldn't call that a nitpick, it's a major annoyance. Version numbers become useless with that kind of policy.

I would assume that it will supersede the model that they currently have. So eventually 2.5 flash will be the new and improved 2.5 Flash rather than 2.6.

Same way that openai updated their 4-o models and the like, which didn't turn out so well when it started glazing everyone and they had to revert it (maybe that was just chat and not api)

Google has historically always made bad UX choices like this. Conway’s law definitely applies here. Too many different silos building every Google project.

Gemini 2.5 Flash runs circles around ChatGPT 5 for many of my tasks, I’m surprised it’s not more popular than it is.

Not sure prices are changed though. :/

My one big problem with OpenRouter is that, as far as I can tell, they don't provide any indication of how many companies are using each model.

For all I know there are a couple of enormous whales on there who, should they decide to switch from one model to another, will instantly impact those overall ratings.

I'd love to have a bit more transparency about volume so I can tell if that's what is happening or not.

I know we have a lot of workloads at my company on older models no one has bothered to upgrade yet

Price, 2.0 Flash is cheaper than 2.5 Flash but still very good model.

Gemini 2.0 Flash is the best fast non reasoning model by quite a margin. Lot of things doesn't require any reasoning.

Maybe the same reason why they kept the name for the 2.5 Flash update.

People are lazy at pointing to the latest name.

2.0 Flash is significantly cheaper than 2.5 Flash, and is/was better than 2.5-Flash-Lite before this latest update. It's a great workhorse model for basic text parsing/summary/image understanding etc. Though looks like 2.5-Flash-Lite will make it redundant.

Why is Grok so popular

It’s cheaper and faster. What’s not to understand?

I've been finding it leaps and bounds above other models but I'm only using it via aistudio. I haven't tried any IDE integration or similar, so can't talk to that. I do still have to tell it to stop it with the effusive praise (I guess that also helps reduce context windows)

I have the same sentiment. I've never really had success using Gemini outside of translation. Although, even with that, Gemini would often refuse and I had to remind it that it does actually know other languages.

My most recent trials output single commas as responses to basic questions or it simply refuses the task on ethical grounds such as generating a photo of a backpack wearing a hoodie for some reason (it claimed harmful stereotypes and instead generated an ape).

Refusing to do perfectly ethical tasks is probably the most consist problem I've had.

I use Gemini almost exclusively for coding and 2.5 Pro is extremely good at it. It has revised hundreds of lines of academic code for me at a time and the results run correctly with only minor revision.

I will also say whatever they use for the AI search summary is good enough for me like 50% of the time I google something, but those are generally the simpler 50% of queries.

It depends on what you use it for. For answering questions I tend to prefer GPT-5, but for writing (e.g. turn these informally written ideas/bullet points into a report/proposal/etc., now shorten it a bit, emphasize this idea more, etc.) it's the best by far IMHO.

I agree. I think it comes down OpenAI's superior post-training.

ChatGPT is better at:

A) Interpreting what I'm asking it for me needing to provide additional explicit context.

B) Formatting answers in a way that are easily digestible.

> Google Workspace at my job, so Gemini is baked in.

I think the "baked in" Gemini models are different, try using Gemini through the actual Gemini site.

Maybe you are using it wrong.

Did they? I'm looking at the Artificial Analysis leaderboard site now and I only see price as USD/1M tokens.

Don’t forget they also have two versions for their genaisdk and you can also use their genaisdk through vertex great! Best part is all LLMs get horribly confused as well and mix different sdks etc.

I wonder how Gemini subscribers feel!

There several reasons responses from the same model might vary:

- "temperature" - intentional random sampling from the most likely next tokens to improve "creativity" and help avoid repetition

- quantization - running models with lower numeric precision (saves on both memory and compute, without impacting accuracy too much)

- differences in/existence of a system prompt, especially when using something end-user-oriented like Qwen Chat

- not-quite-deterministic GPU acceleration

Benchmarks are usually run at temperature zero (always take the most likely next token), with the full-precision weights, and no additions to the benchmark prompt except necessary formatting and stuff like end-of-turn tokens. They also usually are multiple-choice or otherwise expect very short responses, which leaves less room for run-to-run variance.

Of course a benchmark still can't tell you everything - real-world performance can be very different.

That's a difference in the system prompt, not the model itself.

I can't speak to qwen, but something interesting with Deepseek is that the official API supports almost no parameters, while the vllm hosts on openrouter do. The experience you get with the rehosters is wildly different since you can use samplers.

Yeah, why is it that working with AI makes people completely forget what version numbers mean?

gemini-2.5-flash-preview-09-2025 - what are they thinking?

I thought about joking that they had AI name it for them, but when I asked Gemini, it said that this name was confusing, redundant, and leads to unnecessarily high cognitive load.

Maybe Googlers should learn from their own models.

Because the number is model generation.

If you want stable models I think you could get that through Azure.

The version number is about the architecture of the model, the date is just about the last weights of the model.

Maybe disallow autoplay on your Youtube account can help. Gemini insert YT video in my answers as well, but they don't auto play.

This! I feel he suddenly started doing this even though I've told him to stop. And he knows, every time he tells me he's so sorry. It feels like Google is already monetizing Gemini for their ad market.

Might be be builtin to the model because it is impossible to remove completely...

And I say this because, I added about 50 prompts in the settings to prevent video recommendations and to remove any links to videos. but I still get text saying "the linked video explains this more" even though there is no linked video.

This is not a bad way to monetise the free tier. Non of the other token providers found any way to monetise the free tier but Gemini is doing it on almost every prompt.

I think that the Gemini 3 pro might be next month I am not sure.

can I get the sources of your rumour please? (Yes I know that I can search it but I would honestly prefer it if you could share it, thanks in advance!)

Because they want to retain the ability to do silent changes. They can't let people get used to stable version == stable result.

This is not my experience. In my experience Gemini 2.5 Pro is the best model in every use-case I tried. There are a few very hard (graduate level) logic or math problems that Claude 4.1 Opus edged-out over Gemini 2.5 Pro, but in general if you have no idea which model will perform best on a difficult question, imho Gemini 2.5 Pro is a safer bet especially since it's significantly cheaper. Gemini 2.5 Flash is really good but imho not nearly as good as Pro in (1) research math (2) creative/artistic writing (3) open ended programming debugging.

On the other hand, I do prefer using Claude 4 Sonnet on very open-ended agentic programming tasks because it seems to have a better integration with VSCode Copilot. Gemini 2.5 Pro bugs out much more often where Claude works fine almost every time.

Yeah that's how I feel too. Flash is less verbose and every LLM nowadays seems to be designed by some low-taste people who reward the model for falsely hedging (i.e. "The 2024 Corolla Cross usually has an X gallon gas tank") on stuff that isn't at all variable or questionable. This false hedging is way more of an issue than hallucinations in my experience and the "smarter" 2.5 Pro is not any better at avoiding this issue than Flash

Also 2.5 Pro is often incapable of searching and will hallucinate instead. I don't know why. It will claim it searched and then return some made up results instead. 2.5 Flash is much more consistently capable of searching

I tried to put Pro deep research on an actual research task and it didn’t even return anything just kept on working.

FWIW, the versions are not semver but they do follow a defined and regular version schema: https://ai.google.dev/gemini-api/docs/models#model-versions.

2.5 isn't the version number, its the model generation. it would only be updated when the underlying model architecture, training, etc are updated. this release is, as the name implies, the same model but likely with hardware optimizations, system prompt, and fine-tuning tweaks applied.

And this existing model’s update is evidence how? What were your expectations of this update?

I actually even agree that the progress is plateauing, but your comment is a non-sequitur.

This is a performance update to a previous generation model. It's not a new model.

Not really. A lot of new amazing Qwen models just dropped.

ChatGPT and Claude are mistaken if they think it is incorrect. The parallelism in verb tenses is between "continuing to deliver" and "improving the efficiency". It's a bit wordy, but definitely not wrong.

"Improving the efficiency" sounds fine to me (a native English speaker), what's wrong with it in your opinion?

This is pedantic. It's perfectly fine usage in non-formal English speaking. What's more - who gives a shit? By your own standards, you're inserting a quote in the middle of your comment in an arguably similarly "awkward" way.