The Writing Is on the Wall for Handwriting Recognition
88 points
6 days ago
| 15 comments
| newsletter.dancohen.org
| HN
benterix
1 hour ago
[-]
> If AI can diminish some of the monotony of research, perhaps we can spend more time thinking, writing, playing piano, and taking walks — with other people.

Whenever any progress is made, this is the logical conclusion. And yet, those who decide about how your time is being used, have an opposing view.

reply
notimetorelax
1 hour ago
[-]
I feel that we’re reaching a limit to our context switching. Any further process improvements or optimizations will be bottlenecked on humans. And I don’t think AI will help here as jobs will account for that and we’ll have to do context switching on even broader and more complex scopes.
reply
seethishat
59 minutes ago
[-]
I think the limit has been exceeded. That's the primary reason everything sort of sucks now. There is no time to slow down and do things right (or better).

IMO, cyber security, for example, will have to become a government mandate with real penalties for non-compliance (like seat belts in cars were mandated) in order to force organizations to slow down, and make sure systems are built carefully and as correctly as possible to protect data.

This is in conflict with the hurtling pace of garbage in/garbage out AI generated stuff we see today.

reply
ulbu
1 hour ago
[-]
maybe akin to how faster conputers bred programs that are slower than before.
reply
coolness
5 hours ago
[-]
Great post and amazing progress in this field! However, I have to wonder if some of these letters were part of the training data for Gemini, since they are well-known and someone has probably already done the painstaking work of transcribing them...
reply
lccerina
1 hour ago
[-]
Most likely, and probably inferring the structure on texts with "similar" writing forms. Tried with my handwriting (in italian) and the performance wasn't that stellar. More annoyingly, it is still a LLM and not a "pure" OCR, so some sentences were partially rephrased with different words than the one in the text. This is crucially problematic if they would be used to transcribe historical documents
reply
embedding-shape
1 hour ago
[-]
> Tried with my handwriting (in italian) and the performance wasn't that stellar.

Same here, for diaries/journals written in mixed Swedish/English/Spanish and with absolutely terrible hand-writing.

I'd love for the day where the writing is on the wall for handwriting recognition, which is something I bet on when I started with my journals, but seems that day has yet to come. I'm eager to get there though so I can archive all of it!

reply
GaggiX
1 hour ago
[-]
Are you sure to have used the Gemini 3.0 pro model? Maybe try increasing the media resolution on the AI studio if the text is small
reply
MrSkelter
3 hours ago
[-]
I have a personal corpus of letters between my grandparents in WW2. My grandfather fighting in Europe and my grandmother in England. The ability of Claude and ChatGPT to transcribe them is extremely impressive. Though I haven’t worked on them in months and this uses older models. At that time neither system could properly organize pages though and chatGPT would sometimes skip a paragraph.
reply
vertnerd
2 hours ago
[-]
I've also been working on half a dozen crates of old family letters. ChatGPT does well with them and is especially good at summarizing the letters. Unfortunately, all the output still has to be verified because it hallucinates words and phrases and drops lines here and there. So at this point, I still transcribe them by hand, because the verification process is actually more tiresome than just typing them up in the first place. Maybe I should just have ChatGPT verify MY transcriptions instead.
reply
embedding-shape
1 hour ago
[-]
It helps when you can see the confidence of each token, which downloadable weights usually gives you. Then whenever you (your software) detects a low confidence token, run over that section multiple times to generate alternatives, and either go with the highest confidence one, or manually review the suggestions. Easier than having to manually transcribe those parts at least.
reply
dmd
2 hours ago
[-]
Possibly, but given it can also read my handwriting- which is much, MUCH worse than Boole’s - with better accuracy than any human I’ve given it to- that’s probably not the explanation.
reply
suddenlybananas
5 hours ago
[-]
Shhhhh no one cares about data contamination anymore.
reply
spwa4
2 hours ago
[-]
Then write something down yourself and upload a picture to gemini.google.com or chatgpt. Hell, combine it. Make yourself a quick math test, print it, solve with pen and ask these models to correct it.

They're very good at it.

reply
sph
2 hours ago
[-]
Any self-hosted open source solution? I would like to digitize my paper notebooks but I do not want to use anything proprietary or that uses external services. What is the state of the art on the FOSS side?

Ideally something that I can train with my own handwriting. I had a look at Tesseract, wondering if there’s anything better out there.

reply
vintermann
1 hour ago
[-]
Regular handwriting there are many.

Historical handwriting, Gemini 3 is the only one which gave a decent result on a 19th century minutes from a town court in Northern Norway (Danish gothic handwriting with bleed through). I'm not 100% sure it's correct, but that's because it's so dang hard to read it to verify it. At least I see it gets many names, dates and locations right.

I've been waiting a long time for this.

reply
sph
1 hour ago
[-]
> Regular handwriting there are many.

Please share. I am out of the loop and my searches have not pointed me to the state of the art, which has seen major steps forward in the past 3 or 4 years but most of it seems to be closed or attached to larger AI products.

Is it even still called OCR?

reply
fragmede
1 hour ago
[-]
Totally not what you asked, but making an OCR model is a learning exercise for AI research students. Using the Kaggle-hosted dataset https://www.kaggle.com/datasets/landlord/handwriting-recogni... and a tutorial, eg https://pyimagesearch.com/2020/08/17/ocr-with-keras-tensorfl... you can follow along and train your own OCR model!
reply
embedding-shape
1 hour ago
[-]
Try various downloadable weights that has Vision, they're all good at different examples, running multiple ones and then finally something to aggregate/figure out the right one usually does the trick. Some recent ones to keep in the list: ministral-3-14b-reasoning, qwen3-vl-30b, magistral-small-2509, gemma-3-27b

Personally I found magistral-small-2509 to be overall most accurate, but it completely fails on some samples, while qwen3-vl-30b doesn't struggle at all with those same samples. So seems training data is really uneven depending on what exactly you're trying to OCR.

And the trade-off of course is that these are LLMs so not exactly lightweight nor fast on consumer hardware, but at least with the approach of using multiple you greatly increase the accuracy.

reply
macleginn
3 hours ago
[-]
I became convinced of this after the release of KuroNet: https://arxiv.org/pdf/1910.09433 (High-quality OCR of Japanese manuscripts, which look almost impossible to read.)
reply
pjmlp
4 hours ago
[-]
Maybe for English, for the other human languages I use, it is still kind of hit and miss, just like speaking recognition, even with English it suffices to have an accent that is off the standard TV one.
reply
macleginn
3 hours ago
[-]
As always, this depends on the amount of training data available. Japanese is another success story: https://digitalorientalist.com/2020/02/18/cursive-japanese-a...
reply
pjmlp
2 hours ago
[-]
Interesting, thanks for sharing.
reply
NitpickLawyer
4 hours ago
[-]
ee lay vhen!
reply
TonyTrapp
2 hours ago
[-]
They don't do Scottish accents!
reply
zkmon
50 minutes ago
[-]
It's painful to see that beautiful hand-writing of the past is now pretty much extinct. For me, handwriting of a person speaks a lot about them, not just their mind, but physical state as well.
reply
girvo
2 hours ago
[-]
> "transmitted": In the second line of the body, the word "transmitted" is crossed out in the original text

Am I nuts or is this wrong, not “perfect”?

It doesn’t look crossed out at all to me in the image, just some bleeding?

Still very impressive, of course

reply
__alexs
4 hours ago
[-]
Call me when it can do Russian Cursive.
reply
decimalenough
4 hours ago
[-]
Seems to do an OK job:

https://g.co/gemini/share/e173d18d1d80

This is a random image from Twitter with no transcript or English translation provided, so it's not going to be in the training data.

reply
shatsky
4 hours ago
[-]
No, transcription has nothing to do with written text, it guessed few words here and there but not even general topic. That's doctors note about patient visit, beginning with "Прием: состояние удовл., t*, но кашель / patient visit: condition is OK, t(temperature normal?) but coughing". But unreadable doctors handwriting is a meme...
reply
GaggiX
3 hours ago
[-]
That's Gemini 2.5 Flash btw

The result from Gemini 3 Pro using the default media resolution (the medium one): "(Заголовок / Header): Арсеньев (Фамилия / Surname - likely "Arsenyev")

    Состояние удовл-

    t N, кожные

    покровы чистые,

    [л/у не увел.]

    В зеве умерен. [умеренная]

    гипер. [гиперемия]

    В легких дыха-

    ние жесткое, хрипов

    нет. Тоны серд-

    [ца] [ритм]ичные.

    Живот мяг-

    кий, б/б [безболезненный].

    мочеисп. [мочеиспускание] своб. [свободное]

    Ds: ОРЗ [или ОРВИ]" and with the translation: "Arsenyev
Condition satisfactory. Temp normal, skin coverings [skin] are clean, lymph nodes not enlarged. In the throat [pharynx], moderate hyperemia [redness]. In the lungs, breathing is rigid [hard], no rales [crackles/wheezing]. Heart tones are rhythmic. Abdomen is soft, painless. Urination is free [unhindered]. Diagnosis: ARD (Acute Respiratory Disease)."
reply
myth_drannon
43 minutes ago
[-]
Right, it can do modern writing but anything older than a century ( church records and census)and it produces garbage. Yandex Archives figured that out and have CER in a single digit but they have the resources to collect immense data for training. I'm slowly building a dataset for finetuning TROCR model and the best it can do is CER 18% ... which is sort of readable.
reply
DarkNova6
4 hours ago
[-]
> Here’s Transkribus’s best guess at George’s letter to Maryann, above:

Transkribus got a new model architecture around the corner and the results look impressive. Not only for trivial cases like text, but also for table structures and layouting.

Best of all, you can train it on your own corpus of text to support obscure languages and handwriting systems.

Really looking forward to it.

reply
ferguess_k
36 minutes ago
[-]
Don't worry, handwriting itself has diminished throughout the decades since the introduction of computers an especially smart phones.

Ah, maybe I'll pick up Qin seal when I retire, if I retire.

reply
tigerlily
4 hours ago
[-]
Surely the true prize is to be able to ditch computers altogether and just write with pencil on paper.
reply
iamflimflam1
4 hours ago
[-]
If I went back in time to the 90s when I was doing my PhD I would absolutely blow my mind with how well handwriting OCR works now.
reply
th0ma5
4 hours ago
[-]
My question for OCR automation is always which digits within the numbers being read are allowed to be incorrect?
reply
lifestyleguru
2 hours ago
[-]
It feels unbelievable that in Europe literacy rate could be 10% of lower. Then I look at documents even as young as 150 years... fraktur, blackletter, elaborate handwritting. I guess I'm illiterate now.

Hopefully next generations will feel the same about legal contracts, law in general, and Java code bases. They're incomprehensible not because of fonts but because of unfathomable complexity.

reply
sph
2 hours ago
[-]
Which Europe and which century do you live in where literacy rate is below 10%?
reply
lifestyleguru
1 hour ago
[-]
Speaking about the past centuries.
reply
tpm
1 hour ago
[-]
You can learn fraktur or blackletter in a day and cyrillic in a few days, if you already know the latin alphabet.
reply
lifestyleguru
1 hour ago
[-]
> learn fraktur or blackletter in a day and cyrillic in a few days

Not a chance, sorry.

reply
nikanj
3 hours ago
[-]
The writing is on the wall for handwriting. Zoomers use speech recognition or touchscreen keyboards, millennials use keyboards. Boomers use pens
reply
lccerina
1 hour ago
[-]
I call out the Lindy effect. Handwriting survived printed characters, typewriters, and the last 50-70 years of computers and keyboards, it will survive this too.
reply