But this turned out to be better performing than our current implementation of Unstructured/Textract. At pretty much the same cost.
I've tested almost every variant of document OCR over the past year, especially trying things like table / chart extraction. I've found the rules based extraction has always been lacking. Documents are meant to be a visual representation after all. With weird layouts, tables, charts, etc. Using a vision model just make sense!
In general, I'd categorize this solution as slow, expensive, and non deterministic. But 6 months ago it was impossible. And 6 months from now it'll be fast, cheap, and probably more reliable!
Here’s our pricing comparison:
*Gemini Pro* - $0.66 per 1k image inputs (batch) - $1.88 per text output (batch API, 1k tokens) - 395 pages per dollar
*Gemini Flash* - $0.066 per 1k images (batch) - $0.53 per text output (batch API, 1k tokens) - 1693 pages per dollar
*GPT-4o* - $1.91 per 1k images (batch) - $3.75 per text output (batch API, 1k tokens) - 177 pages per dollar
*GPT-4o-mini* - $1.91 per 1k images (batch) - $0.30 per text output (batch API, 1k tokens) - 452 pages per dollar
[1] https://community.openai.com/t/super-high-token-usage-with-g...
For our testing we ran a 1000 page document set, all treated as images. We got to about 25M input / 0.4M output tokens for 1000 pages. Which would be a pretty noticeable difference based on the listed token prices.
gpt-4o-mini => (24M/1M * $0.15) + (0.4M/1M * 0.60) = $4.10
gpt-4o => (24M/1M * $5.00) + (0.4M/1M * 15.00) = $126.00
Are you suggesting that this wasn't intentional? The name is clearly a play on "zero shot" + "xerox"
We're not planning to name a company after it or anything, just the OS tool. And if xerox sues I'm sure we could rename the repo lol.
I actually had a leg to stand on (my use was not infringing at all when I started using it), and I came out of it somewhat cash-positive, but I absolutely never want to go through anything like that ever again.
> Yup definitely a play on the name. Also the idea of photocopying a page,
But you? My God, man.
With these words you have already doomed yourself.
Best wishes.
At least they didn't say "xeroxing a page".
I guess I just don’t understand - how are you proceeding as if this is an acceptable starting point?
With all respect, I don’t think you’re taking this seriously, and it reflects poorly on the team building the tool. It looks like this is also a way to raise awareness for Omni AI? If so, I’ve gotta be honest - this makes me want to steer clear.
Bottom line, it’s a bad idea/decision. And when bad ideas are this prominent, it makes me question the rest of the decisions underlying the product and whether I want to be trusting those decision makers in the many other ways trust is required to choose a vendor.
Not trying to throw shade; just sharing how this hits me as someone who has built products and has been the person making decisions about which products to bring in. Start taking this seriously for your own sake.
Edit: Reading the comments below, yes, it was.
Very disrespectful behavior.
am I only one finding these sort of takes silly in a cumulative globalized world with instant communications? There are so many things to be named, everything named is instantly available around the world, so many jurisdictions to cover - not all providing the same levels of protections to "trademarks".
Are we really suggesting this issue is worth defending and spending resources on?
what is the ground for confusion here? that a developer stumbles on here and thinks zerox is developed/maintained by xerox? this developer gets confused but won't simply check who is the owner of the repository? What if there's a variable called zerox?
I mean, I get it: the whole point of IP at this point is really just to create revenue streams for the legal/admin industry so we should all be scared and spend unproductive time naming a software dependency
Absolutely.
Sure, sometimes non-competing products have the same name. Or products sold exclusively in one country use the same name as a competitor in a different country. There's also companies that don't trademark or protect their names. Often no one even notices the common name.
That's not whats happening here. Xerox is famously litigious about their trademark; often used as a case study. The product competes with Xerox OCR products in the same countries.
It's a strange thing to be cavalier about and to openly document intent to use a sound-alike name. Besides, do you really want people searching for "Zerox OCR" to land on a Xerox page? There's no shortage of other names.
All 5 minutes it would take to name it something else?
1. Prompt with examples. I included an example image with an example transcription as part of the prompt. This made GPT make fewer mistakes and improved output accuracy.
2. Confidence score. I extracted the embedded text from the PDF and compared the frequency of character triples in the source text and GPT’s output. If there was a significant difference (less than 90% overlap) I would log a warning. This helped detect cases when GPT omitted entire paragraphs of text.
- Request #1 => page_1_image
- Request #2 => page_1_markdown + page_2_image
- Request #3 => page_2_markdown + page_3_image
What are character triples? Are they trigrams?
What did you use to extract the embedded text during this step? Other than some other OCR tech
The $10/1000 pages model includes layout detection (headers, etc.) as well as key-value pairs and checkbox detection.
I have continued to do proofs of concept with Gemini and GPT, and in general any new multimodal model that comes out but have it is not on par with the checkbox detection of azure.
In fact the results from Gemini/GPT4 aren't even good enough to use as a teacher for distillation of a "small" multimodal model specializing in layout/checkbox.
I would like to also shout out surya OCR which is up and coming. It's source available and free for under a certain funding or revenue milestone - I think $5m. It doesn't have word level detection yet but it's one of the more promising non-hyper scaler/ heavy commercial OCR tools I'm aware of.
const systemPrompt = `
Convert the following PDF page to markdown.
Return only the markdown with no explanation text.
Do not exclude any content from the page.
`;
For each subsequent page:
messages.push({
role: "system",
content: `Markdown must maintain consistent formatting with the following page: \n\n """${priorPage}"""`,
});Could be handy for general-purpose frontend tools.
Are you supporting the Batch API from OpenAI? This would lower costs by 50%. Many OCR tasks are not time-sensitive, so this might be a very good tradeoff.
> This is not an OCR problem, but of course, I can't have a look into the software itself, maybe OCR is still fiddling with the data even though we switched it off.
But the point stands either way; LLMs are prone to hallucinations already, so I would not trust them to not make a mistake in OCR because they thought the page would probably say something different than it does.
It was a problem with employing the JBIG2 compression codec, which cuts and pastes things from different parts of the page to save space.
> But the point stands either way; LLMs are prone to hallucinations already, so I would not trust them to not make a mistake in OCR because they thought the page would probably say something different than it does.
Anyone trying to solve for the contents of a page uses context clues. Even humans reading.
You can OCR raw characters (performance is poor); use letter frequency information; use a dictionary; use word frequencies; or use even more context to know what content is more likely. More context is going to result in many fewer errors (of course, it may result in a bigger proportion of the remaining errors seeming to have significant meaning changes).
A small LLM is just a good way to encode this kind of "how likely are these given alternatives" knowledge.
With a traditional OCR architecture maybe you'll get a symbol or two wrong, but an LLM can give you entirely new words or numbers not in the document, or even omit sections of the document. I'd never use an LLM for OCR like this.
Done properly, this should strictly improve the results.
And additionally, this also might work if you are feeding the output into a bunch of humans to proof.
Check it out, https://cluttr.ai
Runs entirely in browser, using OPFS + WASM.
Try with any PDF document in the playground - https://pg.llmwhisperer.unstract.com/
You are an expert in PDFs. You are helping a user extract text from a PDF.
Extract the text from the image as a structured json output.
Extract the data using the following schema:
{Page.model_json_schema()}
Example:
{{
"title": "Title",
"page_number": 1,
"sections": [
...
],
"figures": [
...
]
}}
https://binal.pub/2023/12/structured-ocr-with-gpt-vision/This will perform worse in cases where whatever understanding the large model has of the contents is needed to recognize indistinct symbols. But it will avoid cases where that very same understanding causes contents to be understood incorrectly due to the model’s assumptions of what the contents should be.
At least in my limited experiments with Claude, it’s easy for models to lose track of where they’re looking on the page and to omit things entirely. But if segmentation of the page is explicit, one can enforce that all contents end up in exactly one segment.
So far, I have collected over 500 receipts from around 10 countries with 30 different supermarkets in 5 different languages.
What has worked for me so far is having control over OCR and processing (for formatting/structuring) separately. I don't have the figures to provide a cost structure, but I'm looking for other solutions to improve both speed and accuracy. Also, I need to figure out a way to put a metric around accuracy. I will definitely give this a shot. Thanks a lot.
Why did you choose markdown? Did you try other output formats and see if you get better results?
Also, I wonder how HMTL performs. It would be a way to handle tables with groupings/merged cells
I've not seen a meaningful difference between either, except when it comes to tables. It seems like HTML tends to outperform markdown tables, especially when you have a lot of complexity (i.e. tables within tables, lots of subheaders).
The majority of comments are related to prices and qualities
Also, is there any movements about product detection? These days I'm looking for solutions that can recognize goods in high accuracy and show [brand][product_name][variant]
I like the optimism.
I've needed to include human review when using previous generation OCR software; when I needed the results to be accurate. It's painstaking, but the OCR offered a speedup over fully-manual transcription. Have you given any thought to human-in-the-loop processes?
On the human in loop side, it's really use case specific. For a lot of my company's work, it's focused on getting trends from large sets of documents.
Ex: "categorize building permits by municipality". If the OCR was wrong on a few documents, it's still going to capture the general trend. If the use case was "pull bank account info from wire forms" I would want a lot more double checking. But that said, humans also have a tendency to transpose numbers incorrectly.
That said, there's huge value in lossy transcription elsewhere, as long as you can account for the errors they introduce.
I think that is one of the key findings from GraphRAG paper: the gpt can replace the human in the loop.
However, commenters around here noted that these have likely not been fine-tuned to correlate with accuracy - for plaintext LLM uses. Would be interested in hearing finding for MLLM use-cases!
That said, sometimes no amount of understanding will improve the OCR output because a structure in a document cannot be converted to a one-dimensional string (short of using HTML/CSS or something). Maybe we'll get image -> HTML models eventually.
Probably the model is hallucinating and adding "Hungarian language is not installed for Tesseract" to the response.
I would love to use this in a project if it could also caption embedded images to produce something for RAG...
Just
maintainFormat: true
did not seem to have any effect in my testing.The main thing we're doing is converting documents to a series of images, and then aggregating the response. So we should be model agnostic pretty soon.