Section Insights
Evaluating Quality in Document Processing
How do you evaluate quality in OCR and document processing?
Vic, the founder of Data Lab, discusses the importance of parsing in document pipelines and how it affects downstream quality. He highlights common parsing failures and their implications for various workflows.
- Parsing is a crucial step in document processing pipelines.
- Common parsing errors can lead to significant downstream issues.
- Understanding the context of document processing is essential for quality evaluation.
Challenges with Edit Distance as a Metric
Why is edit distance not a reliable metric for benchmarking?
Vic explains that while edit distance measures character differences, it can misrepresent quality by penalizing formatting changes rather than content accuracy. This can lead to misleading evaluations of model performance.
- Edit distance can over-penalize formatting issues and under-penalize content errors.
- A high edit distance does not necessarily indicate poor quality if the content is correct.
- Choosing appropriate metrics is critical for accurate benchmarking.
Tailoring Metrics to Use Cases
How should benchmarks be tailored to different document processing use cases?
Vic emphasizes that different document processing pipelines require specific metrics based on their unique needs. He suggests segmenting benchmarks by use case to better evaluate model performance.
- Different pipelines have different priorities; metrics should reflect that.
- Segmenting benchmarks can reveal which models perform best for specific tasks.
- Validating metrics through practical tests is essential for reliable evaluation.
Practical Steps for Choosing a Parser
What steps should be taken to choose the right parser for document processing?
Vic outlines a process for selecting a parser, including sampling document types, generating verifiable questions, and understanding failure modes. He stresses the importance of accuracy in critical use cases.
- Sampling should cover all document types and edge cases.
- Verifiable questions help ensure the accuracy of the parser's output.
- Understanding the specific needs of downstream applications is crucial.
Skepticism Towards Benchmarks
How should individuals approach benchmarks and model evaluations?
Vic discusses the skepticism surrounding benchmarks due to potential manipulation and the commercial pressures to rank highly. He advises a careful, analytical approach to evaluating benchmarks.
- Be critical of benchmark claims and understand the context behind them.
- Commercial incentives can lead to biased benchmarking practices.
- A thorough evaluation process is necessary for trustworthy assessments.
Transcript
0:00 When everyone is number one on the benchmarks, how do you know how to evaluate quality? both of OCR and just more generally, I think a lot of the principles I'm going to talk about are generally applicable to benchmarking. So, I'm Vic. I'm the founder of Data Lab. And I've over the last 2 years built projects that have more than 80K GitHub stars collectively, including Marker, Surya, Chandra, and Indra. several of those are currently state-of-the-art, several were state-of-the-art in the past in OCR and and other areas as well. they're used by organizations like 11 Labs, Anthropic, Llama, and many other enterprises. And we recently released two benchmarks for extraction and for parsing. So, these are specific to documents, but extraction is extracting specific values from documents, parsing is how do you get all of the text out. Our models also do other things, but all related to documents and document pipelines. So, if you're running a document pipeline, so if you're going from scan through to parsing, chunking, retrieving, this is a rag pipeline. There's many other types of pipelines, extraction, accessibility, many others. parsing becomes your crucial step because all of the downstream quality depends on it.
1:14 So, I'm going to talk a lot about parsing today, but a lot of what I talk about is going to be generally applicable to different steps in the document pipeline. So, some examples of parsing failures. a three can be read as an eight, and all of your downstream, if you're doing KYC or financial workflows or legal workflows on top of that, everything just becomes subtly wrong. we had a blog post about this recently. models will output values in the wrong script, so Cyrillic instead of English, sometimes even Chinese instead of English. It's quite an interesting phenomenon. and then sometimes the format that things are output on can change your downstream pipeline as well. So, today I'm going to talk about these errors, how you can catch them, how you can eval. So, first of all, the state of benchmarking in general in AI is not great. the state of benchmarking in documents is also not great.
2:03 so a couple of recent benchmarks, that I'm not going to pick on these specifically. There's other benchmarks that have quality issues. but one recent benchmark is called ParsBench. And in this benchmark, we took our model from 23rd to third place with 70 steps of reinforcement learning. And this model that's released is now open source. It didn't actually get any better at OCR. It just got better at aligning to what the benchmark expected. So, lots of benchmarks do this. They'll they'll be very specific on formatting things that don't actually matter.
2:33 and in a way that kind of is biased towards whoever authored the benchmark. another one is ExtractBench. and on this one, if we fixed a couple of errors that the the harness in the scoring had, then we shot up from 64 to 93. So, model providers, both of these benchmarks were by Llama Index. I'm not picking specifically on Llama Index here. Others do this as well. but in both these cases, these benchmarks are used heavily for marketing and form people's decisions on which models to use. So, it it pays to be skeptical of of benchmarks in general. And I'm going to talk you through how you can dive deeper into them and and pick things apart. So, this is why OCR is hard to evaluate. So, ParsBench is, as I mentioned previously, the Llama Index benchmark. We released a benchmark called OmniParsBench. I know, confusing naming. it's Omni because it incorporates, documents from many parsing benchmarks. and then all OCR bench has been the standard benchmark by Allen AI that people have used for the last year or so. So, scores change a lot depending on benchmark. And rankings change a lot. So, the two that move the most here are Dots OCR and Nimatron Parse. Nimatron Parse is ranked very highly on ParsBench, very low, on OmniParsBench. And I'll talk a little bit as we go through about why this this happens. But, the fact that different benchmarks measure different things is the root cause here. And it's not always OCR quality. So, this is kind of interesting. So, we actually have some pages from ParsBench in OmniParsBench. We fixed the tests and we changed what they're measuring and that actually changed the rankings quite a bit. Like you'll notice Dots OCR goes up quite a lot, Nematron Pars goes up goes down quite a lot. And one of the reasons for this is ParsBench is actually testing your ability to output values from charts. But two issues with that.
4:25 One, those values actually don't appear on the chart. Like there's nothing to OCR. And two, those values are wrong. So, it's trying to infer how tall the bars are, but it's inference is often wrong in the ground truth. And this penalizes Dots OCR very heavily because Dots OCR is not trained to do charts. And it reduces it makes the model look really terrible. In fact, Dots OCR is one of the strongest open OCR models. Definitely in the top three currently.
4:51 So, this is a deeper look at what I was talking about with the charts. So, the ground truth on ParsBench for this is 310 for the part that's highlighted in blue. But 310 is not correct, right? The actual value is 303. And you have to output it in a specific format because charts ultimately there's no correct way to represent a chart. So, if you output it in a different format, you're penalized. So, this this is an example of a test that is very ambiguous and hard to satisfy unless you're the one creating benchmark. In which case, you know the exact right format and your model is probably the one that is making these types of errors. You're probably labeling with your model.
5:30 So, what makes a metric good? So, a few criteria to look for. The metric has to be valid. So, a correct output has to pass a wrong output has to fail. I know this seems really simple, but a lot of benchmarks do not do this well and I'll walk through some examples soon. Specific. So, if if something fails, you have to understand why and you have to understand how to fix it. So, simpler metrics are better than more complicated metrics. It has to be sourced. So, you have to be able to tell somebody why this answer is correct. And it can't be because I used an LLM that told me the answer is correct to to generate my benchmark ground truth. It has to be something like I generated the page myself, I pulled the value from the PDF in some way, I had a human look at it.
6:14 you need a higher standard for your metrics. And discriminating. So, a metric that everybody fails or everybody passes is not very useful. You need a metric that some people pass and some people fail. So, I'm going to walk through a few different types of metrics and how they look on those axes I just mentioned. So, one is edit distance. So, edit distance is I'm literally going to take a bunch of text, compare it to some more text, and see how much it differs on a character basis. So, have characters been added, have they been removed, have they been changed? And if you look at edit distance, it it's it's kind of a you you think of it as a good metric, right? You think, "Oh, if if something changes from the ground truth, then it's incorrect, and that's actually what I want." But, in fact, imagine that you output the exact correct value, but in HTML instead of markdown. Your edit distance will be very very high because there's a lot of differences between a markdown representation and HTML representation.
7:10 And some models put out markdown, some models put out HTML. But, all of your values are correct. So, edit distance actually is not a good metric in this case because it indicates a high divergence when there isn't one. On the other hand, if going back to the earlier example I gave you, one or two values in a number are wrong, then your edit distance will be really low, but your number will be wrong and in a way that all of your downstream pipelines will be reading and and pulling from. So, edit distance over-penalizes small formatting issues and under-penalizes actual content issues. So, it's actually not a very good metric for benchmarks. And this is OmniDocBench, which is a very popular Chinese benchmark, uses edit distance quite a bit, which is why I'm not a huge fan of that benchmark. you also see a lot of newer benchmarks doing layout and table metrics. So, the problem with layout metrics is everyone does layout slightly differently. Layout is basically how you draw bounding boxes around blocks on the page.
8:12 and if you if your definition of a paragraph is slightly different from somebody else's definition of a paragraph, your layout scores are going to be wildly different. so when we were building OmniParse Bench, we actually figured out a way to to not do this over segmentation. Like we're able to match layout blocks in a fair way. but many benchmarks do not and they basically penalize you if you don't output blocks in the exact same way as the benchmark author, which really doesn't matter that much, right?
8:40 Like as long as you generally know where a word or a piece of text is, it doesn't matter if your blocks look the exact same way as whoever made the benchmark. On the other hand, lots of benchmarks have problems with table metrics. and they just parse tables very badly and score them very badly, which I'll talk about in the future as well. and then finally, using an LLM as a judge is very tempting, but I would caution against it unless it's very narrowly scoped. so one real case, this is from a few months ago or a year ago, we're using GPT-4.1 as a judge. And when you explicitly tell GPT-4.1 to mark not found as incorrect, it marks not found as correct. So there's lots of issues with LLM judgments that you can subtly get tripped up on. Another huge issue is just non-determinism. Your judge changing from run to run is is not going to give you a stable way to eval things. All right. So the best way that we found is Almost OCR style unit test.
9:40 So this is what AllenAI pioneered with their Almost OCR benchmark and paper. And in my opinion, it's quite good. So the idea is you do basically unit test. So inside this page, I want to find this string. Inside this page, I want this string to come before the other string or after the other string. It's it's a very simple, easy to understand way to judge things. And you get a yes no. Like pass or fail. in tables, you kind of say, "Is this cell under this header? Or is this cell near this other cell?" So this unit test style is in my opinion the best way to benchmark. Of course, there are problems with the unit test style, too.
10:16 everything, unfortunately, is ambiguous in some way, and it's your job as a benchmark evaluator to clear up that ambiguity. So, correct is often ambiguous. So, these are tests from the Olmo CR benchmark. On the left side, it's it's a little bit blurry and hard to see, but basically, there are some people's titles written out at the top left, and there's a line break. But, the test actually does not have a line break in it. So, it penalizes anyone who puts a line break in, and most models do. So, this is a test that basically a good model, the better your model is, the more likely you are to fail, which is not great. The other is it the text actually has a lower case P, but the test has an upper case P. So, there's no way to pass this one, either. It's It's case sensitive. So, there's lots of like little ambiguities like this that you need to think about and handle. And the best way to find these is to inspect your tests and your data really closely.
11:09 So, here is a couple more examples of of tests that fail because of small formatting differences. so, this is just an a LaTeX example from math. basically, in LaTeX, equations can be written in many ways, and they can be equivalent, but the tests do exact string matching. They They do a few things to fix that, but they don't always fully fix them. And the the core here is that Olmo CR is basically saturated. So, when new models launch on Olmo CR, they do increasingly elaborate post-processing to make their scores higher. the underlying model quality is not improving. What's changing is they're now post-processing their outputs to get closer and closer to what the benchmark expects, which is not at all what you want to see, right? Like, you want benchmarks to actually drive innovation and improvement in underlying quality. And at some point, Olmo CR stopped measuring quality, and they started measuring how well you can game the benchmark. which happens, right?
12:06 It's not a knock on Allen AI or Olmo CR. This just happens to benchmarks after a while. And you can see it in this diagram, but the raw score is kind of what the model score out of the box, and then they there's increasingly aggressive normalization. So, when we launched TrOCR 2, we scored 85.8 with pretty light normalization. Infinity Parser 2 and Layout OCR 3 have done increasingly large amounts of normalization to boost their score. So, almost the hour is now not a great signal anymore. I've talked a lot about benchmarks and pitfalls. I want to start talking a little bit more concretely about how we can solve them. So, ultimately different pipelines need different metrics. So, the first step is think about what are you actually doing, right? If you're If you're doing rag, some things are going to be very important. Heading, section order, tables.
12:58 If you're doing extraction, getting the numbers exactly correct is going to be important. So, just just thinking about your use case and what's really important can help you do a few things. One, many benchmarks, including our new OmniParser bench, will let you segment by category which model is doing the best, right? Look at it a subset of tests. So, you can see here depending on the subset you look at, different models are best on on OmniParser bench. So, that can be a really useful first cut. Just think about what your use case is and slice the benchmark by your use case. And then you can do checks on metrics to figure out if they're valid or not. So, swap the format. So, if you switch from markdown to HTML, is metric invariant to that, right? Like does the metric fail if you swap? If it does, it's maybe not the greatest metric. Try corrupting a correct output, right? if you change a correct output to all lowercase or all uppercase or change things around with digits, does the metric fail?
13:56 It should, right? You don't want a metric to pass when you're actually incorrect. And then look at the shared failures. So, on a lot of benchmarks, the every single model that's strong will be marked bad on a certain set of tests. And you know those tests are probably wrong because all of the models are agreeing. So, that's that's a really interesting way to diagnose failures and then look for bias. So, one example here is again on ParsBench from LlamaIndex, there is a significant uplift on their visual grounding set for all of their models because the I'm I'm hypothesizing here. I don't know if this is correct or not, but their ground truth was most likely annotated by their own models.
14:39 So, no other model can really match up to to their own layout annotations precisely. So, that bias is really important to look for. And then this is this is again ParsBench. If you break it down by category, you see very different scores for these models, right? For example, Dot OCR and Nimatron, which we talked about earlier, Dot OCR is heavily penalized on charts. But, if you look at the other categories, it's actually stronger or at least very close to Nimatron. So, chart, semantic formatting, and visual grounding are the categories in this benchmark that are particularly weak. semantic formatting is basically how closely do you align with the format LlamaIndex expects. Visual grounding is again that BBox category. So, on content and tables, which are the the categories in the benchmark that actually matter and measure quality, Dot's is ahead. So, that's another way to break benchmarks down.
15:33 this is again I I'm not trying to pick too heavily on LlamaIndex, but I want to give an example of how not to benchmark. and then later I'll give an example of of how to think about benchmarks in a better way. So, there was a company called Pulse, which improved their prompt, right? They they rewrote their prompt and they shot up from 51 to 74 on semantic formatting. Their OCR quality did not change at all, but they went from something like 20th to something like third on the leaderboard. So, this is this is something to watch out for, right? Like, is your benchmark actually measuring the underlying quality? and then the final thing is benchmarks can have scoring bugs, too. benchmarks these days are pretty rushed. and often they have significant scoring bugs in them. So, these are a few bugs that we found a few of the a few are not included here, but these are a few of the bugs that we found in this benchmark. All right. So, you have to think about what the goals of the benchmark are, right? Like, is the benchmark built to be mostly fair?
16:31 does it have careful unit tests? Have they been human reviewed? Is it wildly biased one way or another? So, again, just pulling together all of the things I've talked about, like, think about like what the benchmark is trying to convey, how it was built, where the biases is, and that can help you figure out what the right metrics are. So, here's kind of a a checklist of things to think about when looking at benchmarks. Okay. Now, let's talk about some of the things we did in take these lessons to do your own evals.
17:00 So, a few things. you want to make sure that every test is visibly true on the page, correct transcriptions pass, wrong transcriptions fail, and if tests are incorrect, just drop them. So, this is every test we have we sourced from a something we can verify, right? Like, model panels with human verification, our own authoring, synthetic documents, OMOS bench, which was already human reviewed, or exact human transcriptions on publisher data. So, these are all ways where you can source exact ground truth without having to go ask models to to generate everything themselves. So, these are examples of OmniParse Bench tests. So, here we have the table cell test.
17:45 So, you can see we test basically where the cell is, and then what the headings are that are associated with it. And this came from exact ground truth. It came from publishers' exact ground truth that was put out in XML format. You can also fill in forms, and then check that the exact value you filled in is present. and then for handwriting, you can look for human transcription, either your own or someone else's existing human transcription. So, these are ways to avoid some of the the errors in formatting and and ground truth that other benchmarks have.
18:17 Another thing to do is be very invariant to format. So, HTML or markdown, dollar signs or math tags, you want to make sure you have the same conversion and normalization for everything. And you want to make sure it's fair. Equations compare by the rendering, not the string. Numbers normalize before you compare because some models put out commas, some don't. And then a few other rules as well. So, how do you make sure that you have the same rules for everyone?
18:44 no custom prompts, the same timeout, the same the same headline metrics. and then how do you make sure you can slice the benchmark? So, you can actually look at the things you care about, right? You can slice for Omniparse bench, for example, you can slice by language, you can slice by type of document. So, you can really narrow in on the things you care about. Okay. So, let's go through an example. I've talked about a lot of principles in mostly an abstract way. Let's talk about it in a more concrete way. This is an example of choosing a parser for rag over annual filings. So, in this case, you want to make sure that your numbers are extremely accurate, otherwise you just get the wrong downstream answers in your chatbot or whatever other application. So, the first thing you want to do is think about the documents that you'll be using in production. So, and then think about taking samples of each of those document classes. If, for example, your volume might change over time, right?
19:38 Like you might start with some documents and then maybe you expect demand to grow, you can always resample at certain intervals to to build up a bigger and bigger benchmark set. So, make sure that you evenly sample and cover all of the edge cases that you see in your in your distribution. Then, write questions with answers you can verify. And ideally, ground truth verify either through a human just reading something or through exact PDF ground truth or some synthetically, something along those lines. Generate a bunch of these questions. Basically, document QA. So, if you're you ask question, you have the gold answer that you know is 100% correct. Then, think about where your failure modes are and what your downstream customers really care about.
20:20 So, is it wrong numbers in answers? Like if you're working in a financial use case, that can be really important. Is it tables? Right, for scientific use cases, getting tables exactly aligned is really critical. is it ordering? So, just think about the potential failure modes and where which ones you care about and which ones maybe aren't so important. And then you can build metrics around those using that unit test methodology I talked about earlier. And you want each test to pass if it's correct and fail if it's incorrect. Again, you would think this is simple, but going back to the edit distance comparison, this is not always true for metrics. And you want to make sure to normalize. So, you really don't want your benchmark to be dependent on if a dash is in the output or not if the dash doesn't matter for your downstream application. So, try to normalize anything away that doesn't really matter.
21:09 So, in this example, when we did this, when we designed some tests on some documents, we end up with these results. So, all of the parsers are pretty close on this. So, this is reading, so just looking at the the page and reading directly versus rag with a plain chunker, rag with a table aware chunker, and this is how many questions the model were it correct based off of each parser. So, Data Lab, Mistral, Reducto, they're all pretty close at all of these. So, this can indicate that maybe the documents you're looking at aren't hard enough. So, this is when you might go back and try to sample some of the harder edge cases in your distribution, and that can help you separate. So, when we do this, when we separate, Data Lab comes out ahead here, but this was not my goal. This is just we ended up running the experiment and that's how it shook out, but it it can shake out differently depending on what documents you're using and what your kind of downstream metric is. so, in this case, if you get harder pages, so scanned, handwritten, etc., you end up separating the parsers a little bit more. And this is one quick example of how you can build that unit test that I was talking about. So, in this case, your unit test is your ba- you're checking how many tons of revenue freight were carried, and you can actually sum these up to get the correct answer. So, you can you can verify that the the two the first two values match the final sum. Okay? And then, if you if you run your benchmark, it's really important to go back and look at the failure cases. Why did things fail? Why did your test fail? And this is where you'll get a lot of value because by looking at those tests, you'll learn things. Like, where is your benchmark actually wrong versus where are the parsers wrong? so, these are just some examples of issues we found while looking at failures in this particular example. So, headers split from values, characters coming out in the wrong script, numbers being read incorrectly, and then this is something you can analyze and verify.
23:03 And again, evals are buggy all of the time. you want to make sure that your metrics are not buggy and go back and verify. Human verification is really critical, easily skipped these days, but really really important. This is the whole recipe for how to eval things on your own documents. And we've designed OmniParseBench to be very flexible. So, our scorers and and test cases are open. our core is also open. So, you can use the scoring engine on your own documents. So, if you do have pull a set of documents, you design some tests, you can actually run our scoring, which is pretty well validated, over your documents as well. So, there's some links to how you can get started here.
23:43 and because this is a set of folks who are very well experienced with documents and with evaluating pipelines, I will throw a hiring shout-out here. we are hiring a lot of different roles, engineers of all types, product managers, product marketers. Feel free to connect with me even if you're not interested in working here, feel free to connect with me. I'd love to chat on evals. I'm also very happy to chat if you're interested in a role.
24:10 >> Okay, so I have a question actually that I've been thinking about in the last few weeks. And I love that you brought up this fact that benchmarks have a lot of problems with them. I've also noticed because I've had to look into it recently that this extends beyond just OCR. like a lot of benchmarks more generally you know have very serious issues and when you look at the data like you did here and like when you go and you open the benchmark you run it you look at the actual failures that that benchmark flagged you like just like you showed you'll find a lot of issues with the greater with the environment with the data and you start to wonder wow this benchmark has a lot of noise on one hand but on the other hand like when people are advertising their model or their projects you know you'll see these pages that like show like a table of all these benchmarks and say like hey we are the best like we're the best model we're the best whatever and so people it feels like people are sort of trusting the benchmarks like you know that they see like on social media whatever they'll see like a table of benchmarks but okay this model is better I have started to become like super skeptical after looking at all of the data and so it's interesting like what do you suspect is happening like it feels like okay you have these benchmarks a lot of them not good quality and I know that people are not doing what you did here is like you know look at each benchmark like scrutinize it carefully look at what's failing and like decide for yourself like whether you trust that benchmark but then like it feels like things are nevertheless improving like models are getting better for a mere mortal like let's say you're trying to consume you know these models what do you do you know like when you're looking at these benchmarks in this like marketing like what are you supposed to do >> Yeah it's really really hard I'll talk about a specific case so Mistral launched their OCR 4 model recently and they took a pretty good benchmark the almost zero benchmark but they rewrote the ground truth and they didn't release the ground truth they rewrote and then they reevaluated everyone against the rewritten ground truth, which is which is kind of wild to me, right? Like it just defeats the whole point of benchmarking. I think the problem is there's a lot of incentive to be number one, right? Everybody needs every release to be number one. And there's huge commercial incentives to do that.
26:46 So no one has an incentive to be benchmarking fairly. how does how does an individual decide? I think ultimately the process I talked through, even though it's time-consuming and it takes a lot of work, is the only way to do it. You have to be able to define your own eval set, build your own fair metrics, and and evaluate. Easier said than done, I know. but otherwise there it's really hard to to disambiguate noise. and I mean the unfortunate part is Claude written or vibe coded evals are often also flawed.
27:17 So there's there's just I I hope what will happen eventually is we get some reasonable independent arbiter, one, or two, we just we just define some really strong open-source way to to build benchmarks. but I think the industry right now to me, at least, is like doing a lot of short-term thinking. Folks are trying to get marketing wins, they're putting benchmarks out, they're gaming benchmarks. but it erodes trust in benchmarking in the long run. and I think as an industry we need to figure out how to rebuild that trust.
27:49 >> Yeah, I agree. there's a term that I started using called benchmark gate, which is this like sort of phenomenon of benchmarks not being that then yet everybody is trusting trusting them. What do you suspect is happening? Like do you suspect that people are they have their own like private benchmarks or evals that they're using to make their things or products better, and then at the same time like at a very like separately they are bench-maxing for marketing purposes, and it's like totally like kind of separate? Or like what is happening?
28:27 >> So here's the problem, right? Like if you release a really, really good comprehensive benchmark, you're giving away all of the edge cases and you're giving other people a way to hill climb on those. So, you do want to release a reasonable amount, like enough to keep things fair and honest, but you can't release everything. So, yeah, every lab has tons of private evals, data they just can't release. So, the benchmarks that end up getting released are either just kind of people doing it because they're really passionate about it, like almost CR, or they're just marketing.
28:59 So, the things that are public are the lowest quality benchmarks, unfortunately. and it's just it's kind of like it's kind of like the dark forest problem in in the three-body problem, right? Like the second you put something out in the world, everyone else can see it and and also benefit from the same knowledge. So, I'm not sure how we solve that. >> And things outside OCR, cuz I like, you know, it's obviously you are looking at all this data inside OCR space. Let's just say, you know, general for like models, do you do you personally pay attention to the benchmarks? Like how do you interpret them now? Now that you have like this experience, like what do you what do you just use them and see like which ones you kind of like feel like they're best or like, yeah, do you even pay attention to them at all?
29:44 >> Yeah, outside the OCR space, for me, it's benchmarks do matter, but we run our own internal benchmarks on a bunch of categories across all models outs- outside of OCR as well. So, those are the ones that I really end up caring about, like the released benchmarks I kind of go by. >> Tell me more about that. So, you you run you have like private benchmarks you run across a bunch of models that are not OCR. Why do you do that? That's really interesting.
30:10 >> Yeah, I mean a few reasons. One, we do a lot of document pipelines and there's non-OCR components to those pipelines. Two, we also do a lot of coding using AI internally and it's important to be able to tell which models are most effective for different tasks. >> and how do you go about like this internal coding like eval? What does that look like? >> Yeah, I mean for us, it's very similar to the process I talked about, right?
30:34 Like how do you get actual ground truth? How do you define the metrics, and then how do you run the benchmark across them? we do that for a lot of different tasks. >> Cool. That's really cool to a great to hear. This is a very interesting presentation. I I really loved it. Thanks so much, Vic. >> Thanks for inviting me. Appreciate it. >> Real quick, if you're building an AR product and you're stuck on how to improve, we put together this free resource on evals to help you. It draws from more than a thousand hours of working with companies on building AR products. When you open the guide, it'll look like this. And what you do is you select the stage that you're at. Maybe you're new to evals, you don't even know what evals are. Maybe you do have evals, but you don't really know what to test.
31:12 Or maybe you don't trust your eval scores. Or maybe you think your product is just too hard to eval, or it's too time-consuming or expensive. Whatever the case may be, we have a detailed guide customized for what stage you're at. Go to bit.ly/evals-guide and we'll send it to
Summary
- Current benchmarks in OCR and document processing often favor specific formatting, leading to misleading evaluations.
- Parsing is a critical step in document pipelines, as errors in this stage can propagate downstream, affecting overall quality.
- Many benchmarks, like ParsBench and ExtractBench, have quality issues and can be gamed, resulting in inflated scores that do not reflect true model performance.
- Good metrics should be valid, specific, sourced, and discriminating to effectively evaluate model quality.
- The use of edit distance as a metric can be misleading, as it may penalize formatting differences rather than content accuracy.
- There is a need for independent, fair benchmarks to rebuild trust in the evaluation process within the industry.
- Organizations often have private benchmarks that are more comprehensive than public ones, leading to a disparity in perceived model performance.
- The development of unit test-style benchmarks is suggested as a more effective way to assess model accuracy and reliability.
Questions Answered
How do you evaluate quality in OCR and document processing?
Vic, the founder of Data Lab, discusses the importance of parsing in document pipelines and how it affects downstream quality. He highlights common parsing failures and their implications for various workflows.
Why is edit distance not a reliable metric for benchmarking?
Vic explains that while edit distance measures character differences, it can misrepresent quality by penalizing formatting changes rather than content accuracy. This can lead to misleading evaluations of model performance.
How should benchmarks be tailored to different document processing use cases?
Vic emphasizes that different document processing pipelines require specific metrics based on their unique needs. He suggests segmenting benchmarks by use case to better evaluate model performance.
What steps should be taken to choose the right parser for document processing?
Vic outlines a process for selecting a parser, including sampling document types, generating verifiable questions, and understanding failure modes. He stresses the importance of accuracy in critical use cases.
How should individuals approach benchmarks and model evaluations?
Vic discusses the skepticism surrounding benchmarks due to potential manipulation and the commercial pressures to rank highly. He advises a careful, analytical approach to evaluating benchmarks.