0:05 Okay, let's get started. So, today is the second day on data. Um, last time we talked about how data doesn't really just fall from the sky. You actually have to think about where it comes from. Um, so in general, the internet consists of live services. um these um the data on those services have to be either dumped or crawled and then there's additional step where you have to process the data and we also talked about various um social societal considerations um terms of service copyright you have to either get a license or appeal to fair use. So there's a lot of complexity that goes into data. Um so today we're going to look more at the data pipeline from transforming the data, filtering the data, dduplication and mixing. Um and then we're going to top it off by talking a bit about um post-training data um in particular how folks are using synthetic data these days. So the first part is going to be mostly about pre-training. That's what you should have in mind.

1:17 So we talked a little bit about data transformation already, but just as a kind of reminder. So raw data doesn't come as text. Even as if you've you've scraped something. If you ever look at inside Chromic Crawl, it's it's not text. It's either HTML. Sometimes it could be PDFs or directories in the case of uh GitHub. So most of the tension on transforming data is dealing with HTML because most of the web is in HTML and for this processing there's a lot of this is fairly heristic um there's removing of boilerplate um like navigation and ads um and extracting content um which is like the main part of the page and there's some you know subtleties around what constitutes content usually um you get rid of the footers and headers um and maybe menus and those things and you try try to extract the content but you could imagine in cases where some of the the navigation elements might be might be helpful to learn what web pages look like. So what is content and what is not content is not always clear and then what do you do about images and tables are on in web pages. Um so inherently this is a lossy process because you need to linearize HTML which at least is either hierarchical or visual if you think about the rendered uh output and do a sequence of tokens. In particular tables are um a bit tricky to deal with. Simple tables you can render um using markdown but if you have nested tables then that becomes quite uh challenging. you have to give ups at some point or approximate at some point. Um so typically HTML to text processing is is rule-based and the reason for this is that rulebased um processors are very fast um and and also um they you're not trying to do too much here. You don't need too much intelligence. So rule-based generally works. Um now I think there could be a case for model based um interventions at this point. They have to be very fast and they have to you know do something um more more intelligent. But if you ever look at data you'll notice there are um imperfections in the data just because any rule-based uh processing is going to be um is have some failure rate. As we showed last time, the accuracy uh does matter depending on which tool you choose. Um if you use Brazil parse or Traflera, then it's you know I guess Brazilia parse on these extended DCLM you know eval um works better um than the others.

4:19 Um I'll talk also briefly about PDFs. So there this uh hugging face work um which released a data set called find PDFs. Um so PDF files if you have open them up not in um PDF reader um looks looks kind of like this. Um and this needs to get kind of rendered. Um so PDFs can be found on just like web pages on the the internets and common crawl u does um have you know some PDFs. Um, generally common call focus on text, but sometimes if you're just given a URL, you might not even know before you fetch it whether it's a PDF or or not if it doesn't have the extension.

5:02 Um, there's a lot if you read this blog post there's a lot of details which I'll spare you of. Um, for example, one thing is that uh many of the PDFs that are in comic crawl are truncated because PDFs are big. So then that re demands that rec crawling is necessary and then there's a question once you have the PDF file how do you actually convert it to text um and you know some PDFs are might be just scans as well right so they're essentially images so there's a bunch of tools that um this paper uh tried out um but mostly this kind of involves um running OCR um using a VLM and this obviously can be much more expensive than what we were doing before with text. Fortunately PE or unfortunately uh the PDFs are a very small fraction of the whole internet but um they are very valuable um because generally if you bother to make a PDF that means you probably have something um interesting to say as opposed to a web page. So the uh quality of a PDF is average PDF is generally higher than uh for um for a HTML file.

6:14 there's a lot of clean up and filtering and PDFs more so even then than web pages a lot of layout information is missing um because uh in HTML you have various tags like H1 and P that gives you some semantic information PDFs are by design all about layout so you don't necessarily um they preserve the the kind of the semantic structure okay so there's a bunch of transformation information that happens at this point. You have uh text um um and but you're not done. You're far from done. So the next step is filtering.

6:56 So um I'm going to talk somewhat abstractly about what filtering is. Here's a sort of the building block. So suppose you have some target data that you uh want to get. Um this is usually a small amount of high quality data and lots of raw data. This is the the the fresh shipment of the uh the tokens that you do transform from the the previous step and the goal is to find uh a subset of this raw data that is similar to target. So this is the general skeleton for filtering and may almost all kind of filtering falls into this uh this uh schema.

7:39 So filtering, you could there's many reasons you might want to filter. Uh if you're training um an English language model or a German language model, you might want to identify the language and filter things that don't match that that language. Um the main reason for filtering is is quality filtering. You want to find things that are high quality as opposed to low quality. You don't want just uh you know spam. You want um encyclopedia information. Um and and then another application is uh toxicity filtering. Of course, the internet has plenty of nasty content and maybe you don't want to train your language model on that.

8:19 Okay, so for a filtering algorithm, you want that you want some generalization from the target data, right? Because you already have the target data. You don't want to just only get the target data but um you also want it to be extremely fast because you have to run it on the whole internet. So this could be uh 100 trillion tokens worth of worth of data. So generally filtering is you end up with a very small fraction single digits fraction of your entire um data.

8:54 Okay. So, so remember the general framework given the target uh and raw you're trying to find a subset. Okay. So the general scheme is that you estimate some sort of model based on R&T and derive a scoring function and then you keep the examples in R based on their score. So typically there are two uh types of classifiers. One are generative models. Uh so this is basically um you have your target data you can just um you know estimate a model of that data and remember this has to be cheap so probably you're not training a big language model generally um klm basically says I'm going to train a five gram uh model um a more common thing um which I think we're mostly seeing these days is just train a classifier and The classifier says, I'm going to predict a positive label for examples that are in t my target and negative labels for things that are in raw but not in t. So you basically it looks like t are your positive examples. Um some random subset of r your negative examples. You maybe balance it and then you train a classifier.

10:16 And the tool that people generally use is fast text because it's fast uh and it's generally it's just a linear classifier bag of words. And then once you have this model now for every new document you can score it and you set some appropriate threshold depending on your quality bar and you keep the examples sometimes stochcastically sometimes not. Okay. So, so this is a very much a model based filtering approach. Remember from the last lecture, there are many data sets um from you know generally a few years ago that did not use modelbased filtering because they wanted to not bias things too much. These days I think uh basically everyone does some amount of modelbased filtering because um uh unless you are compute plentiful which in which case you probably don't need to do as much filtering. You can just train on everything. Most people are compute uh poor um you have to be very smart about how you're filtering. Otherwise you're just wasting flops on kind of lowquality content.

11:27 Okay, so let me talk through some instantiations of of filtering. So remember I mentioned language identification. Um so the goal is just to find given a piece of text, detect whether it's of a particular language. Um so Meta has trained these set of fast text language identification models which often you can just use off the shelf. So it supports 176 different languages. It's been trained on uh a bunch of multilingual sites. So, Wikipedia, remember, has a lot of languages. There's also translation sites um and sites for different languages. So, um a language ID is is generally a fairly easy problem compared to many other tasks that we're dealing with. So, a simple classifier, if you look at a few words, you can tell that it's Spanish or or or Japanese. Now this does you know there are subtleties because of code switching and text and there's some dialects. So it's I wouldn't say it's like an absolutely solved problem but you know this is not really the bottleneck for training a good language model.

12:34 Um okay and then you can just so once you have your classifier you just choose some threshold. Um this is generally fairly heristic. So um another thing you can do with filtering is that suppose you're looking for data of a certain type. So let's say you want to get really good at math. So you go want to go and find a bunch of math data. So uh open math text is this um paper from 2023 where the goal is to create a large corpus of math text. Um this pipeline is consists of a few steps. Um so it's not training a single classifier. So first you use some rules to filter does it contain latte commands. Um you they also use kenlm. So this is a generative approach um and trained it on the proof pile which is a known data set of math and keep it if the perplexity is uh you know below some threshold. Um and then you also train a fastax classifier to predict whether it's mathematical writing um or not. And there's, you know, two different thresholds. If it has latte in it, then it's a higher bar. If it's uh sorry, if it's it's a lower bar, but if it's uh doesn't have latte in it, it's a higher bar. Um and then as a result, they got you know 15 billion uh tokens which they use to train models. And as a paper they show that um this more targeted security data collection results in models that are better than math at math than models that were trained on 20 times much data which were not filtered in this way.

14:12 Okay. So the the quality filtering again think about it as a tool. Um you can define quality however you want. There's no universal notion of quality. If you want define quality to be math then you can go and get math and you get better at it. Um so GPT3 I think I mentioned this last time but just to put it in this framework. So positive examples are Wikipedia um web text. So these are pages that um link out from high uh you know star Reddit posts and some books and negatives are generally sampled from the the web. Um they train a linear classifier and keep documents if the linear classifier is scored um highly enough. Um there's the first llama paper the positives were pages referenced by Wikipedia not Wikipedia articles themselves um and the it's just kind of the same idea. So uh 51 from Microsoft um is interestingly had um you know also falls into this framework. So um they um started with the raw data is not already a Python subset of the stack remember which is this uh all you know code and um they they're uh the the prompt they define a prompt which is determine educational value and then they use GPD4 to classify uh a subset of R 100k subset of the R with this prompt.

15:47 and see if it's uh and the ones that are positive are kept and called the target. Right? So the target here is actually output of an expensive um you know classifier and then you train a cheaper classifier in their case a random forest but you could have probably used fast text as as well. Um and uh and then you select data that is um classified positive by this this classifier. And they also show that using this uh data set you do uh much better than if you're just using um the raw data. So this performance goes up in fewer steps to a higher value.

16:32 Okay. So toxicity kind of works the same way. There's this data set Jigsaw toxic uh comments um where it came from this project where the goal is to help people have better discussions online. So here the data is the Wikipedia talk pages um which can I guess for controversial sites get quite uh heated and this has been annotated with whether there are any any toxic comments on there and you can similarly define positive and negative examples and you can train a classifier on that.

17:08 Okay. So, so now I think you have the tools or some examples and inspiration. You can identify um a particular type of data that you want. You build a classifier for it and then you can go filter common crawl for that. I want to talk about one uh subtlety here u which is that you know the notion of what you want for data is actually depends on what model you want to train.

17:38 So in particular, it depends on the number of tokens uh you're training on. And so there's no optimal threshold. So before when I looked at the the classifier gives you a score um there's you can't say 0.9 and say that's the best because it depends on what you want to do. Intuitively if you are going to train for a longer period of time then you can tolerate lower quality data. If you're training for shorter then uh you you want higher quality data in general. Okay. And of course you know if if you could wave of magic one if you're training for longer you want more high quality data but that's not an option that you're given the data pool is what it is.

18:24 So here's an a kind of a a preliminary experiment um uh that Michael Ryan did and uh so basically this plot shows for a 157 million parameter model um we're taking a very small uh pool so uh 100 uh you know works which is a tiny fraction of common crawl and um we're look training and training um for more uh over time. So let's take a look at the blue curve. So the blue curve is DCLM. And so the the loss starts here and then it comes down and each of these lines is when you uh epoch over the data. So this is one epoch and then the next blue is the second epoch and so on and so forth. So there's not that much data. So eventually you have to repeat your data and um you know the loss continues going down um because the second time you look at the data you're still learning things and at some point you start to to overfit.

19:27 Okay. Um whereas if you look at resilient pars this is basically no filtering. Um so here it's it's much worse in the beginning. Um but as you continue and continue um at some point you also epoch um it starts to go down you know slower. So you kind of see this trend where high quality data is better in this regime where you're not epoing but once you get to lots and lots of tokens um high quality data is no longer that uh great. And of course you with high quality data you you wouldn't even get you wouldn't want to go into this regime because you're overfitting.

20:06 You'll probably stop here. But even this at this point this is worse than if you had trained for longer using low quality data. >> Yeah. >> Just a question about like the um computing the metrics. This is actually not like totally about graph but like when you're like each of those dots correspond with one training one training run, right? >> Yes. um do you ever need to do the confidence interval when you're doing these kind of pre-training experiments or like is it kind of enough just to do it one time?

20:35 >> So the question is each of these points is a single training run and uh should you do it multiple times and get confidence intervals? Um ideally that would be good practice. Um often you'll see in these papers that um these are kind of scarce because each training run is fairly expensive. Um so you but but in reality when we have done these experiments it's generally tends to be stable I would say at least for pre-training.

21:07 >> Yeah. >> Yeah. So you mentioned uh like trading for longer and high quality data as that's not like a combination that we're considering but like like if if we are trading with uh for longer on this high quality data um would it have diminishing returns compared to like training for longer with quality data? >> Yeah. So the question is if you were able to get higher quality data and you train for longer would it still have diminishing returns? Um, every data set is going to have the machine returns eventually. Uh, that's finite. Um, but it would just it would probably be here.

21:46 It would just like be down here and just just keep on going down. Okay, great. Let's let's move on. So, summary, filtering is is pretty critical for building a good model, especially when you're computed like most of us. Um technically if you have infinite compute you don't need to filter and you can train on everything and it'll be a giant model but um realistically everyone has to filter.

22:16 Um so and the recipe here is figure out what good data looks like and then you can train a classifier and that will extrapolate to the rest of your model. and what the good data looks like. You can either find it by saying, "Aha, there's this data set out there that I really like and I just want more of it." Or you can craft a prompt um to a language model and then use that to construct um to do a sort of a preliminary filter of a large pool and then use that uh good quality data to train a smaller classifier and extrapolate to everyone else.

22:55 Okay, next I'm going to talk about dduplication. So at this point we have uh filtered our data set so we only have what we deem to be high quality data. Um but you know often data still has duplicates in it and um there's you know two types of duplicates. There's exact duplicates. So this happens when um for example if you look at these um mirror sites the whole point of a mirror is that it's a duplicate and you know sometimes the web crawler isn't smart enough to know that this mirror is exactly this mirror so it just crawls um many uh sites and you'll get this exact same content.

23:40 Um also when you fork a repo um that is also um a a duplicate. um even if you make changes probably you're making changes to a few files so 99% of that repo might be uh the same. So duplicate is is um abundant um and yeah sometimes there's actually near duplicates which are not mirrors or derivative but they just happen to be uh same text differing by a few tokens. Um, usually probably this came from a kind of a uh it could be copying or came from a different uh common source. So here's some examples of near uh you know duplicates. So terms of service and licenses. Um so I guess the MIT license probably shows up um you know on a lot of places. Um and so many cases it is a exact um copy but only of that license unless someone made a typo when they copied it. Um but the rest of the page um that the license might be part of is might be different. And we talked about how a lot of websites have the same um headers and footers. Those are also you know duplicates. Um there's also cases where for whatever reason um you know there's uh this is from LM1B so the one billion or benchmark there's like these articles where you just have you know typographic uh differences like uh there's one version with a comma and one version without a comma I don't really know why but uh that's happens and sometimes you see these templates where there this is like some you know probably low quality content where it's like a it's a essentially looks like an ad of some sort and someone just uh you know templatize replace Canada with USA.

25:34 Um so if you train like on this data but you are just training on different variations of this with different entities this is going to be wasting your GPUs. There's a more extreme cases. So this is why it's good to look at your data. So this audit of the C4 data set found this product description 61,000 times in the data set. And if you trace back, this is really bizarre. I don't know why this is like some um description of this gas mask. And this just showed up in this data set um in common crawl 61,000 times. So um yeah, the web is weird. Um so why do duplicate? So the first clear uh idea is clear because you want to train more efficiently. Duplication reduces your data set size without really losing information because you're just removing duplicates. And it also has this benefit of avoiding um memorization which this uh paper you know talks about. Um for example, if you have some um copyright content that's duplicated a lot then if you train on it then you memorize it and also privacy concerns. uh but mostly I think uh it's uh just to uh make sure that you're not uh wasting flops related to due duplication that's also decontamination which is arguably even more important to and it's the same sort of uh deal is that you want to make sure that your uh test set is not in your training set.

27:14 Okay. So so how do we ddup? So here's the design space to think about. So first of all, what are the items that you're dubbing? Do you do at the sentence level, the paragraph level, or the document level? Um, how do you determine a match? Is it an exact match? Is it existence of a common sub item? Is it the fraction of common sub items u for near dduplication? And then once you found a duplicate between two pages, what do you do? Do you remove all the instances or do you remove all but one?

27:48 Okay, so um and the key algorithmic challenge is that dduplication is fundamentally about comparing items to other items. Um and normally if you're doing filtering this is about an individual item. This item is it good or not? And this can be paralyzed. It can be you know it's sort of linear time which is good. Um and even in linear time we're trying to make it fast by having rule-based or very small models. But due duplication clearly you can't do the n square thing where you compare everything to everything. You need linear times uh algorithms to scale especially at this web scale.

28:29 So um typically the dduplication literature um uses hash functions to get around this. Um so we'll develop some of these uh ideas. These are quite nice ideas um from um thanks to um you know the algorithms uh community. So I think everyone knows what a hash function is. It takes of some sort of value like a string and maps it into something like a string or integer. Um and a hash value is much smaller than item. Hash collisions are when two distinct items map to the same age. Okay. So uh when you look at hash functions there are really fancy hash functions which are cryptographic uh nature. These are collision resistance used in cryp um uh cryptography and um and bitcoin and things like that. And then there's these like uh faster ones which are used for hash tables where hash collisions aren't the end of the world. And so we'll be using um these. Okay. So take a string and map it to some value. That's what um hash functions do.

29:38 Okay. So exact duplication is is conceptually very simple. Um you you take a string um you see if there's an exact match and you remove all but one. Okay. So here you have a bunch of um elements and you you hash them and you ddup. Okay. So this is um and exact duplication is is is very nice. It's very clear what's happening, but this isn't really good enough for the messy web data because often you have these near duplicates. Um and one note is that there's many ways to have um you know written this. Um this is written by in a sort of like um a bit of a map reduce uh style way which makes it more easy to easily paralyzable and scale. Um so C4 um this uh this paper from the the T5 paper um which process common crawl did exact duplication. the items they operate on were three sentence spans. Um, and you uh do um exact match and they remove all but one.

30:53 So, one, if you're kind of paying attention, you realize that there's something kind of a bit strange about this because you're looking at three sentence spans. And if you find two documents with three sentence span, you remove all but one. That means you're just going to rip out three sentences from that document, which is a little bit strange because it breaks the coherence, but you know, this is what they did. Um okay so that's exact dup dd duplication. So how do we do near uh dduplication? So first of all we have to define what approximate uh match means here. Okay. So to do that um we're going to define this uh called jakard similarity which is a fairly standard notion. And jakard of two sets is basically the size of the intersection over the size of a union. Okay. So given these two sets uh 1 2 3 4 and 1 2 3 5.

31:55 Uh when you compute jakard you take the intersection that's 1 2 3 you take the union that's 1 2 3 4 5 and you divide and the jakard is 0.6. Okay. So the chakard is a number between 0 and one. Zero means that they're disjoint. one means that they're identical. Okay, so a fairly natural notion and um we're going to say that two documents are near duplicates if their jakard similarity is above some threshold let's say 0.99.

32:28 So now the question is how do you find near duplicates in linear time? Okay, so fortunately this is a solved uh you know algorithms question and the answer is to use um minash. So um well okay so the first step is to use minash that's not the final answer. So minash is a random hash function so that the probability of a hash collision is exactly uh the jakard of am and b. So this is a very kind of nice property because um hash functions hashing is good for making things linear time. Jakard is the metric that we want and we're sort of connecting these two uh right now or in expectation. So what's interesting is that normally you want hash functions to define hash functions so that everything is um hashing to different distinct elements are hashing to different things. You don't want collisions but here you actually want collisions u not arbitrary collisions but you want to control the collisions in a certain way to align with a similarity. So similar things you want to collide more than the disperate things.

33:44 Okay. So um so the minash is essentially you um actually let's see maybe I will okay so here's the the min hash. It's fairly simple. You take the set and you hash every element in that set and you take the minimum element. Okay. So um this mean might if you're seeing this for the first time it may seem a little bit strange. Why are you taking the min um you can take a max too doesn't really matter it's just a way to break um ties.

34:18 So here's the the picture that uh you should have in your mind. So okay so if you have um um a remember a contains one two three four and b contains 1 2 3 5. So this is a characteristic matrix representation of these two sets, right? And so what the random hash uh function is is doing is that it induces a permutation over the items. Okay, so it might be 4 3152 or something else. And then you look at which item is first according to this permutation in A and which item is first in B.

35:00 So each item has the same probability of being first. Okay. So if you have if it the random hash function puts one first then um first in A will be equal to first in B. And if it's two first then it's the same. If it's three first it's also the same. But if it the random permutation puts four first then the first element in A is going to be different from the first element in B. And same with five.

35:39 Okay. So look at this representation. Basically the random hash function says elect one of these rows to be you know first and then when I the min is just telling me the min um the the the min equal of a equal the min of b is basically telling me whether that row has um is the same. Okay, so that is the I guess the proof of um why minash why this property holds which is that you have a hash function in expectation over your random choice of hash functions that um the probability of collision is the similarity metric you want okay so let's just check this in code so um I'm going to generate 100 different hash uh you know functions S each hash function is given by a seed um and I just check whether the minash is equal to minash and the estimated jakard. So um I'm going to just look at the number of the fraction of matches and you get 0.6.

36:56 Okay. And the key thing with a minash is that um I don't have to do the the n squared thing. I can compute the minash of A and the minash of B and minash of a different set and I just look for collisions. Okay, any questions so far about Minhash.

37:29 So now we can hash our items, but we're not done yet because a collision doesn't tell us that jakard is above some threshold, which is what we want. We want to find a and b such that jakard is greater than 0.99. All we've done is said, okay, well, if we got a collision, the probability of that collision of two things colliding is the jacard. But that's not really that useful by itself. it's sarcastic and I can't really get anything reliable here.

38:00 So um the next thing idea is this thing called locality sensitive hashing which essentially solves this uh problem. So this is a very classic idea in theoretical computer science and it's quite nice. Um so the idea here is that we have a and b colliding with probability equal to jakard. Okay, so it is true that more similar items will collide more often, but it's very stochastic, right? The variance is, you know, quite large here.

38:36 So our goal is to have A and B collide if the Jakard is greater than some threshold. So we in some in some sense have to sharpen these these probabilities somehow like the probability can't just be literally equal to the jakard. So the solution is to use more hash functions u and these hash functions are going to be independent. So um this part is you know gets a bit technical but um we'll walk through it.

39:08 So you break the n hash functions into u b bands of r hash functions. So if you have 12 hash functions you have three bands. Each band has our uh four hash functions. Okay. So there's a band here H1 through H4. Here's a second band H5 through H8. Third band H9 through H12. Okay. And then so what we're going to try to do so so each hash function gives us either um collision or not collision.

39:42 Okay. Um and so every hash function there's a probability of uh colliding. So the key here is that we want to say um compute the probability that A and B collide. We say that A and B collide if for some band all of its hash functions return the same value. Okay. So um if H5 and H6 and H7 and H8 all return um the kind of the same value. So sorry if h a of a equals h5 of uh sorry h5 of a equals h5 of b and h6 of a equals h uh six of b and so on that means this band is triggered and then I would say they collide or if this band if all of these say um coll uh agree then I say it's collision or these two but you don't have to have all the hash functions uh um you know uh return the same value.

40:49 Okay, so there's sort of this and or structure here that is um is is doing the the lifting here and we'll see why this uh this works. So um so now let's say you have a particular jakard of A and B um what is the probability that A and B collide according to this uh definition? Okay. So we can calculate this. Um so let's say that uh this jakard similarity is 0.8.

41:26 Okay. And we have five uh you know bands and each band has 10 hash functions. So then we can say what is the probability of a fixed band matching. So that's 0.8 eight to the r because each band has r hash functions. So the probability that they all have to match is uh 0.8 to r. Okay. So the probability of a band of a fixed band matching is is generally quite low. It's exponential in r.

42:04 Okay. And then now the probability of collision is the probability that some band matches and that's going to be the probability of one minus the probability of so this is a probability of um a band not matching and this if you raise it to the B there's B bands this is a probability that all of them don't match and then one minus that is the probability that some match okay so the probability of collision is you know four and you expect it to be higher because you're given you know beach tries to get a match.

42:44 Okay. So if you plot this um this is what it looks like. Okay. So you have um on the x- axis the similarity here um and we were looking at 0.8 eight um and and you look at uh the probability of a a collision here. So we would expect that if it's similarly zero then it should be zero. If it's one it's one. But notice that this is kind of a interesting sshaped which is is nice because this is what we wanted to sharpen. We wanted to basically say if the similarity is below some threshold then we want the probability to match to be as low as possible. And if the similar is above some threshold, we want it to be as high as possible. So we're trying to get this to be like a phase transition.

43:39 Okay. And if you you know plot this this function as a function of similarity, that's um what you get. Okay. So um Oops. Okay. So, so let's just look at some examples. So, let's uh concretely we have um similarly 0.7 to 0.98. Okay.

44:12 Um and we're going to look at how u b= 10 r equ= 10. So we um look at the collision probability according to our definition and we see that um this gives us a range of um you know 0.25 to to one. Okay. So if you were to set a threshold you know this is this is not you know bad. uh we can like set it here and you know these with um you know some probability will uh you know there's still false positives because even if these are below a threshold let's say 0.9 these might still collide but we can filter them out um if we if we want but um the ones above are mostly kept.

45:04 So what happens when you move uh so increasing r so remember r is a number of hash functions within a bucket. So when you increase r um the the threshold sharpens and moves the curve to the right everything becomes harder to match right because you have more hash functions inside a bucket you have that it sharpens that exponent. So the probabilities you see here are kind of sharper here. And now everything moves to the to the right. Um, and now it's fairly unlikely that if something has loaded your card, you're going to to match it.

45:43 So before it was like 0.25 and now it's, you know, 08. Okay. So we're sharpening the probabilities and then increasing B has the effect of moving the curve to the left. It makes it easier to match. B is the number of bands and some band has to match. If you have more bands then there's more chances of matching. So if you look at this then um before and after. So now we're making before this 0.9 was only 72 and now it's 0.92.

46:16 And of course this this also increases but you know not by that much. Okay. So and you can drive these uh phase transition to be as you know as sharp as you want by increasing B and R. But of course, if you can increase BNR too much, then it can be more expensive. But okay, so let's look at a more real world setting. So in this paper on dduplication, the they had um you know B 20 bands and R is 450. So to give you an idea of the magnitude of these um these numbers um and In general this phase transition happens at the at a threshold which is one over b raised to the power of one over r. So if you want to uh find a you know if you want to filter based on jakard greater than 0.9 then you basically have to set bnr such that this is 0.9 and then by changing BNR you can make that phase transition uh sharper.

47:29 Okay. So remember probability that a fixed band matches is one over b. If you have at this threshold then the probability that a and b collide is um you know uh approximately uh it's a it's a constant. So the probability of collision is 64. Okay. And this this makes sense that basically if you have a phase transition um the sort of center of that phase transition is like point you know 64 and everything below it should go to zero as BNR increases and um everything above it should go to one as BNR increases.

48:10 Okay any uh any questions about LSH? So this this method is called minash LSH um because LSH really works for any hash functions. For um the duplication of language model processing, we're using the minash which approximates your card. And so um that uh that basically is the one you should use here. Okay. questions about um DDUP.

48:55 So one note about DDUP is that um there's often as we'll see data sets that are coming in and sometimes the um duplication will happen within a data set but you actually have to do duplication across your entire data set because often data sets can be redundant. So um you know sometimes that's not uh done but it should be. Okay, let's go on to the next topic which is data mixing. Okay, so so far we've transformed our data from really uh you know raw HTML or PDFs into text.

49:34 We filter for high quality. we dduped. So we have a smaller set of high quality um documents and um and this generally happens um for a given you know data source but language models are trained on multiple data sources. So um in Marin um currently uh this this website tracks the different uh data sources that um the next model will be trained on and you can see that um there's a bunch of things there's some neatron there's fine PDFs which we talked about um there's some uh you know institutional books um and um some code and all these things.

50:24 Okay, so you have all these different you know sources um and the question is how do you combine these? So if you look at an older paper the pile um this is the set of sources that they had at that time um and they essentially assign a particular weight to each component. So basically you have a distribution of resources.

50:56 Okay. And so where does that weight come from? So more you know just cut and dry let's say you have three sources you want to find a data mixture and a data mixture is just a distribution over your sources. Okay. So if you're just thinking about this problem uh from first principles um well not from first principles but like just from what would you what you might do um vibes which is I guess opposite of first principles is you just manually set it based on some intuition which is more often than you might think what people do right so I think this is this is definitely fairly vibes based um And even more recent papers um you know you just look at it maybe you use some method and then you just like tweak things. Um you can also um do uniform sampling which is you just put a uniform distribution over your sources um and you sample every um you know chunk you you get you just sample a source. Um you can also do uh proportional mixing which is that you sample proportional to the number of tokens in a source. So if you have a data set with more you know tokens then you put higher weight. So this is also you know generally a rational thing to do but you might worry that the if you have a a huge lowquality data set that's going to eat up a lot of your tokens. So this doesn't seem quite optimal either.

52:35 So intuitively you should upweight your higher quality sources. Okay. But there's two things that um are important to keep in mind. One is that you do want to ensure some diversity, right? So often sources are incomparable for literature, code, and papers. um you can't really say that uh you know this paper is higher quality than this code because they're just incomparable you know objects maybe and if you want your language model to do well you don't want to just put all your um uh your mass on just papers the second thing which we'll talk a bit more about is uh each source is actually finite so if you put too much weight on a small source then you need to you essentially run out of that source and you're you need to just epoch over it.

53:31 Meaning that you keep on training on the same literally the same tokens and this is going to be bad for reasons we'll uh see. Okay, so let's let's uh try to unpack this a little bit more concretely this last point. So imagine you had just have two sources a lowquality source you have 10 trillion tokens and you have a high quality source that has 10 billion tokens. Generally high quality sources are smaller. Okay. So if you just do a naive data mixture, let's just do uniform. So half on high, half on low.

54:06 And I'm going to train for one trillion tokens. Um so when I say train for one trillion tokens, that doesn't mean unique tokens. I just means uh train for that many steps divided by batch size essentially or times batch size. Um okay. So then you look at let's look at this which is the number of times a particular element in the lowquality data set was trained on. So that's the number of epochs uh a particular data uh point was trained on.

54:39 So p low times train tokens is the number of uh times I'm want to request uh tokens from um this lowquality source and divided by the total number of lowquality sources. So um this is less than one. So basically that means um 5% of the tokens in this lowquality data set I'm even going to touch and train on once. Um, and then if you look at high quality, you'll see that uh the number of epochs is 50, right? Because I only have 10 billion tokens here. And if I'm half half, that means 500 billion tokens. Um, I'm I need to train up 500 billion tokens of high quality data. But I don't have 500 billion tokens of data.

55:32 So I have to repeat each data point 50 times. Okay, so this is actually really important and you know some big model trains range uh runs have kind of messed this up is that you can't just like naively look at the the data sets and define a distribution and then you sample because um if you just look at the quality if you look at the data you don't you don't you're sort of missing how many data points are there. Okay, does this make sense? Maybe pause for questions. This is a kind of important point.

56:14 Okay. So, yeah. >> Um, I guess like does it like why do we need in the first place? like doesn't it achieve comparable performance level way I don't know fewer you know >> so the question is why do you need 50 epochs and I guess the point is that you don't need 50 epox um worst case it's wasting compute sorry best case is wasting compute and worst case you're overfitting um but 50 the reason it's 50 is that if you just go in and you define this data mixture then you're going to be end up doing 50 epochs without realizing unless you're paying close attention So this is basically the lesson is like look at how many epochs you're actually doing on your data.

57:10 >> Yeah. >> How does this how do the mixtures get expressed during training? Like so when you switch every step do you train like steps of 10 saying oh I want to do 10 code 10 to see because I feel like it was like is there also any intuition between >> what decisions are made there. >> Yeah good question. So the question is like how when you train how do the mixtures get uh kind of realized and so uh in general um you you sample um so you have you want to fill a batch right?

57:42 So you can sample a essentially for each element um which uh mixture component it comes from and you fill up a batch with um you know sequences from that mixture component. Usually you every sequence comes from one mixture component. You're not sampling like per token. >> I guess so each batch should be mixed. So I guess the unexture assumptions are at the batch level and then you just train for that. It's not like oh there's one step is one step is >> yeah in general you want to to reduce variance you want to train from multiple um you know a batch should have some mixture in it. Yeah.

58:25 Okay. So so this idea I mean this problem has been noticed uh you know quite a long time ago. Um and so in this paper they introduced a unimax and I in there that that case it was uh they were training in multilingual models. It was like very clear that some languages uh were very low resource. So um you had to do something about this. So in this paper they noticed that before um some works would uh essentially um take the proportional mixing and raise it to some power um to kind of flatten it out the distribution. Um but the idea in this paper is that um let's be a bit more kind of explicit about that. Let's sample the sources uniformly but with a hard cap on the number of epochs. Okay, so this is the key idea here is that you say I'm only going to take 20 epochs over a particular, you know, data source. So if you've done that, then you know, too bad. You go you you don't get any more um tokens. You move on to the next thing. So this is sort of like a a safety net in in some sense. So in particular if you look at the number the probability assigned to a source times the number of training tokens that should be less or equal to the cap.

59:51 Okay. And there's a simple procedure for actually just determine the mixture uh subject to this uh constraint. Um okay so let's switch gears a little bit and talk about how we define the mixture in the first place. Right? Because if you have 50 sources, there's 50 numbers you have to fill. And we noticed that proportional mixing isn't really enough. Maybe you can try to get estimate some quality metric and then you can sample proportional to that. Um but that is also heristic.

60:26 So the the most principled uh methods and most kind of easiest to understand like what's happening is these regression based mixing methods. So ragmix um mix and other papers like this and the idea here is is kind is relatively simple. So first you let's say you're trying to train a large scale model. So you go to a small scale um let's say one you know I guess let's say 300 million parameters um and you you try different data mixtures. So let's say you have three uh components. You try different mixtures and then you train these small models. So you train a swarm of small models. Each model gives you a loss on some target metric. It could be some downstream evals, it could be perplexity, whatever you want. Okay.

61:20 So you use these data points to basically fit a regression where the regression model maps these um inputs which is the the data mixture weights to the loss. Okay. So now you have basically a very cheap model that tells you if I were to train on this mixture what loss would I get? then you can essentially optimize that um you know um data mixture uh sorry optimize that function for the optimal data mixture and then that is the data mixture you use to train the large scale model.

62:03 Okay, so that's the kind of the simple procedure and you'll notice some kind of similarities with with scaling laws where you're trying to do some cheap computations to figure out some answers and then uh you know scale up. Okay, so there's a few design decisions here. One is you know what are the mixtures that you're trying out here? You need a distribution over distributions here. Uh so often people use some durlay uh distribution.

62:34 Um you also have to define the regression method. Um people have tried linear models or boosted decision trees. Um you have to figure out the target here. Um often this is based on downstream eval. So, we have to be very careful not to overfit, right? Because we're doing pre-training and then, you know, supposedly pre-training is supposed to be training this general purpose model and we're not trying to fit some downstream evals. And if you're not careful, for example, if you have a bunch of code evals, then well, guess what? You're going to upweight all the code data. That's, you know, not rocket uh science. And if you then go and you say I want to generate some poetry, you might uh you know realize that you've over over fit. So you have to be very careful about this. So proportional mixing and uniform mixing don't have this problem because you're not looking there's no downstream evals.

63:27 And the the final thing is that you know how much of a difference between small and large scale um is there and this is a cost and accuracy trade-off. Of course, if you train really really small models, um this might not be representative. But if you train large models to do this, you well, there's no point in doing any of this. You just train you're basically doing hyperparameter tuning at the largest scale, which is too expensive.

63:53 Um so this Omix paper actually does a nice um has this nice table which shows you for a number of different um methods that all fall into this kind of um framework. um this the size of the proxy models they're generally fairly you know tens or uh uh tens of millions of parameters. How many you know points do you sample mixtures do you sample um m here being the number of parameters in your mixture sorry the number of domains rather um and you can use dishlay you can use exponential um and then the regression model uh you can fit a log linear model which um you know tends to work uh pretty well um and then uh you can use different ways is to solve this optimization um you know problem.

64:50 Okay, so there's sort of two leaps of faith here that you just have to be very careful of is that um the regression model was trained on a bunch of these small proxy runs and you're optimizing this. So the hope is that your the minimizer um your optimal data mixture that regression model is still accurate. Um, and this is a little bit kind of, you know, you have to be very careful because, um, if you, let's say, sample a random mixture, you'll probably be fine because, um, because of, you know, classic generalization, you sample a mixture, you and you fit a function, it should be able to predict in distribution. But when you're optimizing, you're essentially trying to go to potentially the extremes where you might not have as much coverage. So that's one thing to be careful of. And the other thing is that um optimal uh data mixtures you just hope that they transfer for small scale to large scale.

65:54 Um and this in general it seems like at least at the scales that open community works with um tends to be true or not blatantly false. Um but clearly there are uh scale dependent effects. If you just think about the the the previous slide when we were talking about filtering, if you're training for many more tokens, then probably low quality data is is okay. So clearly the optimum is not the same, but you just kind of hope for the best here.

66:24 Okay, so hold on. There's one scale dependent effect that we we already talked about but and we need to address here which is that um you imagine so we have a lot of lowquality data 10 trillion tokens 10 billion tokens of high quality data and if we train a small model on low token counts you know remember uh what what happens is that you might this is a madeup um you know uh example But um you might put a lot of mass on your high quality data because if a low token counts, you're not epoching. So it's like, oh wow, Wikipedia is so great. Let's just train on Wikipedia. Um but if you then go and train your large mixture on this large model on this mixture, um then you're going to end up epoing a lot on this high quality data and you'll overfit and that's definitely um bad.

67:24 So one way to fix this which is done in the OMIX um this Omix uh paper um is you can just uh cap the number of epox solution. There's another solution here which um goes by probably multiple names but one name for it is simulate epoching. And the the the principle here is that make your small scale look like your large scale. This is kind of a general theme for the course. We saw it in kind of uh when Tatsu talked about MUP. Um you parameterize your model so that your your hyperparameters transfer.

68:02 And so the kind of same principle applies here. And the problem with uh you know doing this kind of naive scaling is that um at low you can't be not epoing at small scale and epoch then epoing at large scales because then those are qualitatively different um operations on your data set. So the way you would handle this is you um you downstream your sources proportionally.

68:32 So um so if you have your small runs were 10 billion tokens um and your big run were one trillion tokens then you have um a you know one in 100 kind of downsampling um and and then uh you basically use the down sample mixture. And so the idea is that if you have the down sampled you know data um then this solution is not going to look good because you're not training on all you're not you don't get to just train on all Wikipedia you're going to train on some minuscule fraction of Wikipedia and you'll realize oh actually I'm going to be epoing a lot and that's going to have really bad uh loss which means that in the optimization you're going to look for more balanced uh uh approaches. is so you're kind of simulating the data scarcity that you would get at in your high regime at a low scale.

69:34 Okay. So to summarize this section um the problem is how do you weigh different sources? You have Wikipedia, you have your common crawl, um you have your your code, you have some math data that you you scrape from somewhere. Um how do you balance them? Um regression based mixing is is a nice framework I think to think about things. You you estimate your a function form uh that maps your mixture weights to a loss at small scale optimize and then you generalize to large scale. And you just have to be very careful with uh this uh because of epoing and overfitting kind of issues.

70:17 And you have to either use capped epoing or simulate epoing. So one lesson here is that um if you're trying to optimize anything you have to be very careful because you are in danger of optimizing the the wrong uh thing. Okay. Any questions about data mixing before I move on? um when you're a dev sample assume something is like too small that like you can't really generalize as well.

70:52 >> So uh the question is if you're downsampling is there's a risk maybe you get too small of a data set. Um yeah absolutely. So um I think right you might get into a case where you just have so few tokens. Um I think then what will happen is that your optimum will just put very small mass on that and when you scale up maybe you get the right set of tokens. So in principle like it should work out but there might be like rounding errors where like you end up instead of training once on this data set you train like zero times by accident. But you can you can sort of work around these by just like um think you always train once on this data. Yeah.

71:44 Okay. So, yeah. >> Within data mixing, um, do you ever because like I guess these sources just happen to align with different topics, but do you ever take like a diverse data set and then try to do like data mixing within that? >> Yeah. >> So, the question is uh do you do what do you apply data mixing to the like different you know domains or even within a data set? That's actually I forgot to mention that. So I'm glad you brought it up. So in Neotron uh paper they uh actually in in some of the OMO stuff as well you basically think about let's say I give you a just common crawl right you can actually break this up so you can group by domain. So they the AI2 folks have this like web organizer so you can group into topics but you can also quality filter. Um so then you you have this like two-dimensional grid where you have domains and you have quality and each of these like cells is something that you would data mix.

72:45 So that's one automatic way to determine the the domains and then on top of that you add ex extra sources that people hand you. Okay. So I'm going to quickly go through post training uh data here. Um so up until now this is basically pre-training or maybe mid-training data. It's generally fairly task agnostic except for the reg mix stuff where you're you're trying to minimize the loss but still the data itself is still task uh relatively task agnostic and you're trying to develop uh basic skills. So when you look at post- trainining a lot of the data becomes very kind of task uh you know dependent um and I'm not going to do a comprehensive review I just want to point out some of interesting um post-raining you know data sets that have been recently released in particular for coding since that's uh of I think great interest these days. Um so the general recipe is um you define a set of you know environments um in the case think about code which might be GitHub um you know repos you define a set of uh tasks or or prompts and then you collect responses from a strong model or teacher. So implicitly you know at least in the open community almost all the post- training data that's uh most of it is you know synthetically you know generated um you can also replace strong model with uh a human uh but you know that is slow and costs a lot of money but you know if you're at the frontier and of um if you I guess a few years ago you kind of had to go and pay a lot of people a lot of money to give you responses. Um, and nowadays, even at the frontier, you can do sort of hybrid human AI things.

74:47 But anyway, the the point is that there's some teacher out there that's giving you responses. Okay. So, um, a few kind of I'm just going to talk to a few works here. So, one is this works, uh, called open thoughts. Um and this idea here is um this came around it was motivated by you know when 01 came out and there was a lot of intention on reasoning mostly for like math and science um how do we get really good post-training data sets for this and um you know they eventually came out with 1.2 two million examples uh using this teacher model. Um so the steps are first of all um there are a bunch of so what are the environments and the tasks and questions. Um there are a lot of different sources um that they they drew from there human sources like sack exchange um or num math and then there's like some synthetic sources as well.

75:51 These are just a subset of the the coding ones, but there's also more for math and chemistry. This is a big project that involved a lot of different uh contributions from many uh people. Um so you can see that there's um you know some of these are like coding exercises, some of these are um you know more realistic things that show up on um I guess code golf is not realistic but there are some more realistic uh um examples here. Um so code review this is probably more realistic. Um and they noticed that they did a fairly comprehensive uh analysis of you know given this data set what how do you generate responses. They show that um having a few sources was actually good but rather than trying to do all sources sampling multiple generations is helpful like 16. Um one thing that was interesting is that having better models um aren't necessarily um you know better teachers. So for example QWQ 32B which is you know now a very old and small model is was a better teacher than deepseeek R1 which was at the time probably one of the strongest open models. Um they also found that basic answer filtering wasn't helpful. Um and so the entire pipeline looks something um you know like this where you have a bunch of these sources uh you know going in um and then you you duplicate you randomly um you know sample I guess this down sample the questions and you generate multiple answers and that gets uh into the final data set. So the 1.2 2 million isn't um is examples but divided by 16 gives you the number of actual questions.

77:43 Okay. So there's an another so that was for um you know math and science and and some code. Um I think more recently there's been a lot of interest in developing in particular a gentic coding um model. So not just a model that can just generate some code but actually a model that can do software development. Um so this uh Swissmith paper um um had the idea of you're given a repository, you're using an language model to automatically generate tasks. So they have an agent that um takes a repository and actually makes it usable for um uh like uh installing dependencies and so on. Um and then you generate tasks which are generally um you know you modify the code in some way maybe introduce some bugs um and those get verified and you get some task instances. So these are synthetic tasks um but you get you know 50k of them which is actually um at the time which was last year uh quite large um so since then there's been um a bunch of other uh work which I'll uh you know mention um so there's this sweet zero paper from you know Nvidia um which had this kind of um actually Okay, I'll also talk about that. So this is a kind of an interesting idea where the observation was that unlike math su task have just a lot of heavy dependencies. Most GitHub repos like don't even run and you have to install all these dependencies and out of date especially if you roll back to like when a PR was. It's just kind of a mess. Um and this is just a nightmare. Um and so they were thinking about how can we actually get a data set that just um on all the the the repos and rather than having a repo specific docker image um they noticed that the models are actually good enough that they can solve a lot of these tasks without execution feedback. Um so you know if you look at some of these uh models if you were allowing educ execution you get like 80 if you don't allow educa not education execution you get almost 70. So which is you know not bad for not being able to execute code. So um somehow these models have some internal semantics of of of code. Um so they were able to generate 300,000 agent trajectories. Um all of these are real GitHub uh you know PR so they're realistic unlike the the Sweenith Smith examples. Um they use the open hands scaffold and did some there's a lot of details like how to prevent um agent hacking. So um normally you uh hand an agent this um instruction you explore you test you implement and their uh 30 version was basically saying you cannot run Python code you can only do like set and grap and all these like basic um um operations.

81:05 Okay. So then they distilled from a big coding model um and filtered um because sometimes a coding model will ignore these instructions and still try to execute anyway. Um and then there they showed that with this um oh they also had 13k agent trajectories that do require execution feedback. So they train models where they first uh fine-tune on these sweet zero um examples and then fine-tune again on these sweet hero examples and they were able to you know I guess the frontier is still pretty uh high up but they were able to make some you know progress here. Um I'll quickly go through rebench was essentially another uh attempt to grab tons and tons of uh you know uh PRs. Um, so and all of these kind of look like get a bunch of GitHub repos, try to install the repo, like most of them probably, you know, fail and try harder and then you use a um a language model to give you the um the responses.

82:12 So in this uh this actually just came out uh today. Um so you can take the sweet zero idea and you can now scale up to 12 million agent trajectories. Um the nice benefit of three zero is that it's just you know almost it's very lightweight. Um here they use the sweet ben rebench uh tasks. So in this one remember they were trying to get things to execute. They only got 32,000 of them to execute and 120 of them didn't execute. So you just but zero doesn't care. you can use all of them. Um this is a very small model because now this data set is quite large. Um and so um so you know I think as you can see the data sets are you know getting more and more sophisticated. You go from you know environment free things like math to now coding and now the coding data sets are growing uh quite a bit. Um but the general idea is that you know you have these these prompts um which you know there's kind of trade-offs. You can have fully synthetic um you can have semi-ynthetic where you have a real environment and synthetic tasks or you can have real um and the responses generally are coming from just capable models but they need also need to be good teachers. Code environments are a pain and you there's a lot of filtering and other details which we don't have time for. Okay, to summarize this lecture, we talked about filtering. So define what good looks like and then train a lightweight classifier. Go over your web crawl and you can get a a small subset that matches what you're looking for. Dduplication is important to avoid overfitting and saving flops. uh mixing um try mixture at small scales extrapolate to large scale um and then we looked at some you know post-raining data um I will say that though that a lot of the data work is can be very grungy it's very domain specific um and requires looking at concrete examples to make these high quality data sets so this lecture is not really representative what data work is like but hopefully I've given you a idea of the data landscape out there. Okay, that will be it for today.