Transcript
0:05 okay great let's get started so welcome everyone to the fireside chat or talk on ai and language so today we're going to do something a little bit different i want all of you good to go to slide oh and the guest code is cs221 so we're going to try to use this platform for doing q a and also i'm going to have a number of polls throughout the talk so the first question if you click here
0:33 you'll hopefully see that it's what city are you in right now and i already got some responses san jose palo alto stanford seattle college station fort mill cupertino new york city so welcome everyone from all over and if you go oops i guess this zoom thing is in a way okay i guess i can't see that but anyway you can go to there should be a q
1:04 a tab where you can go and type in your questionnaire try to monitor that throughout the hour okay all right so i wanna start by asking you a very simple question what is the difference between these two cute little kittens and these two kids here anyone know the answer both can see smell
1:36 taste move around the environment kids are sometimes cute too what's the what's the main difference let's make this interactive someone just just shout out an answer humans can talk humans can talk yes thank you that is the main difference and while animals do have
2:06 some sorts of communication especially songbirds and dolphins and and honeybees have their waggle dance none i think could boast as such a rich and complex language as the human language so it's really i think language is something that's uniquely human and defines who we are so before getting into talking about ai and nlp i want to talk spend some time just talking about why language is special and hopeless that we can get a richer appreciation for a
2:35 language so if i had one slide to summarize language this would be it so this is one of my favorite xkcd comics some of you probably seen it but i'll just read it anyway because i think it really highlights the the right mode so anyway i could care less i think you mean you couldn't care less saying that you could care less implies that you care at least some amount i don't know we're these unbelievably complex brains
3:05 drifting in the void trying to in vain to connect with one another by flinging words out into the darkness every choice of phrasing and spelling and tone timing carries countless signals and context and subtext and more and every listener interprets these signals in their own way language isn't a formal system it's glorified chaos and you never know for sure what any words will mean to anyone all you can do is try to get better at guessing how
3:29 your words affect people so that you have a better chance of finding ones that will make them feel something like what you want them to feel everything else is pointless so i assume you're giving me tips on how you interpret words because you want me to feel less alone if so then thank you that means a lot but if you're just running my sentences past the mental checklist so you can show off how well you know it then i
3:53 could care less so what do we learn from this so first language is is social it's it's meant for communication right i think a lot of us coming more from a kind of data or ml background i think language is just a body of text but it's really this this dynamic thing that humans invented communicate with each other the other thing is that language or talk is is cheap and something about language requires
4:23 incredible amount of trust between the people so that it actually can can function but interestingly it can also be used to deceive which is interesting right now and it's just kind of miraculous how language allows us to express all these different thoughts from poetry to math to how to fix a bike and so on so where did language come from the short answer is no one really knows and it's really hard to pinpoint it because
4:54 while writing came around 3000 bc and before then it was a long period of spoken language and spoken language doesn't leave fossils or anything and there was so much skeptic kind of controversy around that that was actually banned the study of origins of language was banned for about a hundred years and in paris so but but we do just kind of conservatively put an estimate it started maybe 2.5 million years ago when homo sapiens first came on the scene to
5:27 sometime between 100 000 years ago which is when modern humans really started doing things which is a huge range but to put in perspective this is a very recent development compared to the history of all of life on earth and we know that it served an evolutionary purpose so if you read sapiens this book by yuval harari language is perhaps one of the key reasons why homo sapiens became so dominant because they allowed you to
5:59 communicate and coordinate on such massive levels and for example detecting when coordinating on the hunt for example or communicating about food sources and so on and interesting language allows you to talk about things that aren't here and now that that is probably the one of the most powerful things and in fact it tells allows you to talk about things that don't even exist there's a whole genre called fiction that's about that and things which are in the abstract
6:28 so in contrast like i said before you know our kind of sister fields of you know computer vision robots robotics tap into capabilities that have been around for you know much much longer like vision is over 500 million years ago and language is you know barely let's say conservatively maybe of a million or two years ago so just for fun let me do a poll let me actually have to create the poll
6:59 first so what language do you speak languages do you use here speak let's see multiple choice free text okay let's see if this works i have to disable okay so go to the poll and let you fill that out so we know that language there's not one language there's multiple languages
7:30 and furthermore languages have evolved so you can draw kind of a giant family tree of languages and this branch just shows the indo-european languages which covers all of europe and iran and some parts of northern india which developed like 10 000 years ago and this branch often to kind of germanic languages and romance languages germanic went into german and english and so on and today there's 6
8:01 500 languages many of which are actually going extinct because language again is social so if you don't have anyone to talk to your language just kind of disappears and language is changing all the time you know english has definitely evolved since shakespeare but you know i think in grade school you're probably told that they is supposed to be plural and you shouldn't use say they to refer to a singular person but now it's especially with kind of this trend for
8:32 having gender neutral pronouns they is you know kind of proudly singular and miriam webster kind of declared it as the word of the year in 2019 you can think about internet slang and emojis is also kind of a continuation of language into the kind of digital sphere and so on okay so i'm getting a lot of english mandarin chinese japanese so you know quite a bit of python 2 python 3. yes very nice okay so
9:04 one thing that is i often get asked is you know all these languages are some more harder or more powerful than another it's clear that they're and i think widely accepted that all languages are kind of basically equivalent but there is kind of this hypothesis around the 20s called the superior wharf hypothesis that says the structure of language affects speaker's world views and you see this in kind of a fiction like george orwell's 1984 which talks
9:34 about a new language called newspeak which was simplified so that they could make sure people couldn't think to even critique their government this has been challenged by a universalist school chomsky and pinker who thinks that language and thoughts are universal and all the differences are very superficial governed by a few parameters and it is true that languages do differ they are largely the same most languages have nouns and verbs that refer we all
10:06 are humans living in the in the same world but there are some differences you know one for an example is that english lacks what is called inclusivity which is the distinction between when you say we it's ambiguous whether you mean to include the speaker or not to include the speaker whereas some languages like tamil or some chinese dialects actually have that distinction or mandarin chinese lacks the distinction between past tense and present tense but of course has other
10:37 ways of you know accommodating for that so one question maybe to just have another poll is let me stop that poll is do you believe believe that language shapes thought now i know that these questions are you know obviously not binary but i just wanted you to kind of you know get a gut feeling are you leaning more
11:08 towards yes or no on that and i will activate that poll so if you are do you kind of abide more by the superior warf hypothesis that the structure of language does influence how you think about the world or do you think that all humans are really the same and as we just happen to learn different languages and those differences are fairly minor so can you guys see the the numbers
11:40 okay so we have about 90 percent superior dwarf and about 10 no okay so this is a richly hot hotly debated topic in linguistics even you know to this day so another fascinating thing about language is that we're not born knowing it babies can make sounds but it takes them a few years to be able to actually acquire language and importantly despite what
12:12 their parents might think they're they're not taught from explicit instructions or from teachers rather they're taught where they learn naturally from immersion of language and you know by the time they're five they're actually fairly kind of have a fluent grasp of the language and speak grammatical sentences and so on and its language acquisition is very multi-modal and grounded so language accompanies sight and sounds and actions and touch and all these things
12:43 and it's active you can't teach a child by just putting in front of a tv and expecting language acquisition to occur so you know one of the big big questions around language acquisition is is is the nature of nurture debate which also affects kind of other areas as well so the great question is language you know innate and
13:14 so chomsky who was a famous linguist in the 50s he came up with this idea of what he called poverty of the stimulus and he said that basically sentences that a child hears can't possibly be be responsible for the the richness of language that's you know exhibited in actual in actual humans so
13:44 so he thought he concluded naturally that language must be a large part of language must be innate so i'm trying to delete this poll and add it add a new one at the same time i'm talking okay so i'll ask you this poll do you think language is an egg and if you think about it he does have a fair you know point because you know
14:15 you know we do as children here's so few of the examples and then we can you know not nearly enough to really capture all the cases and we are constantly run into new language all over the all the time and we have to generalize compositionally to longer sentences to new context and we kind of all land on this kind of same language so you know i think he does have a point on the other hand you
14:42 know he was an experimentalist he was a kind of a you know classic armchair linguist who thought deep thoughts about how things should be and there's also you know one could imagine you know what about the role of grounding and maybe that is the it's part of the experience that really shapes language acquisition and maybe we are just kind of really malleable and so on so it seems like everyone's quite divided on this so 53 yes and 47
15:14 no okay that's always fun so maybe you guys can you know talk about it that's with your friends what i will say oh now now that whoa this is a tight race what i will say is that you know no matter where you're on the spectrum once you have kids you you really think that there's more innateness than there's not so now it's equal 50 50. okay great so let's move on so let's take a look at language you
15:47 know itself and i'm going to introduce some basic concepts so there's a whole field of linguistics that studies language and i really encourage you if you're interested to take a linguistic class i think it's one of the most interesting eye-opening kind of experiences but i'll just go over some kind of basic things quickly so here's a sentence beethoven was born in bonn and displays his musical talents at an early age now what's going on in the sentence the
16:12 linguists ask you know what is the structure of the sentence allows us to understand what it means so first of all there's tokenization which is you know the the sentence is just a stream of characters right and tokenization is a process of converting that into words seems very simple right but as we'll see later it's not as simple as it kind of looks part of speech tagging is the idea that some words are nouns and other
16:40 words are verbs and some of the words words verbs are past tense versus present tense parsing goes a bit a step further and talks about the grammatical relationship between words for example displayed has a subject and object and the subject is in this case what is the subject anyone i know they're english speakers in the audience
17:16 is it bond okay i think maybe it's okay it's beethoven right okay so even though beethoven is very far away from display it's nonetheless grammatically very close to displayed and this is what this is a property of language language has to be linearized so not everything can be close to each other core reference resolution or anaphora corresponds to the fact that his some words are
17:47 pointers to other words or concepts so his refers to beethoven named entity recognition is the task of identifying which entities now usually proper nouns are per people or locations or organizations so what is what is a word so let's go down into a word so okay here's a word light so you know it seems like a word is pretty straightforward forward at least in english but the problem is that the meaning of unit
18:18 what what is kind of conceptually should be a word actually goes beyond a word so light bulb is kind of a unit it's not light doesn't really capture the the really the full meaning of that word sometimes the meaning unit is within a word for example lightning isn't just a blob it consists of light and then the suffix en ing which kind of you know trims it into a verb you know gerund construction there's also cases where words have
18:49 multiple meanings usually called word senses so in all of these sentences you can see that light actually has a different meaning and we figure it out based on context on the converse conversely some meanings have multiple words that refer to essentially the same meaning that's synonymy this also happens with sentences so this is called paraphrase multiple sentences could get at the same meaning
19:20 just some huge caveat is that there's no true equivalences between any two words or sentence sentences there's always different subtleties and meanings so you can think of more as a kind of distance metric there's also notions of relations between words like hypotonomy which is is a relations and mironomy which is hazard relations and this allows you to make do entailment which is whether a sentence logically implies a second sentence entailment
19:52 you can kind of think about the the three set of end of language it's kind of the problem that embodies a lot of different tasks if you could solve entailment you can do question answering you can do sentiment classification you know and so on oh are there i haven't been monitoring the qa i don't know i don't think there's any questions if you have a question maybe just holler so this is all about lexical semantics
20:23 the meaning of words then we talk about compositional semantics so this composition of semantics is kind of rich tradition that goes back to kind of logic so this is fraga who's a was a logician at the turn of the 20th century and there's two ideas model theory and compositionality which i'll explain so the first is that sentences are these are just symbols right it's it's a convention that we say block 2 is blue and what
20:53 this sentence means has to be associated with what is in the world so there's a world in which the block 2 is actually blue and so this is kind of important distinction which we kind of you know we kind of gloss over or you probably just we don't even think of it because of you know language is so natural the second one is compositionality which is that the meaning of a whole is the meaning of the parts so you could that
21:20 is this is compositionality is the key thing that allows us to build more complex meanings out of smaller units and this is probably the reason why we can generalize to all sorts of new contexts because we've learned the meaning of the words and we know how they combine together and that's how we can generate you know we can interpret the new sentences and new contexts quantifiers i think are really you know
21:51 interesting so every is a word that says you know says that well it's hard to explain language without explaining in terms of language so every means every hopefully these pictures tell you what's going on and some is a kind of an existential quantifier there's also a quantifier scope ambiguity which means that if you have every non-blue block is next to some blue block that could mean that every blue block is next to some blue block which is that that
22:22 was very tautological which is is could be different or that there exists some blue block that's actually next to some to every non-blue block so language is you know ambiguous so modality or involve words like must and can and this has to has to do with possible you know worlds in all these possible worlds
22:52 block 2 is blue but block one is only a true in is read in one of the worlds beliefs are interesting so you know we know that clark kent is the same person as superman but and naively you might think that we can just substitute these two in all contexts because they're equivalent but you know lewis believes that superman is a hero is not the same as lewis believes that clark kent is a hero
23:23 and this has to do with the fact that you know this believes sets up a kind of opaque context which you can't just do substitution there's much more to be said you know about this if you study linguistics i just want to give you a flavor for how language can be kind of quite subtle here's some other examples of you know pragmatics so pragmatics the conversational implicature is this phenomenon where there is a sentence that you say
23:52 but there's actually additional meaning beyond that sentence so if two people are talking and a says what earth happened to the roast beef and b says the dog is looking very happy sure the dog is looking very happy that's a sentence but really the implicature is the dog ate the roast beef presupposition is is actually kind of very you know subtle but different is the background assumption that's independent of the truth of a sentence so if i say i have
24:21 stopped eating meat what's the presupposition that means i was once eating meat okay so regardless of whether i have stopped me or meat or if i even have i said i didn't stop eating meat that still pre-supposes i was once eating meat so presuppositions are these very slippery and insidious things that people use to convey to convince other people of things that without them knowing so be it's really useful to know what a
24:52 presupposition is because if someone tries to presuppose something on you then at least you'll have the language to you know detect what it is and it's precisely insidious because it's in the background so you're focusing on you know did i stop eating meat without realizing that you just got you know a presupposition that you might not agree with late on you okay so paul grice who's this full of you know philosopher the established with kind of
25:23 established language as a kind of cooperative game between a speaker and a listener and is that the dynamics of the game is what gives rise to these things like conversational implicature and presupposition right this goes back to xt comment it's really a a game between speakers and listeners who are trying to communicate and agree on something and the conventions and what language means in all these contexts is really kind of context dependent and fluid
25:55 just a few other ideas ambiguity vagueness and uncertainty so let me try to explain what each of these means and how it's different so ambiguity means that a sentence has more than one possible but precise interpretation so here are some headlines and let me know what you think of them okay so stolen painting found by tree okay so what does that mean how about iraqi head seeks arms or local high school dropouts cut in
26:26 half juvenile court to try shooting defendant kids mix nutritious snacks ban on new dancing on governor's desk and you can see if you're i have smiling a little bit that these these headlines are funny because they have a maybe a serious meaning and then a meaning which is totally ridiculous but is nonetheless kind of technically ambiguous vagueness
26:57 is where a sentence is not has one interpretation but it does not specify the full information so if i said i had a late lunch you know there's no ambiguity there it's just that i didn't tell you what time i laid lunch maybe it was one o'clock or two o'clock or something uncertainty is another form of you know not knowing something and it's due to having a perfect model so say the witness was being contumacious
27:28 you know some of you probably know what that means so you're not uncertain but you know i'm some of you probably don't and you have this uncertainty which is not a property of the sentence but of your the speaker's ability to understand you know natural language so all these things are useful to think about differently although often they get conflated especially in kind of more model-free methods
28:01 so i will say that there is another forum of style of linguistics called distributional semantics which is actually goes back to the 50s as well and i'll give you the basic idea so if i give you these sentences the new design has blank lines let's try to keep the kitchen blank i forgot to blink out the cabinet so what does blank mean or which what word goes there
28:36 someone say the answer the answer's a chat oh okay i didn't know there was a this is in the zoom chat yep okay let's see i think i lost my okay chat okay there we go okay there are our answers okay clean great
29:06 okay got it okay so the idea about disreputation of semantics is i didn't have to tell you what the word means the meaning of the word is characterized by the contexts in which it appears so this is idea of the distributional hypothesis semantically similar words occur in similar contexts or more eloquently said by firth you shall know a word by the company it keeps so this is another way of thinking about semantics and actually the one that has
29:38 really been picked up because it's so synergistic with a modern kind of statistical techniques so just going to summarize there's there's two ways of thinking about semantics or meaning of sentences one is composite essential semantics where it's more top down and you model first so you think about how language works you try to you know think about parse trees or you know semantic forms and you can capture a lot that way we
30:10 can we went through a lot of examples where language you can feel kind of language does obey certain types of structures on the other hand you can think about distribution of semantics which is a bottom-up data first approach and generally associated with vector spaces where you think about just words as meaning and not really trying to nail down what meaning is but just associated with the set of contexts in which you know appears so let me do another
30:40 poll and let me hold on i need to create this question and ask what do you think is the best way to achieve natural language you know understanding so is it compositional semantics or distributional semantics
31:10 okay so go to slido and i'm curious what you you think okay so it would have been interesting to go back maybe 10 years and ask this question because i think the answers would have been quite different and i'll talk a little bit more about that in a bit so it looks like
31:42 it's about 30 compositional and 70 maybe quarter three quarters so most of you think that distributional semantics is the way to go which is maybe a concordant with what is happening in the world right now okay so why do i take a few minutes right now i went through a lot of material and maybe i'll just ask if there are any questions
32:15 to discuss no questions
32:49 someone has to have a question so one question is context information it's never spelled out but the meaning depends on who is speaking out where yeah so i've been kind of deliberately vague about what context is traditionally it's a linguistic context which is the words next to a sentence but you could imagine that it could be very much generalized to
33:19 you know context of the speaker in the you know multilink multimodal what's going on in the world what who is speaking to whom the person is speaking and all of that rich you know contextual information it definitely is useful for understanding the meaning of that word so in the beginning you said that humans have the highest level of communication how likely is it that so actually some animal has much higher
33:50 level of communication but we're not smart enough to understand it yeah that's a that's an interesting question so it's yeah it's almost a little bit of a you know philosophical question because you know it is you know in theory possible that some animal has you know brilliant system of communication and we just didn't you know measure it you know properly people have been surprised
34:20 by how sophisticated certain animals are able to you know communicate you know like you know dolphins or elephants or even you know you know bees often people separate the line between having recursion or language that's able to express kind of compositional thoughts versus ones which are maybe very contextual nuanced but but don't have that kind of level of
34:52 abstraction and so you know according to that i think we're pretty sure that humans are the ones that have the most amount of abstraction but then again i guess this is also a very human-centric way of defining you know what highest level of communication means because maybe some other creatures have more you know contextual more nuanced than you in human language elephants communicating below 20 hertz
35:22 infrasound okay and hitchhiker guide thanks that's a good one how about plants communication can be chemical color temperature even touch yeah so there's a lot of other and i guess in general communication is not the you know same thing as as language yeah so i'm using language very narrowly
35:54 here to mean kind of human language and what we know to be human language of course even humans communicate in other ways like gestures and and so on okay great let me move on for the questions so what i want to do next is to talk about building language understanding systems and what has happened over the last
36:25 60 years now that we have maybe a greater appreciation of what language is so we've kind of seen this slide before it's the turing test alan turing in 1950 asked the philosophical question devise the terrain test to test whether you know a a computer could be or a machine could be intelligent by seeing if it could talk to a human and convince the human that it was actually
36:56 a person this is one of the dialogues that they have in from his paper you can read it what i want to emphasize here is that you know turning was not interested in language and wasn't trying to design a language understanding test he was trying to design a test of intelligence and language was just the means to convince someone or to kind of verify that there was something you know up there
37:26 and so i think this is kind of very interesting from an ai perspective because language does seem like it has a unique capability of being able to even you know in order to have intelligence you need to be able to essentially show it in some way and and you can think all the deep thoughts in your head but if you don't have a way of kind of externalizing it then well
37:57 it's it's like the if the tree falls in the forest doesn't make a sound kind of you know setting so and it language has turned out to be a very useful way of kind of probing for understanding one of the early systems i want to highlight is know eliza so joseph weisenbaum when he was at mit developed this system called elisa which was really meant to be a parody of a psychotherapist
38:28 so the system would you would lie on a couch and you would talk to the system and some dialogues would go like this or eliza would say is something troubling you men are all like what's the connection do you suppose they're always bugging me about something or other can you think of a specific example well my boyfriend made me to come here is it important to you that your boyfriend made it you come here this is
38:53 so depressed and so on so what was interesting about this system is that the system itself is very simple this is you know 1960s so there's not that much kind of you know going on in terms of you know complexity it was based on rules and matching so you can imagine a rule that says if the word alike shows then you ask you know what is the connection or if you say always then you say can you elijah says can you
39:24 think of an example specific example and so on so it was very simple but what joseph weisenbaum found out which was really surprising is that the people he showed this to actually started getting emotionally attached and there was one incident where as invalid secretary actually asked joseph to leave the room so that the secretary could have a you know a real conversation with eliza so this is kind of in the 60s i think it
39:54 was very perhaps you know i guess telling of maybe what is to come later i'll talk about gpg3 which is obviously a much more realistic version of this but you could think definitely about some of the you know consequences of of that technology incidentally weisenbaum later in his career became very pessimistic and actually very negative and critical about you know technology maybe because he had this epiphany that well what we're building is actually maybe not so
40:25 good after all so this is one of my kind of favorite natural language you know systems it's been built by terry buenograd was also at mit but he moved to stanford where he was became a you know a hdi faculty for a number of years it's called churlu and the idea is that you have a person who is able to conduct a dialogue about a block's world environment so pick up a red block okay grasp the pyramid this computer is going
40:55 to say when it doesn't understand things you know find a block which is taller than when you're holding and put it into the box so it's you know fairly complicated and that the computer can kind of reason and do a nap for a co-reference resolution and ask for clarifications and so on what i think is remarkable about the system is that it was an end-to-end system included a parser it can do semantic interpretation dialogue planning it
41:24 wasn't just a language system in fact it was more framed as an ai system that could allow a robot to kind of do things in the world you know and so this was you know in some sense kind of the first the real super ambitious project for its time however you know while sherloo worked really well in the limited domain terry werner later wrote this
41:56 paragraph which is interesting said a number of people suggest me that this is a dead end in programming complex interactions between the components made it just really hard to understand what was going on so eventually terry couldn't even build and extend the program because it was just too hard to keep you know in his head so this is you know interesting because as we know language understanding didn't you know really get
42:27 you know solved despite these kind of narrow successes and the history of nlp mirrors quite closely the history of you know ai in general remember in the first lecture i talked about how ai was filled with more kind of these logical based methods which didn't quite scale what's interesting is that at that at that time in ai in general there were people working on neural networks although the
42:58 vast minority of people but in language it was perhaps even less so because i think language is actually a discrete communication system and linguist there was kind of a rich body of work on linguistics and nlp and linguistics kind of you know co-evolved in in certain ways that made it kind of very natural to embrace the kind of all the the logical structure that was embodied in you know in language
43:31 but you know i think people realize that the there was cracks that were you know showing at the seams in the even in the 70s but especially the late 80s and in 1990 it was a time for a new set of methods to come onto the scene so this actually started a bit earlier from speech recognition because speeches and language are you know closely related and speech is definitely something that you know is
44:03 the bridge between the continuous kind of noisy world where you want to be doing kind of more pattern recognition type things and the kind of a logical world so hmms hidden marker models were developed for speech in the 70s and 80s and finally in 1990 there was a kind of famous paper from ibm research colloquially called the ibm models for machine translation they developed a probabilistic model that could translate between you know two languages and at
44:34 before then translation was completely kind of logical and grammar and rule based and this was a radical way of thinking about it this is actually instantly based on a lot of the bayesian networks or that we'll see you later in the course so for a lot of the 90s a lot of these what are called generative models think about them extensions of asian networks really kind of dominated nlp near about 2000
45:05 there were people started turning to discriminative models you know aka you know classification linear classification and there was another famous paper called which introduced conditional random fields which marries the structure that was so inherent in in language with with basically linear classification
45:35 so this was used to do things such as named entity recognition where you would mark up words as names of people or companies and so on and so instead of predicting just one y from x you predict a bunch of y's from x where y's are the labels of all the words so this technology was actually quite influential in in nlp but also more broadly in computer
46:05 vision where for much of the 2000s people were invested in having this was a kind of main way people tackle kind of structured tasks another thing i'll mention is lane dirichlet allocation which also came from models of language and here the emphasis on unsupervised learning where you point lda at a text and it can discover topics in the text so here's a text where it
46:35 can discover things like oh some words are about budget some words about children and some words about arts in kind of unsupervised way and this led to a whole kind of cottage industry of topic paper topic modeling papers and lda you know continues to be something that you know is commonly used in you know in practice what i will say is that a lot of these developments it's interesting to kind of think about
47:06 how they were developed by someone trying to solve address a problem in natural language processing and it led to of more general technology that then was applied in all sorts of different you know areas like computer vision and genomics and so on okay so so now 2000 you know we're ending and we know that at the end of the 2000s deep learning really started gaining momentum
47:39 it wasn't you know it was imagenet was 2009 so it wasn't like a you know a huge yet but there was definitely kind of rumblings and it's interesting how kind of culturally how the nlp community kind of reacted at the time nlp and vision were both all very kind of skeptical and if you think about where nlp had been a lot of people still view languages you know structure heavy and languages has a lot of lane
48:11 structure and no way that this mess of neurons could actually do anything with this kind of intricate structure and you can think about a lot of the work in the 2000s was really a marrying of this structure with kind of statistical methods so it was you can think about as putting probabilistic choices on kind of this very rich structure discrete backbone right so in some ways this is kind of a
48:41 reconciliation of the compositional semantics with the distribution of status you have it you have both but but still i think it was still largely based on kind of traditional kind of linguistic thinking so then i remember there was this 2011 you know workshop at nureps i was at it and there was there were some so neurops is you know the machine learning conference so there were a
49:11 bunch of machine learning people who were using vector based models to argue that this covers semantics and then you have ray mooney who is much more of a logic old-school ai at the time kind of person and a heated argument kind of broke out and he is famous for saying you know you can't cram the meaning of a whole sentence into a single vector okay so so that was captured the attitude you know at the time
49:41 and then things kind of started changing i think the first kind of you know maybe move was you know word to vec which was this way of taking a lots of text and embedding words so that each embedding was characterizing the context of that word so actually word representations have been had been around you know since the 90s but it was somehow this word event was came at the right time that
50:12 really caused people to pay attention and i think one thing that they noticed which gathered a lot of attention was the fact that you could do analogies for example if you embed things in a vector space you see that country and capitals are related by a consistent you know relationship you know with some asterisks there was a recent paper from last year which i'll which i'll just highlight because i thought it was really interesting i mean
50:39 six years later and they used a kind of a simplest method but it was they ran this word event just on you know about three million abstracts of material science papers just strings and they were able to discover certain types of patterns by looking at the vector spaces and actually predict certain types of chem compounds as being having certain kind of material properties like you know therm being thermoelectric so this is kind of an interesting
51:12 view of how something that's so kind of you know dead simple and knows nothing about chemistry and only knows about what crew occurrences can actually you know generate some interesting you know insights so word effect wasn't you know deep learning in the sense that it was only i guess one layer so it was kind of shallow learning and i think 2014 was when the deep learning community really kind of i don't know
51:43 in some sense you know vindicated itself in the nlp community so there's a sequence the sequence learning paper from google in 2014 where they did machine translation and the way they did machine translation was by taking a sentence and running a lstm some if you don't know what it is it's fine it's just some black box that embeds the sentence
52:13 into a single vector and then using that vector it spits out a new a new sentence so if you look watch the module on differential pro programming it'll give you a better idea of what i'm talking about so this was really cramming the meaning of a sentence into a vector like literally and you know at that time the results were you know kind of okay but it was enough of a proof of concept and you know surprising enough that it later
52:43 extensions of this really transform into actually usable you know technology so it's interesting if you look at the progression from rule-based you know machine translation where there's no machine learning to statistical machine translation where they're still you know probably still data driven but based on more or less kind of some sort of structure of language to the neural world where there's really even less structure
53:15 and things have kind of you know gotten you know better there's a researcher called fred gelnak who's famously quoted as saying you know every time i fire a linguist my accuracy goes up i'm not sure he said it was exact words but that's at least kind of an urban legend one other note i'll make is that no machine translation seems to be the task at least in nlp but maybe more broadly that has really
53:46 pushed the limits of kind of you know technologies and i think it was really the driver that got you know c2c technology kind of going so in 2016 google kind of completely transformed their machine translation system and instead of having multiple systems one for every pair so you have like n squared pairs you actually have one system that can do translations between any pair of languages which is you know really kind of you know
54:16 mind-blowing at the time and still kind of i think kind of impressive i'll mention i think i mentioned some of these things already but i think it's worth highlighting that you know these statistical methods do have a lot of biases in them so if you translate these sentences you get genders appearing which are correlated with certain types of professions this is even more extreme so if you take a rare language where there's not much data and
54:44 you pump in something that's just kind of garbage you get some really some you know disturbing translations coming out this was a film a few years ago so they might have fixed this but nonetheless it turns out that you can cram a lot into a vector but there's some really weird stuff in there so then maybe this is a good time to pause if there's any you know questions before the next wave of slides there are some questions on slider
55:14 oh okay sure i guess i was looking at started looking at zoom chat okay so questions oh whoa okay there's yes i'll cover gpg3 how about body language so i think i mentioned that gestures are typically not studied in nlp but but they're definitely fair game for you know communication and there's i think
55:46 interesting kind of subfield mnlp which has to do with grounding and how people use language in the world which it's natural to consider gestures let's see can we build a common electric language with precise meaning so all languages can reference based on it so this was this has been tried so people have there was a language called laguan which was developed to remove all ambiguity you know from
56:18 language so it would be you know precise and everyone would know what you mean personally i think it was a kind of a little bit of a fool's errand because i think that ambiguity is exactly what allows language to be so efficient so the meaning of a of you know as a sentence is a function of not only the words but also of the of the context so if the context already makes some things kind
56:49 of obvious then you don't really need to say it so and also it's just not language has you have to take into account the ease of acquisition and it's much easier to kind of almost by construction to learn the languages that we've evolved to learn and something that's designed is generally not you know going to be super productive so nlu is a combination of plausibility
57:20 and fluency so what is i guess natural language understanding is something i haven't quite defined because i think there's no accepted you know definition of it you can think about it as demonstrating a proficiency or a number of you know tasks such as question answering or translation or i guess if you think about generation you have to think about plausibility and fluency but also you know
57:50 you know truthfulness why did you choose to do research in nlp as opposed to other areas how would you think about sorry this icon on the way what you wanted to study during the future sorry slido i don't know why slido has this icon that makes me hard to read this so i guess how do i choose to do research in nlp i guess this is a very
58:20 much i guess a personal thing so i don't think my answer is you know necessarily good for everyone but i think just the idea of what you can do with language just seems so kind of powerful to me like i said it's one of these things that is so kind of uniquely human and also i think it seems like a window to understanding you know kind of cognition because it's a way to kind of it's a way
58:53 to do the i o from out in and out of brains i suppose is there research on how to incorporate real time changes in language like constant emergence of new words and you know phrases yeah so there is a lot of interesting work studying language change over time you know there's historical linguistics which talks about you know bigger scale changes of you know latin to spanish and french and so on but there's also interesting
59:24 opportunity to do it on the web because internet language changes you know very very quickly there's always new words that come up and also on you know i guess on twitter things are geo you can geo tag things so that you can actually witness the kind of how language you know spreads over time so yeah there is an active area of thinking about you know language change not just new words but also the
59:56 new words changing in meaning like the word awful used to mean to kind of sorry the word awesome i think used to mean something more like awful but now it means it changed kind of flipped in sentiment from you know negative two a positive cool so maybe i will move on thanks for the questions okay so up until this point so 2014
60:27 is when kind of deep learning really started gaining momentum and from 2014 to 18 it was really kind of neuralizing everything you have parsers co-reference resolution systems named entity recognition systems everything kind of under the sun and you know numbers went up things did get better because the models were just more powerful than what existed previously i think a big turning point came in 2018 so there's this paper called deep
60:57 contextualized word embeddings maybe more better known as elmo and the idea behind elmo can be summarized as follows so imagine you're trying to do question answering so you know our group we actually spent quite a bit of work creating the squad question answering data set with a hundred thousand examples so it was a lot of work to get that but in some sense a thousand a hundred thousand is really really small compared to the massive amounts of text
61:26 on the web and so you the idea behind pre-training is that you train a language model to predict the next word given the previous context and so there should this is called i guess self-supervision you just make up a task which is called predict the next word given previous words and then you learn embeddings and then you use those embeddings to drive some downstream tasks where you have much fewer labeled examples and the result was that across the board
61:57 all across a number of different benchmarks the accuracies went up by a few points so i guess it's maybe hard to really appreciate what like three points means but you know it was think about these systems were already hard to improve and one point gain was very good and this is like you know substantial gain across a wide variety of tasks so this got a lot of nlp people really excited later that year
62:27 albert came out from from google and again across the which is i'm not going to go into the details actually if you watch the differentiable programming lecture i do explain a bit more about what you know bird is doing but think about it again as a it's a mass language model which is kind of like predict a word given its context and this was more or less just scaled up and you know
62:57 engineered properly and this again yielded huge gains over you know previous you know methods so so this really i think changed the game in nlp from having you know kind of specific architectures to do different tasks to a world where you have one architecture that does multiple paths so i guess i didn't mention that bird is now used to essentially power all the or bert or plus friends
63:30 some muppet is used to power some downstream nlp tasks like co-reference resolution or semantic parsing and so on so it's really kind of brings us one step closer to having kind of a unified representation back or unified one model that can kind of roll them all in a sense so going back to reading comprehension one thing that is remarkable is that if you look at the leaderboard and the accuracies
64:01 they're way above you know human level performance so these systems look like they're getting superhuman you know performance but one thing that we did a few years ago was to really probe into whether these systems actually understood language so here's a paragraph and a sentence the number of new hugo huguenot colonist declined after what year emperor correctly answers 1700 which is right here but if you add a distracting sentence
64:32 which looks like it answers the question but it doesn't then bert will get distracted and answer the wrong thing and quantitatively all the systems just fall by you know quite a bit when added when added this with this distracting sentence whereas humans obviously don't get followed by that as much so it's you know one thing to kind of keep in mind is that while models have gotten you know impressive results on benchmarks there's still these blind
65:03 spots that means that solving a benchmark is not the same as solving the actual underlying task which is can be miscellaneous if you read headlines where computers can read better than humans where that's just not true computers do squad better than humans that is true okay and what's what's a little bit more maybe worries is that these models that can be easy to break but we don't actually know how to you know fix them except for by training
65:32 larger models and hoping that they break less okay any questions before i talk about gpg 3 because that's going to be the last topic okay so is naming algorithms for cartoon tv characters a thing or just a coincidence for the two instances i would say i would gather that it's very not a coincidence because because you also had ernie that came out afterwards and big bird and it's
66:04 just you know clearly people are going along with a theme there's another cast of characters bark and marge are other i guess they're not muppets but they're other from the simpsons and that's who is another line that facebook has been pursuing is there any work on going to improve comprehension reading between the lines understanding it's actually interesting because these models large models are so contextual and leverage so much
66:37 kind of external world knowledge about text that they're almost kind of reading too much between the lines they're making inferences and making assumptions which is what at least all these biases and in the models because it's not stated in the text they're just learning from you know associations okay let me move on so to get to the now the final thing so 2020 so may
67:11 open ai releases gpt3 i'm skipping a bunch of other models like gbt and gpt2 which is in the truth of time and this was essentially a big language mall i mean big is an understatement it's a ginormous language model trained on common crawl which is our best kind of approximation of the internet so to speak and has 175 billion parameters whereas bert had maybe like 300 you know million parameters or so so this is much much much larger
67:44 so the interesting thing about gpe 3 is this ability to do what they call in context learning so traditionally if you use bert what you would do is you have a model you show an example and then you perform an update and you show another example and you perform an update and so on and this is called fine tuning a language model but gpe3 showed that that was actually not necessary to get some you know interesting performance so you
68:13 could actually do zero shot training so you could say translate english to french and you say cheese and then you give the prompt and then it will actually do something reasonable or you give a few one example or a few examples and notice that this is not training data in a conventional sense where you optimize a loss this is actually the input into the language model so language model all it does is that you give it a string and you ask it to
68:44 predict next string right so this is encoding a task actually in you know natural language which is kind of a very different way of thinking about you know learning let's see do i have enough time maybe i'll so open ai has this playground let me just show you a little bit so there's many things you can do you can so this is the prompt
69:15 and i'm gonna say how can i help you today you know can you tell me the difference between sarsa and q learning i actually don't know whether this will work okay so sure policy inspired by q learning okay so this doesn't really answer the question but that was nice but i didn't answer that
69:46 question yeah so how about something else who founded microsoft okay so this is also you know what lie but so you can kind of see that it generates fluent english but it sometimes doesn't have the best tendency to tell you know the truth this is another example so again this is a prompt and
70:17 let me go to let me dig this up so this is kind of from the course syllabus oops so so this is some complicated expression and then this is again the what you feed into gp3 and it says artificial intelligence is the magic that makes computer do things that are not they're not supposed to do like talking their cars interesting okay so it's
70:50 kind of well you can judge for yourself what to make of this anyway you can if you have access you can burn quite a bit of time playing with this another example i'll show is you can train it on cs221 quizzes so you train it on quiz one question and answers and you see how well it does on quiz 2. so here are you know prompts in so what is bold is given to the
71:21 algorithm so this should be familiar from quiz one and then you ask it which of the following are examples of regression and it answers a c and d which is which is wrong but then it offers an explanation of this which is you know kind of starts off on on target and then it kind of examples of supervised regression house price estimation which is actually pretty good spam detection which contradicts what it answered
71:54 and you know this unsupervised regression i don't know if it's a thing so you can kind of get a sense that gpd3 is really good at generating text which if you just if i didn't tell you this was fake you'd probably read it and you know but you have to actually look pretty closely now to know that you know something is up
72:25 okay there are actually so i'm trying to give you a maybe a more balanced view of gp3 i think if you go on the internet and look at all the twitter you'll just be completely blown away by all the awesome things people are building out there but i want to kind of balance it out with some things which are like yeah maybe it's not doing everything but several other things to mention now that gender bias i think is very hot
72:49 on people's minds in nlp there's they have an entire section in the paper that calls out that yes not surprisingly gpg3 has gender bias i mentioned before that there was actually someone who used gpd3 to generate a blog post and this ended up you know number one on hacker news for a while and there's even papers coming out that says that gbd3 can primed with extremist context can generate more extremist content which is again perhaps not
73:21 surprising given that this was trained on the internet and the internet has many things lovely things in it so some kind of things to think about i think so one clear question here is can we make gpd3 more unbiased and lux less toxic certainly out of the box this is just kind of a wild tool that's kind of like you should not be using gpu directly to generate things that to show show people
73:54 another question is what what are the societal impacts of you know using text automatic generation you know which is kind of related even if it's not generally unbiased things but especially if it's generally unbiased things the obvious worry is that you know fake news is already a problem and this is just a massive amplifier for you know producing large amounts of credible sounding text which is just you know swamps out anything else anything anyone wants to say so that's
74:26 a potentially a pretty dangerous you know world on the kind of more scientific side i think there's still the outstanding question of you know can we achieve language understanding without a model of the world or without you know world experience so gpd3 is kind of only trained on text and furthermore it has no it's just a large transformer model which has no internal structure and you know one question is can it be said to you know understand language or does it
74:58 need you know the grounding experience so if there were more time i would actually do a poll and get people to discuss this because i think it's actually not obvious what the answer is maybe i'll end with final remarks and then i'll take questions so one thing i want to just you know starting at the top i want to highlight that language is just incredibly rich complex communication system you know quote-unquote from x-ray cd glorif glorious chaos
75:30 but yeah which is i think you know fascinating to study at the same time there's a lot of regularity and structure there has to be because i think otherwise we wouldn't be able to productively coordinate and use language as systematically as we have been able to current models kind of you know ignore it which is you know but it's also unclear how to incorporate this information in a way that kind of matters as you can see the field is moving at
75:57 an incredible pace right you know gb3 was on 2 20 20. bird was only two years ago c2c was you know six years ago and only in the last you know six years nlp has been completely you know transformed i would say even in the last two years it's been completely transformed so it's interesting what will happen in the next year and i think there's even more i think urgency to kind of understand what these models are doing
76:24 and make sure that the technology is kind of directed in the right way and then of course there's a lot more to be said and learn from this there are wonderful classes at stanford that you can take 124 to 24 to learn more about nlp so with that i will take any questions let's see where is okay is voice language a better source
76:54 than text language so this is kind of spoken audio it depends what you mean by better so voice does contain prosodic cues which contain more information than what's just in the text unfortunately there's not that much of it compared to all the texts that's on the web i think a lot of the issues of language understanding can be somewhat factored out from you
77:25 know speech often people convert speech spoken language into written language and then you take it from there although there's definitely infrastructure in speech as well how do you feel understanding multiple languages is a plus of research you know how do you feel understanding of multiple languages is a plus of research in nlp so a lot of research in nlp is very english centric which
77:57 you know is you know i think a potential problem if you think about fairness right i think many rare language and low resources languages as they're called don't enjoy such high accuracies and those are probably the people who kind of need it you know the most so there is a community of people in nlp who are very much interested in how to design more efficient learning algorithms that help you know low resource languages finally how has the study of nlp
78:27 helps linguistics grows as a field so this is a really interesting question unfortunately i won't have time to really do its justice chris potts is a linguist at stanford has lots of you know insightful opinions about this linguistics is interesting because it was for much of its history it was so so dominated by kind of formal you know grammars and semantics starting you know from chomsky in some ways that his
78:57 era led to this kind of whole feel being dominated by a certain kind of way you know of thinking and it's almost that some of these other perspectives hasn't really had a chance to you know to breathe i think this is in a lot of the still the this formal semantics is still very much kind of in the logical tradition where you know you have sentences which are you look at carefully and you try to intuit what might be going on it's a
79:26 very different approach than just looking at a broad corpus and trying to make you know sense of you know what's there so but this is starting to change a bit i think at least people are thinking about how these types of models that we're developing in machinery could be useful i think that you know unfortunately there's it's it's a hard ask because in some ways the
79:57 these deep learning models don't give you more much more than existence proof of certain types of data leading to certain types of behavior and it doesn't really give you necessary insight into you know language you know itself because interpretability is kind of a missing you know component but i think this is a really great question and interesting to ask like how these models can actually help us understand language better help us understand language better not help you
80:26 know computer exhibit better language understanding can we alter and add to the objective function of modern models that makes them more logical and coherent yeah so this is a kind of a natural direction that a lot of people are thinking about so why can't you have both you can have you know the richness of modern neural models plus some logical and there's been a bunch of work that tries to fuse the two you know together
80:57 personally i feel like this is far from you know being solved by just kind of simple combination because i think there's something maybe deeper about you know how things should be kind of structured like you can add a regularizer that makes burr to kind of be more consistent for gpd3 that make these more consistent but i think the problem is that why are we using these models in the first place is that they can't be captured by you
81:26 know logical regularities and the many of the advantages that we're getting from them are in places where logic kind of fails to you know deliver so in those areas i think we don't have much of an option we don't have the option of slapping on a logical regulizer because otherwise we would have just built a logical thing how far is real-time spoken communication in multiple languages
81:57 so this is like simultaneous translation so if i am on skype and i speak in english and it comes out in japanese or something this is actually i would say that this is i mean not mature technology by any means right now but it's it's coming and i think there's been work first of all speech recognition i think is you know getting really really good then the other two main challenges in in machine translation what makes
82:27 real time difficult is that you know word order in languages differ so you can't translate word to word so you have to wait a little bit to get enough context and then translate it you know and so forth and but there are models on nlp that you know try to do this and i think you can do a lot by kind of doing you know predictive predicting kind of essentially what you're you're going to say so
82:54 i think that a lot of this can be done without even you know deep deep understanding of you know language what i think we've learned from translation is that you know while translation is getting 90 of the way with translation doesn't really require any understanding of language it's just kind of matching kind of symbols contextually i think getting the remaining bit and having translations that you can actually you know trust and are nuanced then proper i think is going to require
83:26 you know quite a bit more work okay so is there okay another question are there examples of building languages with rrl it making advantages communicate a multi in a agent environment a multi-engine environment so there is a bunch of work on what is called kind of multi-agent communication where people set up some sort of environment and let a bunch of rl agents
83:57 train a bunch of our agents to act in this environment where one of the actions is talk right so this is an interesting experiment where you can actually get certain types of languages to kind of evolve from this procedure and and language does help you play the game or you know solve the environment better but it's rare that these languages just
84:29 automatically line up with our notions of natural languages because often these worlds are too kind of limited for kind of language to be for it to be necessary to take on that kind of richness and also i think just that there's no pressure that these have you know their human language might not is probably not optimal for
84:59 you know for anything it's just what we have and what we kind of happen to evolve so but that's a good question what are major problems society that advancing nlp will solve versus what problems they may create so there are a bunch of places where nlp can be used for societal you know good so the one thing that it can do in principle is to allow a broader set of people to
85:30 especially with you know different you know people who might not speak english to be able to kind of tap into you know english english resources and breaking down the kind of multilingual barriers it's also been useful for doing kind of analysis on studying how people talk so dandrovsky and others have a project on see analyzing how language of police officers and stops compares depending on whether they're
86:02 stopping you know a black person or a white person and this could be and using nlp techniques to try to study that question so i think one huge area is language is used in a kind of all in a social and societal context and you know therefore building tools that help us manage and navigate that you know societal can landscape can be really interesting problems it can cause certainly you know fake generations
86:34 biases and models if we start trusting you know translations or our systems it could lead to you know kind of amplification effects where you know kind of the the haves and have-nots kind of push farther you know apart all right well i think we're out of time so thanks everyone for coming and listening and have a good rest of the week
87:05 you