File attachments with examples datasets in LangSmith

0:01 hey everyone this is aners from L chain I wanted to give you a quick overview of a new feature that we launched that allows you to associate file attachments with examples that you have in your Langs Smith data sets this allows you to more easily run multimodal evaluations and it also allows you to see and edit those attachments very easily alongside the examples in the UI so to kick things off I'll talk about how to populate your lsmith data

0:29 sets with exam with examples that have file attachments associated with them and the first way that I'll outline is by adding the inputs outputs and attachments or propagating the inputs and outputs and attachments from a run that you've traced to an example so as some of you might recall we actually launched support for tracing with attachments a couple of months ago so this is this is nothing new but

1:00 I'll I'll demo this just to show you how you can get a trace with file attachments associated with it so what I've done here is I'm using the python SDK the lenss Smith python SDK and I've wrapped a very simple function with the traceable decorator this function takes in a couple of questions a question about the audio that's passed in a question about the image that's passed in and it returns you know a dummy output in realistic scenarios you might

1:29 be prop propagating these to an llm asking about them or or maybe even like some larger compound AI system so what I've done is I've loaded a few attachments for my local file system or files for my local file system and I am sending them as attachments to this Trace with attachments function so let me do that right now and then I will go into my lsmith project and as you can see a

2:01 new Trace popped up this Trace has the input it has the output and it also has all the attachments that I sent up along with this particular run so I can see the Audio I can see the image I can see the PDF and I can also look at you know the video this is a bit meta it's actually a video of a screen recording of lsmith so now let's say that I you know I really like the trace or it's a

2:31 very interesting you know run to me and I want to add it as an example to a data set what I can do is Click add to data set in this case I'll create a new data set change the date here just to make it unique I will click create and then one thing that I can do here is I can choose which attachments to propagate over to the example so so I'll keep the inputs and

3:00 outputs as as is these are fully editable but I actually don't want to change them in this case I'll propagate over the image and I'll propagate over the audio so I'll get rid of the PDF in the video so now I'll click submit and now we can view the data set and look at the example that we just added great so another thing that I can do is I can edit the attachments as well I can add you know I can upload new

3:30 files I can also rename attachments in this case I'll rename this to my wave and I'll rename this to my image okay so now I'll click submit to propagate this update and one thing that I'll point out is you can actually view earlier versions of your example as well so we have data set versioning and this versioning also works with file attachment so this is the old version of my of

4:03 my example that had audio and image as as the attachment names but then as as you know I've updated this to have new names for the attachments okay great so in addition to adding examples to a data set from from runs you can also upload a new example and upload files in this in this pane

4:34 that opens up you can also use the SDK and you can also consume these attachments when you run evals this is what I'll demo next so what I'll do is I'll copy this data set ID because I'm going to use it in the second portion of my notebook I'm going to get rid of this data set ID and use this one and what I'm doing here is using the Langs Smith python SDK to upload a new example with new

5:03 inputs and new outputs although you know they're in the same shape as my existing example and I'm uploading some new attachments here so I'm uploading a new a wave file and a new PDF file I'm actually using publicly available test files and I'm going to kick this off and I should see another example show up in lsmith so this now has audio question image question audio

5:34 answer image answer my PDF my image my wave file I can view the attachments here let me go back to this one there my earlier one and now what I'll do is I'll Define a task or Target that I want to evaluate so what I'm doing here is I'm defining a Target function that can actually consume these attachments and the way to do this is to take in two arguments in your target so you want to

6:04 take in inputs and you also want to take in an argument called attachments this works fairly similarly in the JS SDK but here I'm demoing the python SDK so once I have this this this function defined I can you know I'll could just quickly walk through what it's doing it's taking the audio and sending it to GPT 40 audio preview by Bas 64 encoding it and it's also

6:35 propagating the image URL that's associated with the attachment the example attachment to the model this is actually one advantage of using file attachments most most models most most model providers support taking in an image URL directly in addition to base 64 content and so what you can do is use the use the attachments object

7:06 that we pass in extract the attachments that you want to use and choose to use a reader or choose to use the URL they're both valid you know in this case I'm actually reading the the contents from the reader for the audio and Basics coding it but here for the for the image completion I'm just passing or propagating the image URL great so what I'll do next is Define

7:40 an evaluator that judges whether the image is whether the image description that was produced makes sense so here I'm using GPD 40 my task actually use GPD 4 mini and you know what I'm doing is using my attachments in the exact same way I'm going into these attachment this attachments object or this attachments

8:12 dictionary I'm looking at the image URL that I'm expecting and I'm pulling out the prend URL and and propagating that as well so now what I can do is run this evaluation and so this actually might take a little bit of time because it is you know running a lot of model IO okay so it looks like everything completed so now I can go back to my

8:44 data set I can look at the experiment that I just kicked off and for each row I can look at the inputs and outputs I can also look at the attachments associated with them and I can also you know expand this and show you the audio question the image question the reference output which was the audio answer image answer and the actual output that was produced by my

9:16 pipeline okay anyways I hope that this was helpful and sorry one thing I'll mention is you can also see the score that's associated with the output of the experiment this comes from the about valuator that that I ran that also used the attachment so anyways I hope this was helpful I'm really excited to see how people use our expanding support for files and multimodal content you know we think that the AI space is you

9:48 know expanding in use cases for for multimodal Content so please try it out and let us know if you have any feedback or or comments thank you so much bye