Announcing our new Course: AI Red-Teaming and AI Safety Masterclass
Check it out →Vote- is a graph-based technique used to select exemplars that are diverse and representative from a dataset of unlabeled exemplars for annotation in a few-shot standard prompt.
It ranks exemplars based on their dissimilarity to others in the dataset, with the most diverse ones receiving the highest rank. In other words, the exemplars that are the least similar to others get more "votes" or are ranked higher.
The idea behind Vote- is that rather than annotating the entire training data, which is often costly and time-consuming, Vote- helps efficiently select the most informative exemplars for annotation that could be further used as exemplars in your few-shot prompt1.
To use Vote-, you first need a dataset of unlabeled exemplars related to your task.
Then, you apply Vote-K, a two-stage process:
Imagine having a bucket where you collect the unlabelled examplars. At first, your bucket is empty. But as you iterate during Stage 1, you gradually fill this bucket by picking diverse and representative examples.
Here are the steps:
Vote- computes vector representations of each unlabeled sample using Sentence-BERT, a model used to derive sentence embeddings that can be compared using cosine similarity.
Using the embeddings, a directed graph is constructed. In this graph:
Using the selected metric reflecting both diversity and representativeness, the algorithm scores each unchosen vertex based on the connections in the graph. The score for each vertex decreases if it is too close to already selected exemplars stored in your bucket, encouraging diversity in the selected set.
In each iteration, the vertex with the highest score is selected for annotation. The process continues until samples are selected, where is the total annotation budget.
Now that your bucket has some annotated examples, you focus on the remaining unlabeled instances to select the most challenging and diverse ones, avoiding the easiest examples.
As a result of these two stages, you have a complete annotated set containing exemplars:
You can now annotate these exemplars and plug them into your few-shot prompt!
You're an ecology student and want to understand climate change on a higher level by asking AI about it. You decide to use a few-shot prompt since you have a dataset of climate change-related questions, but you don't know which of the questions would be useful to use as exemplars and, frankly, you don't have time to sift through all of them. You decide to use Vote- to choose the exemplars for you. Your dataset is as follows:
Vote- will first compute vector representations of each question in your dataset using a model like Sentence-BERT. These vectors allow the questions to be compared for similarity. The algorithm will then create a graph connecting similar questions based on their cosine similarity. In this graph, each question will be a vertex, and edges will connect it to its nearest neighbors (the most similar questions).
Vote- will start selecting the most diverse exemplars based on this graph using the selected metric (some score). Remember, it's an iterative process and you want to pick an exemplars that’s not too similar to the ones you’ve already picked, so each time you calculate a score for each remaining exemplar and select the one with the highest score. For example, the first selected exemplars are:
Now that you've selected three exemplars, you now annotate them with the corresponding answers. In theory, this is supposed to be exemplars, but for the sake of simplicity and for the example we'll ignore that ( would have to be 30).
You now use these exemplars to train a model and have it make predictions on the rest of the unlabeled questions in your dataset.
From the rest of the unlabeled questions, Vote- will now choose questions where the model's confidence in its predictions is lower since that means they're harder and more unique. For example, the model could struggle with:
Vote- will continue selecting these exemplars, and once it's done, you annotate all of the ones it chooses.
Final Outcome:
By the end of both stages, Vote- has selected a total of exemplars:
These exemplars have all been annotated and are ready for use in your few-shot prompt.
Vote- prioritizes diversity when selecting exemplars, but it doesn't inherently evaluate whether the selected exemplars are relevant or useful for the task at hand. This means that while you might cover a wide range of topics, you might end up with exemplars that are too peripheral or not directly helpful for your specific task. Additionally, for very large datasets of exemplars, Vote- can be computationally expensive, and having to label the exemplars can be a pain, especially if you do it manually.
Vote- is a structured approach to finding good exemplars for your few-shot prompt based on the diversity of exemplars and being able to cover a wide range of topics as efficiently as possible. It's a powerful weapon to add to your exemplar-finding arsenal, especially if you find yourself with a dataset of exemplars that are unlabeled. It does have limitations, though; its focus on diversity can end up making it ignore quality or relevance, it can take up a lot of resources to do, and having to label the exemplars can be annoying.
Su, H., Kasai, J., Wu, C. H., Shi, W., Wang, T., Xin, J., Zhang, R., Ostendorf, M., Zettlemoyer, L., Smith, N. A., & Yu, T. (2022). Selective Annotation Makes Language Models Better Few-Shot Learners. https://arxiv.org/abs/2209.01975 ↩