## embed-english-light-v3.0 介绍

embed-english-light-v3.0 是 embed-english-v3.0 的一个更小、更快的版本。它几乎同样能干，但速度更快。专为英文文本量身定制。
在 Embed V3 模型系列中比较所有嵌入模型。


![model-1.png](https://assets.zilliz.com/cms-cn/model_1_4f027c7df8.png)



- MTEB: 用于评估检索、分类和聚类的广泛数据集（56 个数据集）
- BEIR: 专注于跨域检索的数据集（14 个数据集）

## 如何使用 embed-english-light-v3.0 创建向量

生成向量有两种主要方式：
1. PyMilvus：Milvus 的 Python SDK，它无缝集成了 embed-english-light-v3.0 模型。
2. Cohere Python SDK：由 Cohere 提供的 Python SDK。
一旦生成了向量，它们可以存储在 Zilliz Cloud（一个由 Milvus 驱动的全托管向量数据库服务）中，并用于语义相似性搜索。以下是四个关键步骤：
1. 免费注册 Zilliz Cloud 账户。
2. 设置无服务器集群并获取公共端点和 API 密钥。
3. 创建 Collection 并插入您的向量。
4. 对存储的向量进行语义搜索。

#### 通过 PyMilvus 生成向量

即将推出使用 PyMilvus 进行文本编码和使用 Zilliz Cloud 进行向量相似性搜索的演示。
请参阅我们的 PyMilvus embedding模型文档，获取逐步指南。

#### 通过 Cohere Python SDK 生成向量

即将推出使用 Cohere Python SDK 进行文本编码和使用 Zilliz Cloud 进行向量相似性搜索的演示。
有关更多信息，请参考 Cohere 文档。

embed-english-light-v3.0 是 embed-english-v3.0 的一个更小、更快的版本。它几乎同样能干，但速度更快。专为英文文本量身定制。
在 Embed V3 模型系列中比较所有嵌入模型。

专有

Cohere-embed-english-light-v3.0模型-生成向量

文本

Embedding

### Cohere rerank-english-v2.0 简介


rerank-english-v2.0 是一个专门用于提高搜索和检索系统准确性的重排模型，能够根据相关性细化排名结果，可用于处理上下文长度为 512 个 Token 的英文文档。

更多重排和 Reranker 相关详情，请阅读博客：
- [What Are Rerankers and How They Enhance Information Retrieval](https://zilliz.com/learn/what-are-rerankers-enhance-information-retrieval)
- [Optimizing RAG with Rerankers: The Role and Trade-offs](https://zilliz.com/learn/optimize-rag-with-rerankers-the-role-and-tradeoffs)


### 如何使用 rerank-english-v2.0 对结果进行重排


主要有两种方式对搜索结果进行重排：

- [PyMilvus](https://github.com/milvus-io/pymilvus)：[Milvus](https://zilliz.com/what-is-milvus) 的 Python SDK，无缝集成了`rerank-english-v2.0` 模型
- Cohere 的 API


#### 通过 PyMilvus 进行重排

```
from pymilvus.model.reranker import CohereRerankFunction
rf = CohereRerankFunction(
    model_name="rerank-english-v2.0",
    api_key=COHERE_API_KEY
)
query = "What event in 1956 marked the official birth of artificial intelligence as a discipline?"
documents = ["In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.","The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.","In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.","The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems."
]
results = rf(
    query=query,
    documents=documents,
    top_k=4,
)
```

#### 通过 Cohere API 进行重排


```
import cohereco = cohere.Client("your-cohere-api-key")
query = "What event in 1956 marked the official birth of artificial intelligence as a discipline?"
documents = ["In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.","The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.","In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.","The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems."
]
results = co.rerank(query=query, documents=documents, top_n=4, model='rerank-english-v2.0')
```

- 更多详情，请参考[文档](https://docs.cohere.com/reference/rerank)
- 阅读 [Cohere 重排最佳实践](https://docs.cohere.com/docs/reranking-best-practices)

rerank-english-v2.0 是一个专门用于提高搜索和检索系统准确性的重排模型，能够根据相关性细化排名结果，可用于处理上下文长度为 512 个 Token 的英文文档。

Cohere / rerank-english-v2.0

Reranking

#### Instructor 模型家族简介


由 NKU NLP 推出的 Instructor 模型是一个经过指令微调的文本 Embedding 模型。它可以通过仅提供任务指令来创建特定任务的 Embedding 向量用于分类、检索、聚类、文本评估等，涵盖各种领域（如科学和金融），无需额外的微调。

![AImodelHKU.PNG](https://assets.zilliz.com/cms-cn/A_Imodel_HKU_715cc7e0f1.PNG)

图1: Instructor 模型工作原理 (图片来源：NKU NLP)

Instructor 模型提供 3 种变体：instructor-base、instructor-xl 和 instructor-large。每种的性能和可扩展性都有所不同，可满足多样化的需求。

#### instructor-base 简介


Instructor-base 是 Instructor 模型家族中最小的一种变体，旨在平衡效率和性能，适用于追求高质量文本 Embedding 但需要控制计算开销的任务。
instructor-base、instructor-xl 和 instructor-large 对比

![screenshot-20241009-105254.png](https://assets.zilliz.com/cms-cn/screenshot_20241009_105254_928888ca25.png)


#### 如何使用 instructor-base 生成 Embedding 向量

生成 Embedding 向量的主要有两种方式：
- PyMilvus：Milvus 的 Python SDK，无缝集成了 instructor-base 模型
- Instructor 库:Python 的 InstructorEmbedding 库

生成 Embedding 向量后，就可以将向量存储在 Zilliz Cloud（全托管的 Milvus 向量数据库服务）中，用于语义相似性搜索。以下是四个关键步骤：

- 免费注册 Zilliz Cloud 账号
- 创建 Serverless 集群并获取公共 Endpoint 和 API 密钥
- 创建 Collection 并插入向量数据
- 对存储的 Embedding 向量进行语义搜索

#### 通过 PyMilvus 生成 Embedding 向量

```
from pymilvus.model.dense import InstructorEmbeddingFunction
from pymilvus import MilvusClient

ef = InstructorEmbeddingFunction(
    "hkunlp/instructor-base", 
    query_instruction="Represent the Wikipedia question for retrieving supporting documents:", 
    doc_instruction="Represent the Wikipedia document for retrieval:")

docs = [
   "Artificial intelligence was founded as an academic discipline in 1956.",
   "Alan Turing was the first person to conduct substantial research in AI.",
   "Born in Maida Vale, London, Turing was raised in southern England."
]
# Generate embeddings for documents
docs_embeddings = ef.encode_documents(docs)

queries = ["When was artificial intelligence founded",
          "Where was Alan Turing born?"]

# Generate embeddings for queries
query_embeddings = ef.encode_queries(queries)

client = MilvusClient(
    uri=ZILLIZ_PUBLIC_ENDPOINT,
    token=ZILLIZ_API_KEY)
    
COLLECTION = "documents"
if client.has_collection(collection_name=COLLECTION):
    client.drop_collection(collection_name=COLLECTION)
client.create_collection(
    collection_name=COLLECTION,
    dimension=ef.dim,
    auto_id=True)

for doc, embedding in zip(docs, docs_embeddings):
    client.insert(COLLECTION, {"text": doc, "vector": embedding})

results = client.search(
    collection_name=COLLECTION,
    data=query_embeddings,
    consistency_level="Strong",
    output_fields=["text"])
```


#### 通过 InstructorEmbedding 库生成 Embedding 向量

```
from InstructorEmbedding import INSTRUCTOR
from pymilvus import MilvusClient

model = INSTRUCTOR('hkunlp/instructor-base')

docs = [["Represent the Wikipedia document for retrieval: ", "Artificial intelligence was founded as an academic discipline in 1956."],
        ["Represent the Wikipedia document for retrieval: ", "Alan Turing was the first person to conduct substantial research in AI."],
        ["Represent the Wikipedia document for retrieval: ", "Born in Maida Vale, London, Turing was raised in southern England."]]
# Generate embeddings for documents
docs_embeddings = model.encode(docs, normalize_embeddings=True)


queries = [["Represent the Wikipedia question for retrieving supporting documents: ", "When was artificial intelligence founded"],
           ["Represent the Wikipedia question for retrieving supporting documents: ", "Where was Alan Turing born?"]]
# Generate embeddings for queries
query_embeddings = model.encode(queries, normalize_embeddings=True)

# Connect to Zilliz Cloud with Public Endpoint and API Key
client = MilvusClient(
    uri=ZILLIZ_PUBLIC_ENDPOINT,
    token=ZILLIZ_API_KEY)

COLLECTION = "documents"
if client.has_collection(collection_name=COLLECTION):
    client.drop_collection(collection_name=COLLECTION)
client.create_collection(
    collection_name=COLLECTION,
    dimension=768,
    auto_id=True)

for doc, embedding in zip(docs, docs_embeddings):
    client.insert(COLLECTION, {"text": doc, "vector": embedding})
    
results = client.search(
    collection_name=COLLECTION,
    data=query_embeddings,
    consistency_level="Strong",
    output_fields=["text"])
```

由 NKU NLP 推出的 Instructor 模型是一个经过指令微调的文本 Embedding 模型。它可以通过仅提供任务指令来创建特定任务的 Embedding 向量用于分类、检索、聚类、文本评估等，涵盖各种领域（如科学和金融），无需额外的微调。


Apache 2.0

HKU NLP / instructor-base

### Jina Embedding v2 模型简介


Jina Embeddings v2 模型由 Jina AI 开发，旨在处理长文档，并扩展了最大输入 Token 数量，支持高达 8,192 个 Token。截至 2024 年 8 月，Jina AI Embedding V2 提供四种变体，每种都满足不同的 Embedding 需求：

- [jina-embeddings-v2-small-en](https://zilliz.com/ai-models/jina-embeddings-v2-small-en)
- [jina-embeddings-v2-base-en](https://zilliz.com/ai-models/jina-embeddings-v2-base-en)
- jina-embeddings-v2-base-zh
- jina-embeddings-v2-base-de


### jina-embeddings-v2-small-en 简介


jina-embeddings-v2-small-en 是一款专用于处理英语单语种的 Embedding 模型，支持长达 8192 个 Token 。它是 Jina Embeddings v2 系列中最小的变体，经过训练，参数量为 3300 万，最终会生成 512 维的 Embedding 向量。

下表为 jina-embeddings-v2-small-en 和其他 Jina Embedding 模型的对比。


![屏幕截图 2024-10-09 111616.png](https://assets.zilliz.com/cms-cn/2024_10_09_111616_aaa2a9da47.png)




### 如何使用 jina-embeddings-v2-small-en 生成 Embedding 向量


生成 Embedding 向量的主要有两种方式：

- [PyMilvus](https://github.com/milvus-io/pymilvus)：[Milvus](https://zilliz.com.cn/what-is-milvus) 的 Python SDK，无缝集成了 jina-embeddings-v2-small-en 模型
- SentenceTransformer 库：Python 的 sentence-transformer 库

生成 Embedding 向量后，就可以将向量存储在 [Zilliz Cloud](https://zilliz.com.cn/cloud)（全托管的 Milvus 向量数据库服务）中，用于[语义相似性搜索](https://zilliz.com.cn/glossary/%E8%AF%AD%E4%B9%89%E6%90%9C%E7%B4%A2-%EF%BC%88semantic-search%EF%BC%89)。以下是四个关键步骤：

- 免费[注册 ](https://cloud.zilliz.com.cn/signup)Zilliz Cloud 账号
- [创建 Serverless 集群](https://docs.zilliz.com.cn/docs/create-cluster#create-a-serverless-cluster)并获取[公共 Endpoint 和 API 密钥](https://docs.zilliz.com.cn/docs/on-zilliz-cloud-console#serverless-cluster-details)
- 创建 Collection 并插入向量数据
- 对存储的 Embedding 向量进行语义搜索


#### 通过 PyMilvus 生成 Embedding 向量


```
from pymilvus.model.dense import SentenceTransformerEmbeddingFunction
from pymilvus import MilvusClient
ef = SentenceTransformerEmbeddingFunction("jinaai/jina-embeddings-v2-small-en", trust_remote_code=True)
docs = ["Artificial intelligence was founded as an academic discipline in 1956.","Alan Turing was the first person to conduct substantial research in AI.","Born in Maida Vale, London, Turing was raised in southern England."
]
Generate embeddings for documents
docs_embeddings = ef(docs)
queries = ["When was artificial intelligence founded","Where was Alan Turing born?"]
Generate embeddings for queries
query_embeddings = ef(queries)

Connect to Zilliz Cloud with Public Endpoint and API Key
client = MilvusClient(
    uri=ZILLIZ_PUBLIC_ENDPOINT,
    token=ZILLIZ_API_KEY)
COLLECTION = "documents"if client.has_collection(collection_name=COLLECTION):
    client.drop_collection(collection_name=COLLECTION)
client.create_collection(
    collection_name=COLLECTION,
    dimension=ef.dim,
    auto_id=True)

for doc, embedding in zip(docs, docs_embeddings):
    client.insert(COLLECTION, {"text": doc, "vector": embedding})
    
results = client.search(
    collection_name=COLLECTION,
    data=query_embeddings,
    consistency_level="Strong",
    output_fields=["text"])
```

更多详情，请参考 [PyMilvus Embedding 模型文档](https://milvus.io/docs/embeddings.md)。


#### 通过 sentence-transformer 生成 Embedding 向量


```
from sentence_transformers import SentenceTransformer
from pymilvus import MilvusClient
model = SentenceTransformer("jinaai/jina-embeddings-v2-small-en", trust_remote_code=True)
docs = ["Artificial intelligence was founded as an academic discipline in 1956.","Alan Turing was the first person to conduct substantial research in AI.","Born in Maida Vale, London, Turing was raised in southern England."
]
Generate embeddings for documents
docs_embeddings = model.encode(docs, normalize_embeddings=True)
queries = ["query: When was artificial intelligence founded","query: Wo wurde Alan Turing geboren?" ] 
Generate embeddings for queries
query_embeddings = model.encode(queries, normalize_embeddings=True)

Connect to Zilliz Cloud with Public Endpoint and API Key
client = MilvusClient(
    uri=ZILLIZ_PUBLIC_ENDPOINT,
    token=ZILLIZ_API_KEY)
COLLECTION = "documents"if client.has_collection(collection_name=COLLECTION):
    client.drop_collection(collection_name=COLLECTION)
client.create_collection(
    collection_name=COLLECTION,
    dimension=512,
    auto_id=True)

for doc, embedding in zip(docs, docs_embeddings):
    client.insert(COLLECTION, {"text": doc, "vector": embedding})
    
results = client.search(
    collection_name=COLLECTION,
    data=query_embeddings,
    consistency_level="Strong",
    output_fields=["text"])
```

更多详情，请参考 [HuggingFace 文档](https://huggingface.co/jinaai/jina-embeddings-v2-small-en)。


Jina Embeddings v2 模型由 Jina AI 开发，旨在处理长文档，并扩展了最大输入 Token 数量，支持高达 8,192 个 Token。截至 2024 年 8 月，Jina AI Embedding V2 提供四种变体，每种都满足不同的 Embedding 需求：

AI 模型助力搭建 GenAI 应用

如何选择合适的 Embedding 模型

主流 AI 模型

embed-english-light-v3.0模型

rerank-english-v2.0

instructor-base

jina-embeddings-v2-small-en