Inconsistent Query Results Based on Output Fields Selection in Milvus Dashboard

题意：在Milvus仪表盘中基于输出字段选择的不一致查询结果

问题背景：

I'm experiencing an issue with the Milvus dashboard where the search results change based on the selected output fields.

我遇到了一个与 Milvus 仪表板相关的问题，即搜索结果会根据所选的输出字段而发生变化。

I'm working on a RAG project using text data converted into embeddings, stored in a Milvus collection with around 8000 elements. Last week, my retrieval results matched my expectations ("good" results), however, this week, the results have degraded ("bad" results).

我正在进行一个基于检索增强生成（RAG）的项目，该项目使用转换为嵌入的文本数据，这些数据存储在包含大约8000个元素的Milvus集合中。上周，我的检索结果与我的预期相符（“好”结果），但本周，结果却变差了（“差”结果）。

I found that when I exclude the embeddings_vector field from the output fields in the Milvus dashboard, I get the "good" results; Including the embeddings_vector field in the output changes the results to "bad".

我发现，当我在 Milvus 仪表板的输出字段中排除 embeddings_vector 字段时，我得到的是“好”的结果；但是，如果在输出中包含 embeddings_vector 字段，结果就会变为“差”。

I've attached two screenshots showing the difference in the results based on the selected output fields.

我已附上两张截图，展示了基于所选输出字段的结果差异。

Any ideas on what's causing this or how to fix it?        

关于是什么导致了这个问题或如何解决它，你有什么想法吗？

Environment:        应用环境

Python 3.11 pymilvus 2.3.2 llama_index 0.8.64

Thanks in advance!        预先感谢！

from llama_index.vector_stores import MilvusVectorStore
from llama_index import ServiceContext, VectorStoreIndex# Some other lines..# Setup for MilvusVectorStore and query execution
vector_store = MilvusVectorStore(uri=MILVUS_URI,token=MILVUS_API_KEY,collection_name=collection_name,embedding_field='embeddings_vector',doc_id_field='chunk_id',similarity_metric='IP',text_key='chunk_text')embed_model = get_embeddings()
service_context = ServiceContext.from_defaults(embed_model=embed_model, llm=llm)
index = VectorStoreIndex.from_vector_store(vector_store=vector_store, service_context=service_context)
query_engine = index.as_query_engine(similarity_top_k=5, streaming=True)rag_result = query_engine.query(prompt)

Here is the "good" result: "good" result And here is the "bad" result: "bad" result

这是“好”的结果：“好”的结果。而这是“坏”的结果：“坏”的结果。

问题解决：

I would like to suggest you to follow below considerations.

我想建议你遵循以下考虑因素。

• Ensure that your Milvus collection is correctly indexed. Indexing plays a crucial role in how search results are retrieved and ordered. If the index configuration has changed or is not optimized, it might affect the retrieval quality.

确保你的Milvus集合已正确索引。索引在检索和排序搜索结果方面起着至关重要的作用。如果索引配置已更改或未优化，则可能会影响检索质量。

• In your screenshots, the consistency level is set to "Bounded". Try experimenting with different consistency levels (e.g., "Strong" or "Eventually") to see if it impacts the results. Consistency settings can influence the real-time availability of the indexed data.

在你的截图中，一致性级别被设置为“Bounded”。尝试使用不同的一致性级别（例如，“Strong”或“Eventually”）进行实验，看看它是否会影响结果。一致性设置可以影响索引数据的实时可用性。

• Review the query parameters, especially the similarity_metric. Since you're using IP (Inner Product) as the similarity metric, ensure that your embedding vectors are normalized correctly. Inner Product search works best with normalized vectors.

检查查询参数，特别是similarity_metric。由于你使用IP（内积）作为相似度度量，请确保你的嵌入向量已正确归一化。内积搜索在归一化向量上效果最好。

• Verify that the embedding vectors are of consistent quality and scale. If there were changes in the embedding model or preprocessing steps, it could lead to variations in the search results.

验证嵌入向量的质量和规模是否一致。如果嵌入模型或预处理步骤发生了变化，那么搜索结果可能会出现差异。

• The inclusion of the embeddings_vector field in the output might affect the way Milvus scores and ranks the results. It's possible that returning the raw embeddings affects the internal ranking logic. Ensure that including this field does not inadvertently alter the search behavior.

在输出中包含embeddings_vector字段可能会影响Milvus对结果的评分和排名方式。返回原始嵌入可能会影响内部排名逻辑。请确保包含此字段不会无意中改变搜索行为。

• Check the Milvus server logs and performance metrics to identify any anomalies or changes in the search behavior. This might provide insights into why the results differ when the embeddings_vector field is included.

检查Milvus服务器日志和性能指标，以识别搜索行为中的任何异常或变化。这可能会提供关于为什么在包含embeddings_vector字段时结果会有所不同的见解。

• Ensure that there are no version mismatches between the client (pymilvus) and the Milvus server. Sometimes, discrepancies between versions can cause unexpected behavior.

确保客户端（pymilvus）和Milvus服务器之间不存在版本不匹配。有时，版本之间的差异可能会导致意外行为。

• As a last resort, try modifying your code to exclude the embeddings_vector field programmatically during retrieval and compare the results. This can help isolate whether the issue is indeed caused by including the embeddings in the output.

作为最后的手段，尝试修改您的代码，以便在检索时通过编程方式排除embeddings_vector字段，并比较结果。这有助于隔离问题是否确实是由在输出中包含嵌入项引起的。

• Please try out this code if it helps.        如果这段代码有帮助的话，请尝试一下