Skip to main content

Milvus Semantic Search Analysis

Overview​

Milvus is an open source vector database for multi-modal AI applications. More information on Milvus can be found at https://milvus.io/.

About Qarbine​

Qarbine is a unique Modern Data Collaboration Suite™ with powerful Milvus and Gen AI integrations. It enables teams across an enterprise to gain insights based on their Milvus and Gen AI investments. Its suite of over 10 integrated tools improves everyone’s productivity no matter their role or skill level. Qarbine’s functionality is based on previous analytic reporting solutions which earned most of the Fortune 500 and Wall Street as customers.

General Qarbine Execution Flow​

The high-level flow is depicted below.

  

Qarbine Prompts provide a way to obtain runtime values for use by data sources and templates. Prompts are defined in a no code manner using the Prompt Designer which supports a large variety of widgets.

A Data Source is a component responsible for retrieving data. It can use variables supplied by applications or Qarbine prompts to format the underlying query.

A Template defines how to process the data being retrieved from Data Source queries. It also defines formulas, formatting options, and other analysis and presentation options. The Template Designer combines the formatting power of Microsoft Word, with Excel’s formula features, plus PowerPoint’s layout mechanisms.

All of these components are stored in the Qarbine catalog to promote skill sharing and modern data, AI driven insights 24x7.

Enhanced Milvus Querying Options​

Qarbine queries to Milvus use version 1 of the Milvus REST API. The full expressive power of Milvus is available in 3 forms:

  1. JSON specification,
  1. SQL-like and
  1. hybrid of the two.

Qarbine’s unique provision for a SQL-like query experience greatly expands the use of Milvus beyond just coders. All Milvus features are accessible across the 3 query syntax forms. Developers and analysts can now easily leverage the power of Milvus for their daily activities
activities to interact with vector data and gain Gen AI driven insights.

Sample Data​

The following example uses the sample medium articles data referenced by the article at https://docs.zilliz.com/docs/example-dataset. A sample row is shown below.

  

Sample Semantic Search Result​

Below is a snapshot of a sample semantic search result generated by Qarbine which we will soon describe.

  

Qarbine has numerous options to produce publication quality output which is also interactive. In this example the link and copy images are interactive elements.

  

Click on    opens a browser tab on the medium article.
Clicking on   copies the medium article link to the clipboard.

Qarbine “custom cells” are used for the Milvus log and the star claps images.

The Milvus vector search distance is included in the output with 2 decimal places of precision.

When Qarbine is embedded into applications end users maintain context as they interact with the developer portion of the web page and the Qarbine generated one. It is a super productive experience that streamlines workflows and time to insights.

The output can be exported in a variety of formats including PDF. This is one of many ways Qarbine fosters modern data collaboration and the expansion of Milvus vector database usage.

Running the Sample Template​

A Template is designed to present row information and provide some interactive elements as well. It uses a Data Source and a Prompt as part of the flow and user experience.

A basic Qarbine prompt is used to obtain the phrase from the user.

  

The prompt sets a runtime variable named ‘@userPhrase’ which propagates along to the data retrieval stage. The referenced Data Source has the following query specification.

select * from medium_articles
where reading_time > 10
and nearText(@userPhrase, "myGoogleAI")
order by reading_time 
limit 3

It uses the convenient SQL-like syntax and includes a Qarbine provides post Milvus answer set sorting stage as well. These are unique Qarbine features that enhance the Milvus data analysis experience.

When run, Qarbine automatically interfaces with Google Gemini in this example to obtain the raw embedding value of 768 numbers for the nearText() phrase. That embedding is then used in the call to Milvus Cloud. The Milvus answer set is accepted and moved along the execution pathway.

The data is then processed based on the Template definition. The overall result is an interactive and publication quality analysis.

Getting Started​

When Qarbine is embedded into applications end users maintain context as they interact with the developer portion of the web page and the Qarbine generated one. It is a super productive experience that streamlines workflows and time to insights.

The output can be exported in a variety of formats including PDF. This is one of many ways Qarbine fosters modern data collaboration and the expansion of Milvus vector database usage.