Build a Vector Database with Milvus & FAISS

What We’re Building

In this comprehensive tutorial, we will guide you through the process of setting up a scalable vector database using Milvus and FAISS. This system will serve as the backbone for AI-powered applications that require efficient semantic search and retrieval of high-dimensional data such as text, images, and audio.

The completed project will enable you to process queries by converting them into vector embeddings, performing similarity searches, and retrieving the most relevant results in milliseconds. This setup is ideal for applications like recommendation engines, anomaly detection, and real-time chatbots.

Setup and Installation

To begin, we need to install and configure Milvus, a popular open-source vector database, along with FAISS, a library for efficient similarity search and clustering of dense vectors. We will use Docker to simplify the deployment process.

docker pull milvusdb/milvus:v2.2.0
docker run -d --name milvus -p 19530:19530 milvusdb/milvus:v2.2.0

Next, install the Python client for Milvus and FAISS using pip:

pip install pymilvus==2.2.0
pip install faiss-cpu==1.13.0

We will also need to set up some environment variables to manage configurations:

export MILVUS_HOST=localhost
export MILVUS_PORT=19530

Step 1: Initializing the Milvus Client

The first step in our project is to initialize the Milvus client, which will allow us to interact with the vector database. This involves connecting to the Milvus server using the host and port specified in the environment variables.

from pymilvus import connections

# Initialize Milvus client
connections.connect("default", host="localhost", port="19530")

Here, we import the connections module from pymilvus and use the connect function to establish a connection to the Milvus server. The “default” argument is the alias for this connection, and it references the host and port where Milvus is running.

Step 2: Creating a Collection in Milvus

Next, we need to create a collection in Milvus to store our vector data. A collection in Milvus is similar to a table in a traditional database and is defined by a schema that specifies the dimensions of the vectors.

from pymilvus import Collection, FieldSchema, DataType

# Define schema for the collection
fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=128)
]

# Create collection
collection = Collection(name="example_collection", schema=fields)

In this step, we define a collection schema with two fields: an integer ID field that serves as the primary key and a 128-dimensional float vector field for storing embeddings. We then create the collection using this schema.

Step 3: Indexing Vectors with FAISS

To perform efficient similarity searches, we need to index our vectors using FAISS. FAISS provides various indexing methods; here, we’ll use the Inverted File Index (IVF) with Product Quantization (PQ) for optimal performance.

import faiss
import numpy as np

# Create random vectors for demonstration
vectors = np.random.rand(1000, 128).astype('float32')

# Create FAISS index
index = faiss.index_factory(128, "IVF10,PQ4")
faiss.normalize_L2(vectors)
index.train(vectors)
index.add(vectors)

In this code, we create a set of random 128-dimensional vectors and normalize them using FAISS’s L2 normalization. We then initialize an IVF10,PQ4 index, train it with our vectors, and add the vectors to the index.

Testing Your Implementation

To verify that our setup is working correctly, we can perform a simple query to find the nearest vectors to a given query vector. This test will check both the Milvus and FAISS integrations.

# Query vector
query_vector = np.random.rand(1, 128).astype('float32')
faiss.normalize_L2(query_vector)

# Search in FAISS index
D, I = index.search(query_vector, k=5)
print("Nearest neighbors:", I)

This test code generates a random query vector, normalizes it, and searches for the top 5 nearest neighbors in the FAISS index. The indices of the nearest vectors are printed as the output.

What to Build Next

Once you have a basic vector database setup, consider expanding your project with the following enhancements:

• Integrate metadata filtering in Milvus to retrieve contextually relevant results based on additional attributes like category or timestamp.
• Implement a web-based interface using Next.js to allow users to interact with your vector database through a modern frontend.
• Optimize performance by experimenting with different FAISS indexing strategies and parameter tuning for your specific dataset and use case.

In the context of the GCC and Middle East, such vector databases can be instrumental in sectors like oil and gas for predictive maintenance, in finance for fraud detection, and in smart city projects under initiatives like Saudi Vision 2030.