Maintaining a Vector Index: Storing, Updating, and Deleting Embeddings

Effectively setting up and maintaining a vector index is the backbone of any robust Retrieval-Augmented Generation (RAG) pipeline. While generating an initial set of embeddings is often straightforward, the real complexity lies in operational management: how do you handle document updates, how do you keep the index synchronized with a relational database, and how do you prevent outdated semantic data from generating irrelevant answers?

1. Fundamentals of Vector Storage and Metadata

Each document in a RAG architecture is split into smaller text blocks (chunks) and converted into numerical vectors using an embedding model. In addition to the vector itself (for example, an array of 1536 dimensions), storing associated metadata is crucial for filtering and traceability.

When designing your storage layer, you must take into account:

• Unique Identifier (ID): A deterministic UUID based on the document hash or source ID instead of auto-increment integers.
• Payload / Metadata: Contains at least the source title, chunk index, access rights, and a hash of the original text to prevent unnecessary re-indexing.
• Indexing Type: Choice between HNSW (Hierarchical Navigable Small World) for low latency or IVF (Inverted File) for larger datasets with lower memory usage.

2. Incremental Updates for New Documents

When a user adds a new document or modifies an existing one, you do not want to rebuild the entire index. This requires an incremental update strategy. To determine if a document has changed, compare the content hash before calling the embedding model's API.

Below is pseudocode demonstrating how to process and add new or modified documents to the index in a controlled manner:

// Pseudocode: Incremental update of the vector index

Function ProcessDocumentChange(document_id, new_text, metadata):
    new_hash = CalculateSHA256(new_text)
    existing_metadata = Database.GetMetadata(document_id)

    If existing_metadata exists AND existing_metadata.hash == new_hash:
        Print("No change detected. Skipping embedding.")
        Return Success

    // Clean up old chunks if it is an update
    If existing_metadata exists:
        DeleteExistingVectorChunks(document_id)

    // Split into manageable chunks
    chunks = SplitIntoChunks(new_text, max_tokens=500, overlap=50)
    new_vectors = []

    For each index, chunk in chunks:
        chunk_id = GenerateUniqueID(document_id, index)
        
        // Secure API call without hardcoded keys (environment variables)
        vector = CallEmbeddingAPI(chunk) 
        
        chunk_metadata = {
            "document_id": document_id,
            "chunk_index": index,
            "hash": new_hash,
            "text": chunk,
            ...metadata
        }
        
        new_vectors.Add({
            "id": chunk_id,
            "values": vector,
            "metadata": chunk_metadata
        })

    // Batch upsert to the vector database
    VectorDB.Upsert(new_vectors)
    Database.SaveMetadata(document_id, new_hash)
    Print("Document successfully indexed or updated.")

3. Cleaning Up and Deleting Outdated Data

When documents are deleted from the CMS or the source database, the corresponding vectors often remain in the vector index. This leads to 'hallucinations' or outdated citations in the generated answers of the LLM.

To prevent this, implement one of the following two patterns:

• Direct Deletion (Sync): Upon a `DELETE` action in the application, you immediately trigger a `delete_by_filter` or ID-based deletion in the vector database.
• Soft Deletes & Garbage Collection: Mark documents in the database as deleted. A periodic background task (cron job) cleans up all orphaned vectors weekly by comparing which IDs no longer exist in the primary source.

4. Version Control of Embedding Models

An often underestimated risk in production is upgrading the embedding model (e.g., from text-embedding-ada-002 to text-embedding-3-small). Vectors generated by different models reside in different mathematical vector spaces and cannot be compared with each other.

A robust version control scheme requires:

1. Index Naming: Link the model format directly to the index name (e.g., kb_docs_v3_embedding_3_small).
2. Parallel Migration: Build a shadow index in the background using the new model while the old index remains operational.
3. Switching: Only switch the routing layer once the shadow index is 100% populated and validated for retrieval quality.