This report details how the class AddKnowledgeGraphReplicas database migration, introduced in Merge Request gitlab-org/gitlab!191162 ("Add knowledge graph tables")+ , integrates with the larger Zoekt architecture and how GitLab leverages its existing Zoekt infrastructure for the new Knowledge Graph feature.

Introduction

The integration of Knowledge Graph functionality into GitLab aims to reuse the existing, robust Zoekt infrastructure. Merge Request gitlab-org/gitlab!191162+ lays a crucial part of the groundwork for this by introducing new database tables and models. The AddKnowledgeGraphReplicas class, specifically, is a database migration within this MR that establishes the foundational schema for managing Knowledge Graph instances (replicas) on Zoekt nodes.

What is AddKnowledgeGraphReplicas in MR gitlab-org/gitlab!191162?

In the context of MR gitlab-org/gitlab!191162+, AddKnowledgeGraphReplicas is the name of a database migration class (db/migrate/20250512113325_add_knowledge_graph_replicas.rb). Its primary function is to create the p_knowledge_graph_replicas table in the GitLab database. This table is designed to store metadata about each instance (replica) of a knowledge graph.

Key details of the p_knowledge_graph_replicas table created by this migration:

• Purpose: To track individual instances of knowledge graphs and their association with specific Zoekt nodes where they are hosted.
• Partitioning: The table is partitioned by namespace_id for better scalability and management.
• Crucial Columns:
  • id: Primary key component.
  • namespace_id: Links the replica to a GitLab namespace and is used for partitioning.
  • knowledge_graph_enabled_namespace_id: A foreign key to p_knowledge_graph_enabled_namespaces, linking the replica to its parent knowledge graph configuration.
  • zoekt_node_id: A critical foreign key that links the knowledge graph replica directly to an existing record in the zoekt_nodes table (representing a Search::Zoekt::Node). This is the primary mechanism for Zoekt integration.
  • state: Indicates the current status of the replica (e.g., pending).
  • retries_left: For managing operational retries.

Code Snippet: AddKnowledgeGraphReplicas Migration

Loading syntax highlighting...

The Ai::KnowledgeGraph::Replica Model

Corresponding to the p_knowledge_graph_replicas table, MR gitlab-org/gitlab!191162+ also introduces the Ai::KnowledgeGraph::Replica ActiveRecord model (ee/app/models/ai/knowledge_graph/replica.rb). This model defines the application-level representation of a knowledge graph replica.

Key aspects of the Ai::KnowledgeGraph::Replica model:

• Table Mapping: It maps to the p_knowledge_graph_replicas table.
• Associations:
  • belongs_to :zoekt_node, class_name: '::Search::Zoekt::Node': This is the most important association for Zoekt integration. It establishes a direct relationship from a knowledge graph replica to the Zoekt node (Search::Zoekt::Node) on which it resides, using the zoekt_node_id foreign key.
  • belongs_to :knowledge_graph_enabled_namespace: Links the replica to the overall knowledge graph configuration for a namespace.

Code Snippet: Ai::KnowledgeGraph::Replica Model

Loading syntax highlighting...

Strategic Context: Reusing Zoekt Infrastructure

The overarching strategy, as outlined in epics like gitlab-org&17767 ("Graph node management")+, is to leverage the existing Zoekt infrastructure for deploying and managing Knowledge Graphs. This avoids building a parallel system and capitalizes on the mature capabilities of Zoekt.

As stated in the description of Epic gitlab-org&17767+:

We will use existing Zoekt nodes to index, store and serve also graph DBs. The major benefit is that we can re-use existing Zoekt logic (nodes management on Rails side) and infrastructure (deployment of Zoekt nodes) and node logic itself (Zoekt Webservice and Indexer).

The MR description for gitlab-org/gitlab!191162+ echoes this:

• KnowledgeGraph::Replica - represents an instance of knowledge graph on a particular zoekt node. The plan is to deploy knowledge graphs on Zoekt nodes. To assure high availability, we will keep >1 replica for each graph.

How AddKnowledgeGraphReplicas Enables Zoekt Integration

The AddKnowledgeGraphReplicas migration is a cornerstone of this integration strategy. By creating the p_knowledge_graph_replicas table with the zoekt_node_id foreign key, it establishes a direct database-level link between a knowledge graph instance and a specific Zoekt node.

This linkage is further solidified in the application layer by:

1. The belongs_to :zoekt_node association in the Ai::KnowledgeGraph::Replica model.
2. A corresponding has_many :knowledge_graph_replicas association added to the existing Search::Zoekt::Node model (ee/app/models/search/zoekt/node.rb). This allows a Zoekt node to be aware of all knowledge graph replicas it hosts.

Code Snippet: Modification to Search::Zoekt::Node Model

Loading syntax highlighting...

This database schema and model relationship means that GitLab's application logic can:

• Identify which Zoekt node hosts a particular knowledge graph replica.
• List all knowledge graph replicas hosted on a given Zoekt node.
• Distribute knowledge graph replicas across multiple Zoekt nodes for high availability and load balancing.

Leveraging Existing Zoekt Infrastructure - The Benefits

By establishing this link, GitLab can leverage its existing Zoekt infrastructure in several significant ways:

1. Node Management: The existing Rails-side logic for managing Zoekt nodes (e.g., tracking their status, capacity, and assigning tasks) can be extended or adapted to also manage the lifecycle of knowledge graph replicas on these nodes.
2. Deployment and Operations: Knowledge graph components (like KuzuDB files, as mentioned in Epic gitlab-org&17518 ("Knowledge Graph Server")+ ) can be deployed onto the same physical or virtual machines already provisioned and managed as Zoekt nodes. This simplifies deployment, scaling, and operational procedures.
3. High Availability: As mentioned in the MR description, the plan is to have multiple replicas. By associating these KnowledgeGraph::Replica records with different zoekt_node_ids, GitLab can distribute these replicas across various Zoekt nodes, enhancing fault tolerance.
4. Resource Allocation & Monitoring: The zoekt_nodes table already tracks resource information. While MR gitlab-org/gitlab!191162+ doesn't detail how knowledge graph storage is specifically accounted for, the association provides a pathway for future integration with Zoekt's resource management and monitoring capabilities.
5. Unified Infrastructure: It avoids the complexity and cost of setting up and maintaining a separate infrastructure stack specifically for the Knowledge Graph feature.

The Bigger Picture: Knowledge Graphs on Zoekt Nodes

The p_knowledge_graph_replicas table created by the AddKnowledgeGraphReplicas migration stores metadata about where knowledge graph instances are located. The actual knowledge graph data (e.g., KuzuDB database files) will reside on the file systems of these designated Zoekt nodes.

Other components and services will use this metadata:

• Indexing/Creation Workers: To know which Zoekt node to target when creating or updating a knowledge graph instance.
• Query Services: An extended gitlab-zoekt service (running on the Zoekt nodes) or a Rails-based service will use this metadata to locate the correct KuzuDB file on the appropriate Zoekt node to serve queries. Epic gitlab-org&17518+ mentions extending the gitlab-zoekt service to provide an API for querying these graph databases.

The AddKnowledgeGraphReplicas migration is thus a foundational step, enabling the system to track and manage the distributed instances of knowledge graphs across the Zoekt node fleet.

Conclusion

The class AddKnowledgeGraphReplicas in Merge Request gitlab-org/gitlab!191162+ is a database migration that plays a pivotal role in integrating GitLab's Knowledge Graph feature with its existing Zoekt architecture. By creating the p_knowledge_graph_replicas table and establishing a foreign key relationship (zoekt_node_id) to the zoekt_nodes table, this migration enables GitLab to:

• Treat knowledge graph instances as "replicas" hosted on Zoekt nodes.
• Leverage Zoekt's established node management, deployment, and operational framework.
• Avoid the need for a separate infrastructure for the Knowledge Graph feature.

This database schema change is a fundamental prerequisite for the broader strategy of using Zoekt nodes to store, serve, and manage knowledge graph data, thereby enhancing scalability, availability, and operational efficiency.