How to create a knowledge base

Creating a knowledge base is a crucial step in developing effective AI agents with Gaia. This page outlines various methods to build vector collections from your proprietary and private knowledge, ensuring your AI has access to the most relevant information. Whether you're working with plain text, markdown files, structured data pairs, PDFs, or web content. Gaia provides flexible options to import and transform your data into a powerful knowledge base. Each section below details a specific approach, allowing you to choose the method that best suits your data format and needs. By following these guides, you'll be able to create a rich, customized knowledge foundation for your AI applications.

📄️ Knowledge base from a plain text file

In this section, we will discuss how to create a vector collection snapshot from a plain text file. The

📄️ Knowledge base from a markdown file

In this section, we will discuss how to create a vector collection snapshot from a markdown file. The snapshot file can then be loaded by a Gaia node as its knowledge base.

📄️ Knowledge base from source / summary pairs

In this section, we will discuss how to create a vector collection snapshot for optimal retrieval of long-form text documents. The approach is to create two columns of text in a CSV file.

📄️ Knowledge base from a PDF file

In this section, we will discuss how to create a vector collection snapshot from a PDF file. First, we will parse the unstructured PDF file to a structured markdown file. Then, we will follow the steps from Knowledge base from a markdown file to create embedding for your PDF files.

📄️ Knowledge base from a URL

In this section, we will discuss how to create a vector collection snapshot from a Web URL. First, we will parse the URL to a structured markdown file. Then, we will follow the steps from Knowledge base from a markdown file to create embedding for your URL.