.custom File Format

The custom format (JSON format ending with the .custom extension and matching a particular schema) is a specific format used for full control over the ingestion process. This means that all chunking must be performed by the caller, and metadata must be assigned accordingly. The minimal set of values is as follows:

[] // List if items

Each item is as shown below:

{
       "pageContent": "some text", // mandatory
       "metadata": // mandatory
       { 
           "name": "name", // mandatory
           "description": "some description", // mandatory
           "source": "absolute URL" // mandatory
           // key value for extra metadata (optional)
       }
}

Once the document is uploaded, the ingestion process will skip the chunking process based on the text of the file; all related information must be provided in the .custom file. Each chunk to be ingested and stored in the vector store will use the pageContent field with the associated metadata.

Note that if you are using extra metadata elements, you must fill in the associated key/value grid or use the API to upload them accordingly. In addition, since you control the pageContent value, make sure the input does not exceed the maximum value for the selected embeddings model.

View a sample at RAG Custom File Format sample.

See Also

.web File Format