File Support allows you to work with various file types, including documents, images, videos, and audio, to perform tasks like summarizing documents, extracting information, comparing files, and generating text based on file content. This opens up a world of possibilities for using Assistants or Agents in new and innovative ways.
The consumer should be able to interact with Assistants or Agents using files in the following ways:
- Summarize a document: Provide a concise summary of the document's content.
- Extract specific information: Extract specific data points from a document, such as the amount, client name, and deadline from a contract.
- Compare documents: Compare the information in two documents, identifying matching and differing clauses.
- Generate text based on document content: Create new text based on the content of a document.
- Ask questions based on uploaded documents: Similar to RAG Assistants, but with the ability to upload new documents from the Workspace.
File Support provides a robust set of features for managing and using files:
- File Management: You can manage files associated with each Project using CRUD operations (create, read, update, delete) through a File API that uses the Project identifier as the key. You can also configure a storage limit per Project to manage storage space effectively.
- File Referencing in prompts: You can reference files that were uploaded previously through the File API by using variables like {file:filename} or {file:*}. For example, you can use the prompt: "Compare {file:id1} with {file:id2}". If you upload a file directly from Workspace, you do not need to add {file:...} in your prompt because the file is already attached to the interaction.
The system seamlessly handles file uploads, context window size, and summarization, regardless of whether the model is multimodal or not.
- File Types: File support extends beyond documents to include images and videos, using a similar syntax: {file:image1} or {file:video1}. For example, you can use the prompt: "Extract the text from image {file:id1} and summarize its content".
- Workspace Upload: Consumers can attach files when you interact with an Assistant or Agent by selecting existing files or uploading new ones. You can also attach files using the paperclip icon, or paste images directly with Ctrl + V (for example, screenshots).
- Choose your AI solution: Select the Assistant or Agent that best suits your needs.
- Upload your file: Click on the “Upload File” button and select the file you want to use. The Assistant / Agent will recognize the file type and process it accordingly.
- Write your prompt: Enter your instructions, describing what you want the Assistant or Agent to do with the file.
- Use file references: Use the following syntax when you need to reference a file that already exists in the Project storage (uploaded through the File API):
- Documents: {file:filename} (for example, {file:contract})
- Images: {file:image1}
- Videos: {file:video1}
- Audio: {file:audio1}
- Submit your prompt: Send your prompt to the Assistant or Agent.
- Review the results: The Assistant or Agent will process your request and display the results.
- Prompt (File API reference): “Summarize the key points of the document {file:contract}”.
- Prompt (Workspace attachment): “Summarize the key points of the attached contract.”
- Response: “The contract outlines the terms of service for a software subscription, including pricing, payment terms, and support options”.
- Prompt (File API reference): “Describe the image {file:product_image}”.
- Prompt (Workspace attachment): “Describe the attached image.”
- Response: “The image shows a red sports car with black leather interior”.
- Prompt (File API reference): “Summarize the main points of the video {file:tutorial}”.
- Prompt (Workspace attachment): “Summarize the main points of the attached video.”
- Response: “The video demonstrates how to use the new software feature for creating custom reports”.
- Prompt (File API reference): “Transcribe the audio file {file:meeting_recording}”.
- Prompt (Workspace attachment): “Transcribe the attached audio.”
- Response: The assistant will provide a text transcript of the audio recording.