Metadata
Concept
Metadata adds supplementary information to Documents and Blocks, enriching their semantic representation and improving search accuracy. CogSol supports two types of metadata structures: custom metadata and metadata templates.
Custom metadata provides flexibility for ad-hoc information tagging. Users can add metadata to documents or blocks by simply defining a key name and a list of values. This metadata enriches embeddings, improves model retrieval, and enhances search effectiveness.
Metadata Configs represent structured metadata configurations associated with a selected Topic, following predefined templates. For more details, see the Metadata Template section.
Attributes
| Attribute | Description |
|---|---|
| Key Name | Unique identifier for the metadata field (e.g., importance, created_date, description). |
| Values | List of string or integer values. |
Entities That Support Metadata
Metadata can be configured for the following entities: documents, blocks, chats, messages, and attachments.
Document Metadata
- Used in embedding generation for all blocks within the document
- Included in information retrieval for searches that return any block from that document
Block Metadata
- Used in embedding generation for that specific block
- Included in information retrieval for searches that return that block
Chat, Message, and Attachment Metadata
Metadata can be sent when creating a chat, sending a message, or attaching files to messages. This enables custom use cases such as:
- Including a user ID in chat metadata for tracking and personalization
- Providing additional context about an attachment without adding it to the message content itself
In the latter case, it's common to create a Script Tool (executed before answer generation) to extract and process this metadata.
Benefits
Embedding Enrichment Document and block custom metadata is automatically used to enrich embeddings, enhancing the semantic representation. This leads to more accurate retrieval in relevant scenarios, ensuring the right content is selected when needed.
Improved Information Retrieval Since semantic searches are powered by embeddings, enriched embeddings with accurate metadata result in more relevant search results and better retrieval performance.
Flexible Information Storage Metadata allows you to store supplementary information that may be less formal than document content itself. This helps with organization, categorization, and providing additional context without cluttering the main content.