When Robert Pirsig wrote Zen and the Art of Motorcycle Management, he was concerned with the nature of quality. Pirsig noted that quality involves patterns and dynamic qualities. Ultimately, Pirsig argues that attending to the nature of quality is a moral matter and the highest form of intellectual activity.
In the last several months, the question of the quality of the databases that Large Language Models (LLMs) use to analyze and provide answers has become paramount. Quality in this instance involves these factors:
- Source of the content—research-based or generalized opinion?
- Contextualization of the source content—how do the data points “work together?”
- Structure of how the source content is stored in the database—how is the data organized and stored?
The Problem
LLMs (OpenAI, Gemini, Claude, etc.) use the open internet to scrape for content that fits the inquiry. Consequently, answers to inquiries include incorrect and unfounded information, which can lead individuals down the wrong path. Consider that the top 3 sources of content in these circumstances are Reddit, LinkedIn, and Wikipedia. The old phrase, garbage in-garbage out applies here as the primary content is simply free-floating information. Even if the LLM (Large Language Model) “learns”, it is skewed and cannot overcome its original data source.
By now we know that information takes on deeper meaning when it is placed in context to a given purpose or situation. LLM’s are trained to provide confident answers even when false, leaving the discernment to the user. Add in the fact that hallucination rates of widely used LLMs are increasing, not decreasing, and we find ourselves on a runaway train.
The Fix
In the arena of talent management, the vast majority of new AI coaches are built on top of the systems we reviewed above. Their source of content? The internet.
We know that a curated and vectorized database is essential for the best, highest quality answers. This approach turns Garbage In, Garbage Out into Quality In, Quality Out.
A vectorized database (or vector database) is how quality data is stored. It is a specialized system designed to manage, index, and query high-dimensional vector embeddings—numerical representations of unstructured data like text, images, or audio. Unlike traditional relational databases, vector databases focus on semantic similarity rather than exact keyword matches.
Here are the key attributes and qualities of a vectorized database:
Core Capabilities and Features
- Vector Embeddings Management: Designed to store, index, and manage complex, high-dimensional vector data, which represent semantic meaning.
- Approximate Nearest Neighbor (ANN) Search: Instead of an exhaustive search, they use algorithms like HNSW, IVF, or PQ to rapidly identify the most similar vectors to a query, trading exact accuracy for high speed.
- Metadata Filtering: They support storing metadata alongside vectors, enabling hybrid searches that combine semantic similarity with specific filtering criteria (e.g., “find images similar to X, but only from 2024”).
- Real-time Updates: Supports instantaneous or near-real-time ingestion of new data and updates without needing to re-index the entire dataset.
- Support for Multiple Similarity Metrics: Utilizes mathematical distance metrics to determine similarity, including Cosine Similarity, Euclidean Distance (L2), and Dot Product.
Performance and Architectural Qualities
- Scalability: Built to handle millions or billions of vectors by scaling horizontally across distributed systems.
- High Performance/Low Latency: Optimized for fast retrieval of data, crucial for real-time applications like recommendation engines or chatbots.
- High-Dimensionality Handling: Efficiently manages data with hundreds or thousands of dimensions, which would overwhelm traditional relational databases.
- Separation of Storage and Compute: Modern architectures (often serverless) decouple storage from compute to optimize costs, allowing resources to scale up only during queries.
Data Management and Integration
- CRUD Operations: Provides standard Create, Read, Update, and Delete operations for vector data.
- Ecosystem Integration: Designed to work seamlessly with AI frameworks and tools.
- Data Persistence & Backups: Ensures data safety through built-in backup mechanisms, including “collections” for specific subsets of data.
Security and Reliability
- Role-Based Access Control (RBAC): Offers built-in security to manage user permissions and protect sensitive data.
- Fault Tolerance: Implements replication to maintain high availability even if nodes fail.
Key Differences from Traditional Databases
- Semantic vs. Exact Search: Vector databases find data based on “meaning” or “context,” while traditional databases search for exact keyword and key phrase matches.
- Unstructured Data Focus: Primarily used for data types like images, video, and text, rather than rigid rows and columns.
As Pirsig suggested, quality truly is job one, and focusing on quality-thinking requires enthusiasm and mindfulness, which is both a result of, and a prerequisite for, high-quality interaction with the world.
