One of the AI projects I'm proudest of wasn't the most technically ambitious — it was a semantic search tool built to solve a problem every operations team recognises instantly: years of technical documentation, operational playbooks, and past case resolutions that existed somewhere in the organisation, but that nobody could reliably find when they actually needed them.
The problem: keyword search fails on operational language
Traditional keyword search assumes the person searching uses the same words as the document they're looking for. In operational and technical contexts, that assumption breaks constantly. A support agent searching for 'aircraft won't start' needs to find documentation that describes 'APU ignition failure' or 'engine start fault' — different words, same underlying problem. Keyword search returns nothing useful. The knowledge existed in the organisation. It just wasn't findable.
The project got real internal buy-in after we timed how long it took an experienced team member to find a specific procedural answer using existing tools versus a new team member using the same tools. The gap wasn't a training problem — it was a retrieval problem. Experience was substituting for search.
What we built
- ›Document ingestion and chunking: breaking technical documents into semantically coherent sections rather than arbitrary fixed-length blocks — chunk boundaries that split a procedure mid-step actively hurt retrieval quality
- ›Embedding generation: converting each chunk into a vector representation that captures meaning rather than exact wording, so 'won't start' and 'ignition failure' land near each other in the vector space
- ›Metadata tagging: attaching structured metadata — document type, system, date, applicability — alongside the vector, so search could filter as well as rank
- ›Hybrid retrieval: combining vector similarity search with metadata filtering and keyword matching, rather than relying on vector search alone
- ›A feedback loop: capturing which results users actually clicked and found useful, to progressively improve ranking rather than treating the initial deployment as finished
Lessons learned
- ›Chunk size matters more than the embedding model choice — we got a bigger quality improvement from fixing chunking strategy than from swapping embedding models
- ›Pure vector search isn't enough — combining it with metadata filters (document type, date, system) consistently outperformed vector similarity alone, especially for technical content where precision matters
- ›Evaluation is genuinely hard — 'does this feel like a good result' is not a metric, and we had to build a small labelled test set of real queries with known correct answers before we could meaningfully compare approaches
- ›Adoption requires trust, and trust requires transparency — showing users why a result was returned, and letting them see the source document, mattered more for adoption than marginal gains in ranking accuracy
- ›The first version should be narrower than it wants to be — we scoped to a single document category before expanding, which made both the engineering and the evaluation tractable
Measuring success
I'd invest in the evaluation set earlier. We built the retrieval pipeline first and the way to measure whether it was actually good second, which meant early iteration was based on intuition rather than data. A small, well-labelled set of real queries with known correct answers should be one of the first things built, not one of the last.