全文检索向量

概念概述

全文检索向量（Full-Text Search Vectors）指的是：全文检索把文档转换为规范化词素向量，使查询按词与词干匹配并按相关度排序，而非按原始子串匹配。

从更严格的定义看，Full-text search is the process of parsing text into lexemes, discarding stop words, applying stemming, and indexing the result so that language-aware queries can be ranked by relevance. 该概念在工程实践中的价值，在于它能帮助团队用统一的语言描述一个具体的质量属性或设计约束，从而在评审与协作中减少歧义。

为什么需要它

It is used for site search, article lookup, and any feature where users type words rather than exact patterns. 它与 Advisory Locks、Cost-Based Query Optimization、Database Connection Multiplexing 等概念经常一起出现，用于解决相近的工程问题。理解它的适用场景，有助于在多种备选方案之间做出有依据的取舍，而不是凭直觉选择技术方案。

常见误解

• full-text search is the same as using LIKE with wildcards
• ranking always reflects what the user considers most relevant

需要避免的反模式

• storing documents without a maintained search vector
• ranking by a single factor instead of a weighted combination

实践要点

1. A stored search vector is a precomputed, indexed representation of the searchable columns, and it must be regenerated whenever those columns change or queries go stale.
2. Weighting lets titles count more than body text, and the ranking functions combine term frequency, document frequency, and normalisation into a relevance score.
3. Configuration is language-specific, so the wrong dictionary produces poor stemming; choose the language explicitly rather than relying on a default that may not fit the content.

参考来源

https://www.postgresql.org/docs/current/textsearch-intro.html

代码示例

export function binarySearch(sorted, target, compare = (a, b) => a - b) {
  let low = 0;
  let high = sorted.length;
  while (low < high) {
    const mid = (low + high) >>> 1;
    const cmp = compare(sorted[mid], target);
    if (cmp < 0) {
      low = mid + 1;
    } else {
      high = mid;
    }
  }
  const found = low < sorted.length && compare(sorted[low], target) === 0;
  return { index: low, found };
}

class TrieNode {
  constructor() {
    this.children = new Map();
    this.terminal = false;
    this.weight = 0;
  }
}

export class Trie {
  #root = new TrieNode();
  #caseSensitive;
  #wordCount = 0;

  constructor({ caseSensitive = false } = {}) {
    if (typeof caseSensitive !== "boolean") {
      throw new TypeError("caseSensitive must be a boolean");
    }
    this.#caseSensitive = caseSensitive;
  }

  #normalize(word) {
    if (typeof word !== "string" || word.trim().length === 0) {
      throw new TypeError("word must be a non-empty string");
    }
    return this.#caseSensitive ? word : word.toLowerCase();
  }

  insert(word, weight = 1) {
    if (!Number.isFinite(weight) || weight < 0) {
      throw new RangeError("weight must be a non-negative finite number");
    }
    const normalized = this.#normalize(word);
    let node = this.#root;
    for (const character of normalized) {
      let child = node.children.get(character);
      if (child === undefined) {
        child = new TrieNode();
        node.children.set(character, child);
      }
      node = child;
    }
    if (!node.terminal) {
      node.terminal = true;
      this.#wordCount += 1;
    }
    node.weight = weight;
    return this;
  }

  has(word) {
    const normalized = this.#normalize(word);
    let node = this.#root;
    for (const character of normalized) {
      const child = node.children.get(character);
      if (child === undefined) return false;
      node = child;
    }
    return node.terminal;
  }

  startsWith(prefix) {
    if (typeof prefix !== "string") {
      throw new TypeError("prefix must be a string");
    }
    const normalized = this.#caseSensitive ? prefix : prefix.toLowerCase();
    let node = this.#root;
    for (const character of normalized) {
      const child = node.children.get(character);
      if (child === undefined) return false;
      node = child;
    }
    return true;
  }

  suggest(prefix, limit = 5) {
    if (typeof prefix !== "string") {
      throw new TypeError("prefix must be a string");
    }
    if (!Number.isInteger(limit) || limit < 1) {
      throw new RangeError("limit must be a positive integer");
    }
    const normalized = this.#caseSensitive ? prefix : prefix.toLowerCase();
    let node = this.#root;
    for (const character of normalized) {
      const child = node.children.get(character);
      if (child === undefined) return [];
      node = child;
    }

    const matches = [];
    this.#collect(node, normalized, matches);
    matches.sort(
      (left, right) =>
        right.weight - left.weight || left.word.localeCompare(right.word),
    );
    return matches.slice(0, limit).map((entry) => entry.word);
  }

  get wordCount() {
    return this.#wordCount;
  }

  #collect(node, prefix, matches) {
    if (node.terminal) {
      matches.push({ word: prefix, weight: node.weight });
    }
    for (const [character, child] of node.children) {
      this.#collect(child, prefix + character, matches);
    }
  }
}

const trie = new Trie();
for (const [word, weight] of [
  ["apple", 5],
  ["app", 9],
  ["apricot", 3],
  ["banana", 7],
  ["appetite", 4],
  ["application", 6],
]) {
  trie.insert(word, weight);
}
console.log(trie.suggest("app"));
console.log(trie.has("APP"), trie.startsWith("APR"), trie.wordCount);
console.log(trie.suggest("zz"));