Research Knowledge Base

Multi-domain research knowledge base (#190), phase 1. A structured, continuously updated store of research sources and claim-level insights, managed entirely in ActiveAdmin.

Shape

Domains are data, not schema — one generic structure serves all domains. Four are seeded (CreateKnowledgeBaseTables migration):

  1. SLA research (sla-research)
  2. Marketing & advertising research (marketing-advertising)
  3. Competitor analysis & market gaps (competitor-analysis)
  4. AI agents running a company (ai-agents-company)

Two layers of record granularity:

  • KbSource (kb_sources) — one row per source: domain, source_type (academic / practitioner / media / social / other), title, URL (unique per domain), author, publication date, licence/access notes, and fetched content (raw_content, summary, fetched_at, fetch_error).
  • KbInsight (kb_insights) — claim-level, individually citable statements (“spaced learning improves retention 25-50%”) with supporting quote, in-source location, confidence rating (low/medium/high), and a normalized lowercase tag array. Insights are the unit humans and agents actually cite.

Both carry a draft → reviewed → archived status (str_enum) and a full PaperTrail audit trail. Models live in app/models/knowledge_base/ as top-level classes (KbDomain, KbSource, KbInsight) via the Zeitwerk push_dir convention.

Admin

Everything is under the Knowledge Base menu in ActiveAdmin (app/admin/knowledge_base/), admin-authenticated like every other AA surface:

  • Dashboard — per-domain counts, pending-review queues, recent sources and insights, top tags.
  • Domains / Sources / Insights — full CRUD. Insight tags are edited as a comma-separated field; a source’s show page links its insights and offers “Add Insight”.

URL ingestion

On a source with a URL, the Ingest from URL action enqueues KnowledgeBase::IngestUrlJob (Sidekiq, low queue), which runs KnowledgeBase::IngestUrlService:

  1. Fetch the URL (Net::HTTP; 3-redirect limit, 2 MB size cap, 10 s/15 s timeouts, HTML/plain-text content types only).
  2. Extract main text with Nokogiri (scripts, nav, header/footer stripped; <main>/<article> preferred).
  3. KnowledgeBase::IngestPrompt (RubyLLM structured output against KnowledgeBase::IngestSchema) proposes title, author, source type, publication date, summary, and candidate insights.

Everything is a proposal for human review: the source stays draft, human-entered metadata is never overwritten (proposals only fill blanks), and proposed insights are created as drafts. Fetch or LLM failures are recorded on the source’s fetch_error. In development the prompt is faked via FakeAI (LIVE_MODE=true for real calls); the model is configurable via AI_MODEL_KB_INGEST.

Document sources

Not every source is a URL. Sources that exist as files — scanned book chapters, PDFs, slide decks — attach their originals to the source record (has_many_attached :documents, S3 in production, #540). The admin form uploads them (uploads accumulate; they never replace existing documents) and the source’s show page lists each with a download link and a Remove action.

The regular process for a document source:

  1. Create the source with metadata as usual: title, author, publication date, source_type, and licence notes (especially important for scanned copyrighted material — record what may be stored, quoted, and republished).
  2. Attach the original files as documents.
  3. There is no automated ingestion for files (URL ingestion only fetches HTML/plain text): write or paste the summary and, where useful, the extracted text into raw_content by hand or with agent help.
  4. Add claim-level insights as drafts citing chapter/section in location, then review as normal.

The first document source is EuRom5 (#540): the scanned sample chapters and their per-chapter summaries live on one source in the SLA research domain, with the chapter summaries concatenated in raw_content and claim-level insights citing chapter and section.

Postgres full-text search (simple config, expression GIN indexes — same approach as the sentences FTS index): KbSource.search_text covers title/author/summary, KbInsight.search_text covers claim/supporting quote. Both are exposed to ActiveAdmin as ransackable scopes via the “Full-text search” filter on the index pages.

Later phases (explicitly out of phase 1)

  • Embeddings + agent-facing retrieval (pgvector via neighbor, post-#499 consolidation — see #191 for the pipeline precedent)
  • Automated crawling/monitoring of feeds and social media (ToS/licensing review first, same concern class as #416)
  • Public content generation (blog, SEO, social)
  • Multi-agent refresh orchestration (scout/fetcher/critique roles)
This site uses Just the Docs, a documentation theme for Jekyll.