Verb Sentence Matching System

Overview

The sentence matching system automatically identifies and links verb conjugations and infinitives within sentences, enabling intelligent sentence pair generation and enhanced language learning experiences.

Architecture

Data Flow 

┌─────────────────────────────────────────────────────────────┐
│  SENTENCE-DRIVEN MATCHING (sentence → verbs)                │
│                                                              │
│  New Sentence Created (≤80 chars)                           │
│           ↓                                                  │
│  ProcessConjugationMatchesJob (background)                  │
│           ↓                                                  │
│  ConjugationMatcherService + InfinitiveMatcherService       │
│           ↓                                                  │
│  Creates: sentence_conjugations + sentence_infinitives      │
│           ↓                                                  │
│  Updates: sentence.verb_matched = true                      │
│           conjugation.sentences_matched = true              │
│           infinitive.sentences_matched = true               │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│  VERB-DRIVEN MATCHING (verb → sentences)                    │
│                                                              │
│  FindSentenceMatchesJob triggered (manual/API)              │
│           ↓                                                  │
│  Searches all sentences in same language                    │
│           ↓                                                  │
│  Creates: sentence_conjugations or sentence_infinitives     │
│           ↓                                                  │
│  Updates: conjugation/infinitive.sentences_matched = true   │
└─────────────────────────────────────────────────────────────┘

Database Schema

Core Tables

conjugations

  • Stores finite verb forms (je vais, tu vas, etc.)
  • Key fields: language, lemma, full_form, form
  • New field: sentences_matched (boolean) - has this been processed?

infinitives

  • Stores infinitive verb forms (aller, sein, hablar, etc.)
  • Key fields: language, lemma, form, variant
  • New field: sentences_matched (boolean) - has this been processed?

sentences

  • Stores example sentences in all languages
  • Key fields: language, sentence
  • Existing field: verb_matched (boolean) - has this been processed?
  • Only sentences ≤80 characters are matched (performance optimization)

Join Tables

sentence_conjugations Links sentences to specific conjugations with metadata:

  • sentence_id, conjugation_id
  • language (denormalized for fast filtering)
  • lemma, mood, tense_name, person_seq, person_label, conj_form
  • approach - matching method used (‘fts’ for full-text search)
  • matched_text, span_start, span_end - position info (future use)
  • confidence - matching confidence (future use)

sentence_infinitives Links sentences to infinitives with metadata:

  • sentence_id, infinitive_id
  • language (denormalized for fast filtering)
  • lemma, infinitive_variant
  • approach - matching method used (‘fts’ for full-text search)
  • matched_text, span_start, span_end - position info (future use)
  • confidence - matching confidence (future use)

Indexes

Critical for performance:

-- GIN full-text search index on sentences
CREATE INDEX CONCURRENTLY index_sentences_on_sentence_fts 
ON sentences USING gin(to_tsvector('simple', sentence));

-- Language filtering indexes
CREATE INDEX index_sentences_on_language_and_verb_matched 
ON sentences (language, verb_matched);

CREATE INDEX index_conjugations_on_language_and_sentences_matched
ON conjugations (language, sentences_matched);

CREATE INDEX index_infinitives_on_language_and_sentences_matched
ON infinitives (language, sentences_matched);

-- Join table indexes
CREATE INDEX index_sentence_conjugations_on_language_and_lemma
ON sentence_conjugations (language, lemma);

CREATE INDEX index_sentence_infinitives_on_language_and_lemma
ON sentence_infinitives (language, lemma);

Matching Algorithm

Full-Text Search Approach

The system uses PostgreSQL’s built-in full-text search capabilities:

  1. Text Vectorization: Sentences are converted to tsvector using the ‘simple’ configuration
    • ‘simple’ configuration treats all words as-is (no stemming or stop words)
    • This allows matching across all supported languages without language-specific dictionaries
  2. Query Matching: Verb forms are searched using plainto_tsquery
    • Special characters are stripped: j'ai → j ai
    • Word boundaries are automatically enforced
    • Case-insensitive matching
  3. Example Query: 
    SELECT id FROM sentences
WHERE language = 'fr'
  AND char_length(sentence) <= 80
  AND to_tsvector('simple', sentence) @@ plainto_tsquery('simple', 'vous conduisiez')

Language Isolation

CRITICAL: Every query includes a language filter to prevent cross-language matching:

# WRONG - Could match across languages
Conjugation.where.not(full_form: nil)

# CORRECT - Language-isolated
Conjugation.where(language: 'fr').where.not(full_form: nil)

All services, jobs, and rake tasks enforce single-language processing:

  • Services receive a sentence/conjugation/infinitive with an embedded language
  • Jobs validate language consistency
  • Rake tasks process one language at a time

Matching Rules

  1. Word Boundaries Only: “je vais” matches “Oui, je vais bien” but not “jeune vais”
  2. Case Insensitive: “Aller” matches “aller”, “Aller”, “ALLER”
  3. Common Verbs Only: Only matches verb_lemmas.common = true (performance optimization)
  4. Length Filter: Only sentences ≤80 characters are matched (performance optimization)
  5. Same Language: Conjugation/infinitive language must match sentence language

Service Layer

ConjugationMatcherService

Purpose: Given a sentence, find all conjugations that appear in it

Usage:

sentence = Sentence.find(123)
service = Resources::Sentences::ConjugationMatcherService.new(sentence: sentence)
match_count = service.call
# => 3 (found 3 conjugations in this sentence)

Process:

  1. Check if sentence already processed (verb_matched?) - skip if true
  2. Check sentence length (skip if > 80 chars)
  3. Execute single SQL query to find all matching conjugations in same language
  4. Filter out duplicates
  5. Bulk insert new sentence_conjugations records
  6. Update conjugations.sentences_matched = true for matched conjugations
  7. Mark sentence.verb_matched = true

InfinitiveMatcherService

Purpose: Given a sentence, find all infinitives that appear in it

Usage:

sentence = Sentence.find(123)
service = Resources::Sentences::InfinitiveMatcherService.new(sentence: sentence)
match_count = service.call
# => 1 (found 1 infinitive in this sentence)

Process: Same as ConjugationMatcherService but for infinitives

Background Jobs

ProcessConjugationMatchesJob

Trigger: Automatically when new sentence created (if ≤80 chars)

Purpose: Process a single sentence to find all verbs within it

Usage:

# Called automatically via Sentence model callback
# Or manually:
Resources::Sentences::ProcessConjugationMatchesJob.perform_later(sentence_id)

Features:

  • Runs both ConjugationMatcherService and InfinitiveMatcherService
  • Logs match counts
  • Idempotent (checks verb_matched flag)

FindSentenceMatchesJob (Conjugations)

Trigger: Manual or via API

Purpose: Find all sentences that contain a specific conjugation

Usage:

conjugation = Conjugation.find(456)
Resources::Conjugations::FindSentenceMatchesJob.perform_later(conjugation.id)

Features:

  • Skips if conjugation.sentences_matched = true
  • Searches all short sentences in same language
  • Marks conjugation as processed even if no matches found

FindSentenceMatchesJob (Infinitives)

Trigger: Manual or via API

Purpose: Find all sentences that contain a specific infinitive

Usage:

infinitive = Infinitive.find(789)
Resources::Infinitives::FindSentenceMatchesJob.perform_later(infinitive.id)

Features: Same as conjugation version but for infinitives

Rake Tasks

Main Task: match_sentences

Purpose: Bulk processing of conjugations and infinitives

# Basic usage - all languages, both conjugations and infinitives
bundle exec rake immersive:match_sentences

# Single language, conjugations only
bundle exec rake "immersive:match_sentences[fr,conjugations_only]"

# Single language, infinitives only
bundle exec rake "immersive:match_sentences[de,infinitives_only]"

# Both conjugations and infinitives for Spanish
bundle exec rake "immersive:match_sentences[es,both]"

# Force rerun (clears existing matches)
bundle exec rake "immersive:match_sentences[fr,both,rerun]"

# Test with limited dataset (first 50 items)
bundle exec rake "immersive:match_sentences[fr,conjugations_only,,50]"

Parameters:

  • language - Language code (fr, de, es, en, pt, it) or blank for all
  • mode - conjugations_only, infinitives_only, or both (default: both)
  • rerun - Set to ‘rerun’ to clear existing matches and reprocess
  • limit - Optional limit for testing (e.g., 50 to process only first 50 items)

Features:

  • Real-time progress reporting with ETA
  • Processes one language at a time (enforces language isolation)
  • Skips already-processed items unless rerun specified
  • Reports rate (items/second) and total matches
  • Idempotent - safe to run multiple times

Example Output:

================================================================================
SENTENCE MATCHING TASK
================================================================================
Languages: fr
Mode: conjugations_only
Rerun: NO (skip already matched)
Limit: none (process all)
================================================================================

================================================================================
Processing language: FR
================================================================================

--- Processing Conjugations ---
Conjugations to process: 9247
Progress: 9247/9247 (100.0%) | Matches: 15432 | Rate: 45.2/s | ETA: 0m 0s         
Conjugations complete: 9247 processed, 15432 total matches in 204.6s

Test Task: test_sentence_matching

Purpose: Test the system with a small dataset to verify everything works

# Test with 10 items in French
bundle exec rake "immersive:test_sentence_matching[fr,10]"

# Test with 5 items in German
bundle exec rake "immersive:test_sentence_matching[de,5]"

# Test with default 20 items in Spanish
bundle exec rake "immersive:test_sentence_matching[es]"

Tests Performed:

  1. Verify GIN index exists and uses ‘simple’ configuration
  2. Check data availability (conjugations, infinitives, sentences) for language
  3. Test sentence → conjugations matching
  4. Test sentence → infinitives matching
  5. Test conjugation → sentences matching
  6. Test infinitive → sentences matching
  7. Test background jobs can be enqueued
  8. Show statistics for the language

Example Output:

================================================================================
SENTENCE MATCHING SYSTEM TEST
================================================================================
Language: FR
Sample size: 10 items
================================================================================

TEST 1: Checking GIN full-text search index...
GIN index exists with 'simple' configuration

TEST 2: Checking data availability for fr...
  Conjugations (common verbs): 9247
  Infinitives (common verbs): 823
  Sentences (≤80 chars): 45621

TEST 3: Testing sentence → conjugations matching...
  Processed: 10 sentences
  With matches: 8 (80.0%)
  Total matches: 24
  Avg per sentence: 2.40
  Example: "Je vais au marché." → je vais (aller)

...

Query Patterns

Find all sentences using a specific conjugation 

conjugation = Conjugation.find_by(language: 'fr', full_form: 'je vais')
sentences = conjugation.sentences
# Returns all sentences containing "je vais"

Find all conjugations in a sentence 

sentence = Sentence.find(123)
conjugations = sentence.conjugations
# Returns all conjugations found in this sentence

Find all infinitives in sentences for a lemma 

infinitives = Infinitive.where(language: 'fr', lemma: 'aller')
sentences = Sentence
  .joins(:sentence_infinitives)
  .where(sentence_infinitives: {infinitive_id: infinitives.ids})
  .distinct

Find unprocessed conjugations for a language 

Conjugation
  .joins(:verb_lemma)
  .where(language: 'fr', sentences_matched: false, verb_lemmas: {common: true})
  .where.not(full_form: [nil, ''])

Get statistics for a language 

language = 'fr'

stats = {
  conjugations_total: Conjugation.where(language: language).count,
  conjugations_matched: Conjugation.where(language: language, sentences_matched: true).count,
  conjugation_matches_total: SentenceConjugation.where(language: language).count,
  
  infinitives_total: Infinitive.where(language: language).count,
  infinitives_matched: Infinitive.where(language: language, sentences_matched: true).count,
  infinitive_matches_total: SentenceInfinitive.where(language: language).count,
  
  sentences_total: Sentence.where(language: language).count,
  sentences_short: Sentence.where(language: language).where("char_length(sentence) <= 80").count,
  sentences_matched: Sentence.where(language: language, verb_matched: true).count
}

Performance Considerations

GIN Index Management

GIN indexes are powerful but have specific performance characteristics:

Write Performance:

  • GIN indexes use a “pending list” to batch updates
  • Default gin_pending_list_limit is 4MB
  • When limit reached, pending list is flushed (can cause slow writes)
  • Autovacuum periodically flushes pending list

Strategies for Large Tables:

  1. Adjust gin_pending_list_limit based on workload
  2. Tune autovacuum frequency
  3. Use CREATE INDEX CONCURRENTLY to avoid blocking writes
  4. Monitor pending list size with pgstatginindex('index_name')

Read Performance:

  • GIN indexes enable fast full-text search
  • Much faster than regex or LIKE queries
  • Only supports Bitmap Index Scans (not Index Scan or Index Only Scan)

Batch Processing Strategy

The rake task uses several optimizations:

  1. Single SQL Queries: One query per conjugation/infinitive instead of N+1
  2. Bulk Inserts: Use insert_all instead of individual create calls
  3. Skip Processed: Check sentences_matched flag to avoid re-processing
  4. Progress Reporting: Update every 50 items to balance visibility and performance
  5. Denormalized Columns: Query on language directly instead of joining

Expected Performance

On typical hardware:

  • Sentence processing: 40-60 sentences/second
  • Conjugation processing: 30-50 conjugations/second
  • Infinitive processing: 50-80 infinitives/second

For a language with:

  • 10,000 conjugations
  • 1,000 infinitives
  • 50,000 sentences

Full processing time: ~15-20 minutes

Monitoring and Troubleshooting

Check GIN Index Health 

-- Check if index exists
SELECT indexname, indexdef 
FROM pg_indexes 
WHERE indexname = 'index_sentences_on_sentence_fts';

-- Check pending list size (requires pg_stattuple extension)
CREATE EXTENSION IF NOT EXISTS pgstattuple;
SELECT * FROM pgstatginindex('index_sentences_on_sentence_fts');

Common Issues

Issue: Slow sentence matching

Diagnosis:

-- Check if GIN index is being used
EXPLAIN ANALYZE
SELECT id FROM sentences
WHERE language = 'fr'
  AND to_tsvector('simple', sentence) @@ plainto_tsquery('simple', 'aller');

Solution: Ensure GIN index exists and is up-to-date. Run VACUUM ANALYZE sentences;

Issue: Cross-language matches appearing

Diagnosis: Check for queries without language filter

# BAD - no language filter
SentenceConjugation.count

# GOOD - language filtered
SentenceConjugation.where(language: 'fr').count

Solution: Always include language filter in queries

Issue: Some conjugations/infinitives not getting matched

Reasons:

  1. No sentences available containing that form
  2. Sentence too long (>80 chars)
  3. Special characters preventing match

Diagnosis:

conjugation = Conjugation.find(id)
escaped_form = conjugation.full_form.gsub(/[^\w\s]/, '')

# Manual search
sentences = Sentence
  .where(language: conjugation.language)
  .where("sentence ILIKE ?", "%#{conjugation.full_form}%")
# vs FTS search
fts_sentences = Sentence
  .where(language: conjugation.language)
  .where("to_tsvector('simple', sentence) @@ plainto_tsquery('simple', ?)", escaped_form)

Future Enhancements

Potential improvements to consider:

  1. Span Detection: Accurately record span_start and span_end for highlighting
  2. Confidence Scoring: Calculate match confidence based on context
  3. Negative Forms: Handle negative conjugations specially (ne…pas, etc.)
  4. Compound Tenses: Match auxiliary + participle combinations
  5. Accent Handling: Add unaccent extension for accent-insensitive matching
  6. Multi-word Forms: Handle phrasal verbs and verb phrases
  7. Caching: Cache frequently accessed matches in Redis
  8. Analytics: Track which verbs have poor sentence coverage
  9. Auto-generation: Trigger AI sentence generation for under-covered verbs
  10. Quality Scoring: Rate sentence examples by clarity and usefulness
This site uses Just the Docs, a documentation theme for Jekyll.