Testing

This application supports two testing modes to handle external service dependencies.

Test Modes

Stubbed Mode (Default)

In stubbed mode, all external services are mocked with realistic responses. This is the default mode and is used in CI.

External services that are stubbed:

• OpenAI API (GPT models)
• AWS Translate
• AWS Polly (Text-to-Speech)

To run in stubbed mode:

# Default behavior
bin/rails test

# Explicit stubbed mode
bundle exec rake test:stubbed

Live Mode

In live mode, tests make real API calls to external services. This is useful for integration testing and verifying that the application works with actual external APIs.

Warning: Live mode will make real API calls and may incur costs.

To run in live mode:

# With confirmation prompt
bundle exec rake test:live

# Without confirmation (for CI/automation)
LIVE_MODE=true bin/rails test

Configuration

Environment Variables

• LIVE_MODE=true - Enables live mode (external services make real API calls)
• LIVE_MODE not set or any other value - Uses stubbed mode (default)

CI Configuration

The CI pipeline runs in stubbed mode only to:

• Avoid API costs
• Ensure consistent test results
• Prevent rate limiting issues
• Speed up test execution

Stubbed Responses

The stubbing system provides realistic responses:

OpenAI Responses

• Sentence generation: Returns realistic sentence pairs
• Document coaching: Returns helpful suggestions
• Default: Generic response for other prompts

AWS Translate Responses

• Common words: Returns actual translations for basic vocabulary
• Other text: Returns formatted translation strings

AWS Polly Responses

• Returns fake audio data with text-specific identifiers

Adding New External Services

To add stubbing for a new external service:

1. Add the service to test/support/external_services.rb
2. Create a stub_service_name! method
3. Add realistic response generation logic
4. Call the stub method in stub_all!

Example:

def stub_new_service!
  new_service_client = mock("NewService::Client")
  new_service_client.stubs(:api_method).returns(
    stub(result: "stubbed_result")
  )

  NewService::Client.stubs(:new).returns(new_service_client)
end

Browser Timeout Handling

System tests using Capybara with Cuprite (Ferrum) can occasionally experience timeouts, especially in CI environments. These timeouts are often environmental (resource constraints, network issues) rather than actual test failures.

Automatic Timeout Handling

The test suite includes an automatic handler (test/support/capybara_timeout_handler.rb) that:

• Detects timeout errors from Capybara/Ferrum:
  • Ferrum::ProcessTimeoutError
  • Capybara::TimeoutError
  • Ferrum::TimeoutError
  • “Browser did not produce websocket url” errors
  • Other timeout-related errors
• Soft-fails tests - Clears the exception so the test doesn’t fail CI
• Logs warnings - Still reports the timeout for visibility

Example Output

When a timeout occurs, you’ll see a warning like:

⚠️  Soft-failed due to Capybara/Ferrum timeout: Document sentence translation clicking on a sentence to translate makes a request to the frontend API and displays the translation
   Error: Ferrum::ProcessTimeoutError
   Message: Browser did not produce websocket url within 10 seconds, try to increase :process_timeout...
   Location: test/system/documents/document_sentence_translation_test.rb:11

The test will pass, but the warning indicates a timeout occurred.

Disabling Timeout Handling

To disable the timeout handler (e.g., for debugging):

DISABLE_TIMEOUT_HANDLER=true bin/rails test

This will allow timeout errors to fail tests normally.

When Timeouts Occur

Timeouts are most common in:

• CI environments with limited resources
• System tests that interact with JavaScript-heavy pages
• Tests that wait for network requests or animations

If timeouts become frequent, consider:

• Increasing browser timeouts in test/test_helper.rb
• Optimizing slow system tests
• Using request specs instead of system specs when possible

Best Practices

1. Always use stubbed mode for unit tests
2. Use live mode sparingly - only for integration testing
3. Keep stubbed responses realistic - they should match expected API behavior
4. Test both modes - ensure your code works with both stubbed and real responses
5. Document API changes - update stubbed responses when external APIs change
6. Monitor timeout warnings - frequent timeouts may indicate test or infrastructure issues