playwright-mcp/docs/JQ_INTEGRATION_DESIGN.md

🔮 jq + ripgrep Ultimate Filtering System Design

🎯 Vision

Create the most powerful filtering system for browser automation by combining:

• jq: Structural JSON querying and transformation
• ripgrep: High-performance text pattern matching
• Differential Snapshots: Our revolutionary 99% response reduction

Result: Triple-layer precision filtering achieving 99.9%+ noise reduction with surgical accuracy.

🏗️ Architecture

Filtering Pipeline

Original Snapshot (1000+ lines)
        ↓
[1] Differential Processing (React-style reconciliation)
        ↓ 99% reduction
    20 lines of changes
        ↓
[2] jq Structural Filtering (JSON querying)
        ↓ Structural filter
    8 matching elements
        ↓
[3] ripgrep Pattern Matching (text search)
        ↓ Pattern filter
    2 exact matches
        ↓
Result: Ultra-precise (99.9% total reduction)

Integration Layers

Layer 1: jq Structural Query

// Filter JSON structure BEFORE text matching
jqExpression: '.changes[] | select(.type == "added" and .element.role == "button")'

// What happens:
// - Parse differential JSON
// - Apply jq transformation/filtering
// - Output: Only added button elements

Layer 2: ripgrep Text Pattern

// Apply text patterns to jq results
filterPattern: 'submit|send|post'

// What happens:
// - Take jq-filtered JSON
// - Convert to searchable text
// - Apply ripgrep pattern matching
// - Output: Only buttons matching "submit|send|post"

Layer 3: Combined Power

browser_configure_snapshots({
  differentialSnapshots: true,

  // Structural filtering with jq
  jqExpression: '.changes[] | select(.element.role == "button")',

  // Text pattern matching with ripgrep
  filterPattern: 'submit.*form',
  filterFields: ['element.text', 'element.attributes.class']
})

🔧 Implementation Strategy

Option 1: Direct Binary Spawn (Recommended)

Pros:

• Consistent with ripgrep architecture
• Full jq 1.8.1 feature support
• Maximum performance
• No npm dependencies
• Complete control

Implementation:

// src/filtering/jqEngine.ts
export class JqEngine {
  async query(data: any, expression: string): Promise<any> {
    // 1. Write JSON to temp file
    const tempFile = await this.createTempFile(JSON.stringify(data));

    // 2. Spawn jq process
    const jqProcess = spawn('jq', [expression, tempFile]);

    // 3. Capture output
    const result = await this.captureOutput(jqProcess);

    // 4. Cleanup and return
    await this.cleanup(tempFile);
    return JSON.parse(result);
  }
}

Option 2: node-jq Package

Pros:

• Well-maintained (v6.3.1)
• Promise-based API
• Error handling included

Cons:

• External dependency
• Slightly less control

Implementation:

import jq from 'node-jq';

export class JqEngine {
  async query(data: any, expression: string): Promise<any> {
    return await jq.run(expression, data, { input: 'json' });
  }
}

Recommended: Option 1 (Direct Binary)

For consistency with our ripgrep implementation and maximum control.

📋 Enhanced Models

Extended Filter Parameters

export interface JqFilterParams extends UniversalFilterParams {
  /** jq expression for structural JSON querying */
  jq_expression?: string;

  /** jq options */
  jq_options?: {
    /** Output raw strings (jq -r flag) */
    raw_output?: boolean;

    /** Compact output (jq -c flag) */
    compact?: boolean;

    /** Sort object keys (jq -S flag) */
    sort_keys?: boolean;

    /** Null input (jq -n flag) */
    null_input?: boolean;

    /** Exit status based on output (jq -e flag) */
    exit_status?: boolean;
  };

  /** Apply jq before or after ripgrep */
  filter_order?: 'jq_first' | 'ripgrep_first' | 'jq_only' | 'ripgrep_only';
}

Enhanced Filter Result

export interface JqFilterResult extends DifferentialFilterResult {
  /** jq expression that was applied */
  jq_expression_used?: string;

  /** jq execution metrics */
  jq_performance?: {
    execution_time_ms: number;
    input_size_bytes: number;
    output_size_bytes: number;
    reduction_percent: number;
  };

  /** Combined filtering metrics */
  combined_performance: {
    differential_reduction: number;  // 99%
    jq_reduction: number;            // 60% of differential
    ripgrep_reduction: number;       // 75% of jq result
    total_reduction: number;         // 99.9% combined
  };
}

🎪 Usage Scenarios

Scenario 1: Structural + Text Filtering

// Find only error-related button changes
browser_configure_snapshots({
  differentialSnapshots: true,
  jqExpression: '.changes[] | select(.element.role == "button" and .change_type == "added")',
  filterPattern: 'error|warning|danger',
  filterFields: ['element.text', 'element.attributes.class']
})

// Result: Only newly added error-related buttons

Scenario 2: Console Error Analysis

// Complex console filtering
browser_configure_snapshots({
  differentialSnapshots: true,
  jqExpression: '.console_activity[] | select(.level == "error" and .timestamp > $startTime)',
  filterPattern: 'TypeError.*undefined|ReferenceError',
  filterFields: ['message', 'stack']
})

// Result: Only recent TypeError/ReferenceError messages

Scenario 3: Form Validation Tracking

// Track validation state changes
browser_configure_snapshots({
  differentialSnapshots: true,
  jqExpression: `
    .changes[]
    | select(.element.role == "textbox" or .element.role == "alert")
    | select(.change_type == "modified" or .change_type == "added")
  `,
  filterPattern: 'invalid|required|error|validation',
  filterOrder: 'jq_first'
})

// Result: Only form validation changes

Scenario 4: jq Transformations

// Extract and transform data
browser_configure_snapshots({
  differentialSnapshots: true,
  jqExpression: `
    .changes[]
    | select(.element.role == "link")
    | { text: .element.text, href: .element.attributes.href, type: .change_type }
  `,
  filterOrder: 'jq_only'  // No ripgrep, just jq transformation
})

// Result: Clean list of link objects with custom structure

Scenario 5: Array Operations

// Complex array filtering and grouping
browser_configure_snapshots({
  differentialSnapshots: true,
  jqExpression: `
    [.changes[] | select(.element.role == "button")]
    | group_by(.element.text)
    | map({text: .[0].element.text, count: length})
  `,
  filterOrder: 'jq_only'
})

// Result: Grouped count of button changes by text

🎯 Configuration Schema

// Enhanced browser_configure_snapshots parameters
const configureSnapshotsSchema = z.object({
  // Existing parameters...
  differentialSnapshots: z.boolean().optional(),
  differentialMode: z.enum(['semantic', 'simple', 'both']).optional(),

  // jq Integration
  jqExpression: z.string().optional().describe(
    'jq expression for structural JSON querying. Examples: ' +
    '".changes[] | select(.type == \\"added\\")", ' +
    '"[.changes[]] | group_by(.element.role)"'
  ),

  jqRawOutput: z.boolean().optional().describe('Output raw strings instead of JSON (jq -r)'),
  jqCompact: z.boolean().optional().describe('Compact JSON output (jq -c)'),
  jqSortKeys: z.boolean().optional().describe('Sort object keys (jq -S)'),

  // Combined filtering
  filterOrder: z.enum(['jq_first', 'ripgrep_first', 'jq_only', 'ripgrep_only'])
    .optional()
    .default('jq_first')
    .describe('Order of filter application'),

  // Existing ripgrep parameters...
  filterPattern: z.string().optional(),
  filterFields: z.array(z.string()).optional(),
  // ...
});

📊 Performance Expectations

Triple-Layer Filtering Performance

Original Snapshot: 1,247 lines
  ↓ [Differential: 99% reduction]
Differential Changes: 23 lines
  ↓ [jq: 60% reduction]
jq Filtered: 9 elements
  ↓ [ripgrep: 75% reduction]
Final Result: 2-3 elements

Total Reduction: 99.8%
Total Time: <100ms
  - Differential: 30ms
  - jq: 15ms
  - ripgrep: 10ms
  - Overhead: 5ms

🔒 Safety and Error Handling

jq Expression Validation

// Validate jq syntax before execution
async validateJqExpression(expression: string): Promise<boolean> {
  try {
    // Test with empty object
    await this.query({}, expression);
    return true;
  } catch (error) {
    throw new Error(`Invalid jq expression: ${error.message}`);
  }
}

Fallback Strategy

// If jq fails, fall back to ripgrep-only
try {
  result = await applyJqThenRipgrep(data, jqExpr, rgPattern);
} catch (jqError) {
  console.warn('jq filtering failed, falling back to ripgrep-only');
  result = await applyRipgrepOnly(data, rgPattern);
}

🎉 Revolutionary Benefits

1. Surgical Precision

• Before: Parse 1000+ lines manually
• Differential: Parse 20 lines of changes
• + jq: Parse 8 structured elements
• + ripgrep: See 2 exact matches
• Result: 99.9% noise elimination

2. Powerful Transformations

// Not just filtering - transformation!
jqExpression: `
  .changes[]
  | select(.element.role == "button")
  | {
      action: .element.text,
      target: .element.attributes.href // empty,
      classes: .element.attributes.class | split(" ")
    }
`

// Result: Clean, transformed data structure

3. Complex Conditions

// Multi-condition structural queries
jqExpression: `
  .changes[]
  | select(
      (.change_type == "added" or .change_type == "modified")
      and .element.role == "button"
      and (.element.attributes.disabled // false) == false
    )
`

// Result: Only enabled, changed buttons

4. Array Operations

// Aggregations and grouping
jqExpression: `
  [.changes[] | select(.element.role == "button")]
  | length  # Count matching elements
`

// Or:
jqExpression: `
  .changes[]
  | .element.text
  | unique  # Unique button texts
`

📝 Implementation Checklist

• Create src/filtering/jqEngine.ts with binary spawn implementation
• Extend src/filtering/models.ts with jq-specific interfaces
• Update src/filtering/engine.ts to orchestrate jq + ripgrep
• Add jq parameters to src/tools/configure.ts schema
• Implement filter order logic (jq_first, ripgrep_first, etc.)
• Add jq validation and error handling
• Create comprehensive tests with complex queries
• Document all jq capabilities and examples
• Add performance benchmarks for triple-layer filtering

🚀 Next Steps

1. Implement jq engine with direct binary spawn
2. Integrate with existing ripgrep filtering system
3. Add configuration parameters to browser_configure_snapshots
4. Test with complex real-world queries
5. Document and celebrate the most powerful filtering system ever built!

---

This integration will create unprecedented filtering power: structural JSON queries + text pattern matching + differential optimization = 99.9%+ precision with complete flexibility. 🎯