Technical Whitepaper

Advanced Knowledge and Intelligent Search

Version 1.0 | March 2025

Abstract

Qwello represents a significant advancement in document analysis, knowledge processing, and intelligent search, combining sophisticated PDF analysis with advanced AI capabilities. The platform implements a NestJS-based architecture with Cloudflare AI integration, supporting multiple AI models including Grok and Claude. The system features a robust PDF processing pipeline with parallel worker architecture, sophisticated knowledge graph generation with entity resolution, and interactive visualization capabilities. This unique combination enables deep research capabilities, interactive knowledge exploration, and intelligent search through a unified system.

Current Implementation Status

Completed Features

Core Infrastructure:
- ✓ NestJS-based modular architecture
- ✓ MongoDB integration for data persistence
- ✓ BullMQ for job queue processing
- ✓ AWS S3 integration for file storage
- ✓ Redis for caching and job queue management
Knowledge Graph Foundation:
- ✓ KG report generation with chunking for large graphs
- ✓ DeepSeek integration for analysis
- ✓ Entity relationship mapping with JSON structure
- ✓ Interactive visualization with vis-network.js
PDF Processing Pipeline:
- ✓ SPARK Engine (SingularityNET PDF Analysis & Reasoning for Knowledge)
- ✓ Worker-based parallel processing with BullMQ
- ✓ Multi-model AI approach with fallback mechanisms
- ✓ PDF to image conversion with optimization
- ✓ Rate limiting and error handling with exponential backoff
AI Integration:
- ✓ Cloudflare AI provider implementation
- ✓ Primary: x-ai/grok-2-vision-1212 (Vision)
- ✓ Primary: x-ai/grok-2-1212 (Language)
- ✓ Fallback: anthropic/claude-3.7-sonnet
- ✓ Large context: deepseek/deepseek-r1
API Services:
- ✓ NestJS controllers with versioning
- ✓ Rate-limited endpoints with configurable limits
- ✓ Worker pool with CPU-based scaling
- ✓ WebSocket-based progress tracking
Frontend Interface:
- ✓ React-based UI with TypeScript migration in progress
- ✓ Interactive knowledge graph visualization with vis-network.js
- ✓ Entity filtering and search capabilities
- ✓ Document library management

In Development

TypeScript Migration (In Progress)
- Frontend: React components being converted
- Backend: NestJS implementation with TypeScript
MeTTa Integration (Planned for April 2025)
MORK Engine Integration
Hyperon Runtime Integration
SERP API Integration

1. Introduction

1.1 The Knowledge Processing Challenge

Organizations face increasing challenges in research, document analysis, and knowledge discovery:

Complex information retrieval needs
Deep semantic understanding requirements
Integration of multiple knowledge sources
Domain-specific research demands
Real-time knowledge synthesis needs

1.2 The Qwello Solution

Qwello addresses these challenges through a sophisticated integration of technologies:

NestJS-based modular architecture
Advanced PDF processing pipeline
Multi-model AI analysis with Cloudflare integration
Deep research capabilities
Knowledge graph generation with entity resolution
Interactive visualization with vis-network.js

2. Core Architecture

2.1 System Components

2.2 NestJS Module Structure

src/
├── app.module.ts            # Main application module
├── common/                  # Common utilities and helpers
├── config/                  # Configuration management
├── interfaces/              # TypeScript interfaces
├── modules/
│   ├── ai/                  # AI integration module
│   │   ├── ai.module.ts
│   │   ├── ai.config.ts
│   │   ├── types.ts
│   │   ├── prompts.ts
│   │   ├── ai-providers/    # AI provider implementations
│   │   ├── controllers/     # API endpoints
│   │   ├── enum/           # Enumerations
│   │   └── services/       # AI services
│   ├── auth/               # Authentication module
│   ├── chats/              # Chat functionality
│   ├── pdf/                # PDF processing module
│   │   ├── pdf.module.ts
│   │   ├── pdf.config.ts
│   │   ├── types.ts
│   │   ├── controllers/    # API endpoints
│   │   ├── entities/       # MongoDB schemas
│   │   ├── enum/          # Enumerations
│   │   ├── processors/    # BullMQ processors
│   │   ├── repositories/  # Data access layer
│   │   ├── services/      # Business logic
│   │   └── tools/         # Utility tools
│   ├── search/            # Search functionality
│   └── user/              # User management
└── telegram/              # Telegram bot integration

2.3 Processing Pipeline

PDF Ingestion
- PDF upload handling through NestJS controller
- File validation and storage in AWS S3
- Metadata extraction and job creation
- BullMQ job queue entry creation
SPARK Processing
- Worker-based parallel processing with BullMQ
- PDF to image conversion with optimization
- Image processing with Grok Vision model
- Text extraction with fallback to Claude
- Knowledge graph generation with Grok Language model
- Entity resolution and graph merging
Knowledge Graph Creation
- Entity extraction and typing
- Relationship identification
- Entity resolution across pages
- Graph merging and optimization
- Metadata enrichment
- MongoDB storage of results
Report Generation
- Large graph chunking with token-based splitting
- DeepSeek-based analysis for large contexts
- Context-aware processing
- Markdown report creation
- Progress tracking and error handling

2.4 AI Model Integration

Vision Processing:
- Primary: x-ai/grok-2-vision-1212
- Capabilities: PDF image analysis, text extraction, structure preservation
- Fallback: anthropic/claude-3.7-sonnet
- Error handling with retry and exponential backoff
Language Processing:
- Primary: x-ai/grok-2-1212
- Capabilities: Content analysis, entity extraction, relationship identification
- Fallback: anthropic/claude-3.7-sonnet
- Error handling with retry and exponential backoff
Large Context Processing:
- Model: deepseek/deepseek-r1
- Capabilities: Processing up to 128,000 tokens
- Use case: Knowledge graph analysis and report generation

3. Deep Research Capabilities

3.1 Prompt Engineering System

Core Prompt Templates

Document Analysis Templates

// Image to Markdown prompt
const markdownPrompt = (pageNum) => `
  You are an expert document analyzer and formatter. Extract all text from these images and convert them to clean, structured markdown format.

1. EXTRACTION AND STRUCTURE:
   - Extract all text accurately while maintaining each document's logical structure
   - Identify and properly format headings using markdown # syntax:
     * Main title: # Title (H1)
     * Main sections: ## Section (H2)
     * Subsections: ### Subsection (H3)
     * Further subsections: #### Subsubsection (H4)
   - Remove any numbering schemes from headings (like "1.2.3", "I.A.1") but keep the text
   - Preserve the hierarchical relationship between sections
   - Begin each image's content with a marker in this format: "{{{${pageNum}}}}" (where ${pageNum} is the page number)

2. FORMATTING AND SPECIAL ELEMENTS:
   - Convert tables to proper markdown table syntax with aligned columns
   - Format lists as proper markdown bulleted or numbered lists
   - Format code blocks and technical snippets with appropriate syntax
   - Use *italics* and **bold** where appropriate in the original
   - Format footnotes properly (author affiliations with asterisks, other footnotes with [^1] notation)
   - Preserve mathematical formulas and equations accurately using LaTeX syntax when needed

3. CONTENT ACCURACY:
   - Transcribe all text, numbers, and symbols precisely
   - Maintain exact terminology, technical jargon, and specialized vocabulary
   - Keep proper nouns, names, and titles with correct capitalization
   - Preserve the exact structure of tables, including column alignments
   - Maintain the integrity of diagrams and figures by describing their content

4. CLEANUP AND CLARITY:
   - Remove any PDF artifacts or format remnants
   - Remove any duplicated text from layout issues
   - Clean up any OCR errors that are obviously incorrect
   - Ensure consistent spacing between sections
   - Maintain proper paragraph breaks and section divisions

5. DO NOT:
   - Add any commentary, analysis, or explanations about the content
   - Include watermarks, headers, footers, or page numbers
   - Add any text that isn't from the original document
   - Modify, summarize, or paraphrase the original content
   - Merge content between different images unless they are clearly part of the same section
`;

// Knowledge Graph generation prompt
const kgPrompt = `
  You are an expert knowledge graph creator. Convert the provided markdown text from a single page into a structured knowledge graph by identifying key entities, relationships, and concepts.

1. ENTITY RECOGNITION:
   - Identify key entities (people, organizations, concepts, technologies, methods)
   - Extract attributes and properties of these entities
   - Recognize specialized terminology and technical concepts
   - Identify numerical data, statistics, and measurements
   - Be aware that some entities may be referenced but defined on other pages

2. RELATIONSHIP EXTRACTION:
   - Identify relationships between entities
   - Determine the nature of these relationships (e.g., "is part of", "causes", "implements")
   - Capture hierarchical relationships between concepts
   - Identify temporal relationships and sequences

3. KNOWLEDGE STRUCTURING:
   - Organize extracted information into a coherent knowledge structure
   - Maintain the logical flow and connections between concepts
   - Preserve the context in which entities and relationships appear
   - Identify overarching themes and categories

4. COREFERENCE AND REFERENCES:
   - Identify when the text refers to entities that might be defined elsewhere
   - Include these references even if the full entity definition is not on this page
   - Use the most specific name or identifier available on this page

5. OUTPUT FORMAT:
   - Provide a JSON object representing the knowledge graph with entities and relationships
   - The JSON should follow this structure:
     {
       "entities": [
         {"id": "e1", "type": "concept", "name": "Entity Name", "attributes": {"key": "value"}},
         ...
       ],
       "relationships": [
         {"source": "e1", "target": "e2", "type": "relationship_type", "attributes": {"key": "value"}},
         ...
       ]
     }

Your response should ONLY contain the JSON knowledge graph without any additional text or explanation.
`;

Dynamic Prompt Generation

Context-aware prompts based on document type
Token limit management for large documents
Error handling and recovery prompts

Chain-of-Thought Implementation

Multi-stage processing for complex documents
Context preservation between processing stages
Entity resolution across document pages

3.2 AI Model Integration

Vision Processing (Grok Vision)
- Model: x-ai/grok-2-vision-1212
- Input: Base64-encoded images
- Output: Structured markdown text
- Features: Layout preservation, table recognition, hierarchical structure
Language Processing (Grok Language)
- Model: x-ai/grok-2-1212
- Input: Markdown text
- Output: JSON knowledge graph
- Features: Entity extraction, relationship identification, attribute assignment
Fallback Processing (Claude)
- Model: anthropic/claude-3.7-sonnet
- Activation: Rate limit errors or processing failures
- Features: Compatible input/output format with primary models
Large Context Processing (DeepSeek)
- Model: deepseek/deepseek-r1
- Input: Large knowledge graphs (up to 128,000 tokens)
- Output: Analytical reports and insights
- Features: Context preservation across large documents

3.3 Knowledge Graph Generation

Entity Extraction
- Type classification (concept, person, organization, etc.)
- Attribute identification
- Cross-reference with existing entities
Relationship Identification
- Source and target entity mapping
- Relationship type classification
- Attribute assignment
Entity Resolution
- Cross-page entity matching
- Acronym resolution
- Partial name matching
- Attribute-based disambiguation
Graph Merging
- ID mapping between page graphs
- Attribute consolidation
- Relationship deduplication
- Page reference tracking

4. Technical Implementation

4.1 NestJS Backend Implementation

// app.module.ts - Main application module
@Module({
  imports: [
    ConfigModule.forRoot({
      isGlobal: true,
    }),
    ScheduleModule.forRoot(),
    MongooseModule.forRootAsync({
      useFactory: (configService: ConfigService) => ({
        uri: configService.get('MONGODB_URI'),
        dbName: configService.get('MONGODB_DB', 'my'),
      }),
      inject: [ConfigService],
    }),
    AwsSdkModule.forRootAsync({
      defaultServiceOptions: {
        useFactory: (configService: ConfigService) => ({
          accessKeyId: configService.get('AWS_ACCESS_KEY_ID'),
          secretAccessKey: configService.get('AWS_SECRET_ACCESS_KEY'),
          region: configService.get('AWS_REGION'),
        }),
        inject: [ConfigService],
      },
      services: [S3],
    }),
    BullModule.forRootAsync({
      useFactory: (configService: ConfigService) => ({
        connection: {
          host: configService.get('REDIS_HOST'),
          port: configService.get('REDIS_PORT'),
        },
      }),
      inject: [ConfigService],
    }),
    AuthModule,
    UserModule,
    PdfModule,
    SearchModule,
    ChatsModule,
  ],
  controllers: [AppController],
  providers: [],
})
export class AppModule {}

4.2 Cloudflare AI Provider Implementation

// cloudflare.ai-provider.ts - AI provider implementation
@Injectable()
export class CloudflareAIProvider implements IAIProvider {
  constructor(
    protected readonly configService: ConfigService,
    protected readonly httpService: HttpService,
  ) {}

  service(): Provider {
    return Provider.Cloudflare;
  }

  async query(message: ModelRequest, fallback?: ModelRequest): Promise<string> {
    try {
      const response = await this._apiCall<CloudflareAIResponse>([
        {
          provider: message.provider,
          endpoint: message.route,
          headers: {
            Authorization: `Bearer ${this.configService.getOrThrow('ai.apiKey')}`,
            'Content-Type': 'application/json',
          },
          query: {
            model: message.model,
            messages: message.messages,
            temperature: 0.0,
            max_tokens:
              message.model === 'deepseek/deepseek-r1' ? 30000 : 16_000,
          },
        },
      ]);
      
      if ((response as any).success === false) {
        throw { response };
      }
      
      return response.choices[0].message.content;
    } catch (exception) {
      throw exception;
    }
  }

  protected async _apiCall<T>(data?: any): Promise<T> {
    const timeoutMs = 120_000;
    const controller = new AbortController();
    const timeoutId = setTimeout(() => {
      controller.abort();
    }, timeoutMs);

    try {
      const axiosConfig = {
        method: 'POST',
        url: this.configService.getOrThrow('ai.apiEndpoint'),
        data,
        headers: {
          Authorization: `Bearer ${this.configService.getOrThrow('ai.apiKey')}`,
          'HTTP-Referer': 'https://qwello.ai',
          'X-Title': 'SPARK: SingularityNET PDF Analysis & Reasoning for Knowledge',
          'Content-Type': 'application/json',
        },
        signal: controller.signal,
      };

      const response = await this.httpService.axiosRef.request<T>(axiosConfig);
      return response.data;
    } catch (error: any) {
      if (axios.isCancel(error)) {
        throw new Error(`Request aborted after timeout of ${timeoutMs} ms`);
      }
      throw error;
    } finally {
      clearTimeout(timeoutId);
    }
  }
}

4.3 PDF Processing Service

// pdf.processing.service.ts - PDF processing implementation
@Injectable()
export class PdfProcessingService {
  private readonly logger = new Logger(PdfProcessingService.name);

  // Configuration for retries and rate limiting
  private readonly retries = 1;
  private readonly retryDelay = 3000;
  private readonly useFallbackModel = true;
  private readonly retryBeforeFallback = 2;

  constructor(
    protected readonly pdfService: PdfService,
    protected readonly aiService: AIService,
    @InjectQueue('pdf') protected readonly pdfQueue: Queue,
    protected readonly pdfKnowledgeGraphRepository: PdfKnowledgeGraphRepository,
    protected readonly pdfKnowledgeGraphResultRepository: PdfKnowledgeGraphResultRepository,
  ) {}

  public async processPdf(
    pdfBuffer: Buffer,
    user: User,
  ): Promise<PdfKnowledgeGraph> {
    const pageBuffers = await this.pdfService.pdfToCompressedImages(pdfBuffer);
    const totalPages = pageBuffers.length;
    const pdfPages: PdfPage[] = [];

    for (let i = 0; i < totalPages; i++) {
      const pageBuffer = pageBuffers[i];
      const pageBase64 = pageBuffer.toString('base64');
      pdfPages.push({ page: i + 1, data: pageBase64 });
    }

    const knowledgeGraph = await this.createKnowledgeGraph(totalPages, user);
    const chunks = chunkArray(pdfPages, 10);
    const jobs = [];
    
    for (let i = 0; i < chunks.length; i++) {
      const chunk = chunks[i];
      const pdfProcessingJob = await this.pdfQueue.add('pdf', {
        pdfId: knowledgeGraph.id,
        pages: chunk,
      });

      this.logger.log(
        `Added job ${pdfProcessingJob.id} for chunk ${i + 1}/${chunks.length}`,
      );
      jobs.push(pdfProcessingJob.id);
    }

    knowledgeGraph.jobs = jobs.map((jobId) => {
      return { jobId, status: KgStatus.Processing } as PdfJob;
    });

    await this.pdfKnowledgeGraphRepository.update(knowledgeGraph);
    return knowledgeGraph;
  }
}

4.4 Knowledge Graph Visualization

// GraphVisualization.tsx - Interactive graph visualization
const GraphVisualization = React.forwardRef<
  GraphVisualizationRef,
  GraphVisualizationProps
>(
  (
    { data, onNodeSelect, onNodeHover, className = "" },
    ref: ForwardedRef<GraphVisualizationRef>
  ) => {
    const containerRef = useRef<HTMLDivElement>(null);
    const networkRef = useRef<Network | null>(null);
    const nodesRef = useRef<DataSet<NetworkNode> | null>(null);
    const edgesRef = useRef<DataSet<NetworkEdge> | null>(null);

    // Get color for entity type
    const getTypeColor = (type?: string): string =>
      type
        ? typeColors[type as keyof typeof typeColors] || typeColors.default
        : typeColors.default;

    // Create or update network
    const updateNetwork = useCallback((): void => {
      if (!data?.entities || !data?.relationships || !containerRef.current)
        return;

      try {
        // Create nodes array for vis.js
        const nodes = data.entities.map((entity) => {
          const type = entity.type || "default";
          const color = getTypeColor(type);

          return {
            id: entity.id,
            label: entity.name || entity.id,
            color: color,
            font: {
              color: getContrastColor(color),
            },
            originalData: entity,
          };
        });

        // Create edges array for vis.js
        const edges = data.relationships.map((rel, index) => ({
          id: `e${index}`,
          from: rel.source,
          to: rel.target,
          label: rel.type || "",
          originalData: rel,
        }));

        const options = {
          nodes: {
            shape: "box",
            margin: { top: 10, right: 10, bottom: 10, left: 10 },
            font: { size: 14 },
          },
          edges: {
            width: 1.5,
            length: 200,
            smooth: {
              enabled: true,
              type: "continuous",
              roundness: 0.3,
            },
            font: {
              size: 12,
              align: "middle",
            },
          },
          physics: {
            solver: "forceAtlas2Based",
            enabled: true,
            forceAtlas2Based: {
              gravitationalConstant: -50,
              centralGravity: 0.01,
              springLength: 100,
              springConstant: 0.08,
              damping: 0.4,
            },
          },
        };

        // Update existing datasets or create new ones
        if (nodesRef.current && edgesRef.current) {
          nodesRef.current.clear();
          edgesRef.current.clear();
          nodesRef.current.add(nodes);
          edgesRef.current.add(edges);
        } else {
          nodesRef.current = new DataSet(nodes);
          edgesRef.current = new DataSet(edges);
          
          // Create network
          networkRef.current = new Network(
            containerRef.current,
            {
              nodes: nodesRef.current,
              edges: edgesRef.current,
            },
            options
          );
          
          setupEventListeners();
        }
      } catch (error) {
        console.warn("Error initializing network:", error);
      }
    }, [data]);

    return (
      <div
        ref={containerRef}
        className={`w-full bg-black800 ${className}`}
        style={{
          height: "calc(100vh - 164px)",
          position: "relative",
          overflow: "hidden",
        }}
      />
    );
  }
);

5. System Features

5.1 PDF Processing

Multi-model AI approach: Uses Grok Vision for image-to-text conversion and Grok Language for knowledge graph generation
Fallback mechanisms: Automatically switches to Claude when primary models encounter rate limits or errors
Parallel processing: Worker-based architecture with BullMQ for efficient processing of large documents
Progress tracking: Real-time updates via WebSockets during document processing
Error handling: Sophisticated retry logic with exponential backoff and fallback models

5.2 Knowledge Graph Generation

Entity extraction: Identifies key entities, their types, and attributes
Relationship mapping: Determines connections between entities with relationship types
Entity resolution: Cross-page entity resolution to maintain consistency
Metadata enrichment: Adds document metadata, page references, and contextual information
MongoDB storage: Efficient storage and retrieval of graph data

5.3 Visualization and Interaction

Interactive graph: Force-directed layout with zoom, pan, and filtering capabilities
Entity filtering: Filter by entity type, connection count, and search terms
Entity details: View detailed information about selected entities
Report integration: View AI-generated reports about the knowledge graph content
TypeScript implementation: Enhanced type safety and developer experience

6. Development Roadmap (Q2 2025)

6.1 Phase 1: Core Enhancement (March 2025 - Current)

Week 1-2: TypeScript Migration

Backend TypeScript Migration
- Set up TypeScript configuration
- Migrate Node.js modules to NestJS
- Add type definitions
Frontend TypeScript Migration
- Update React components
- Add component prop types
- Add API interface definitions
Type System Implementation
- Core type definitions
- API interfaces
- Utility types

Week 3-4: Query Pipeline

SPARK Engine implementation with NestJS
Worker Pool architecture with BullMQ
PDF processing pipeline
Knowledge Graph generation

6.2 Phase 2: MeTTa Integration (April 2025)

Week 1-2: Core MeTTa Setup

Week 3-4: MeTTa-Motto Integration

6.3 Phase 3: Advanced Features (May 2025)

Week 1-2: Knowledge Processing

Week 3-4: Query Enhancement

6.4 Phase 4: Production Readiness (June 2025)

Week 1-2: System Integration

Document processing
Knowledge extraction
Graph generation
Query system

Week 3-4: Final Release

Performance optimization
Production deployment
Documentation
Monitoring setup

7. Technical Capabilities

7.1 Performance

Parallel query processing with BullMQ
Efficient graph operations with optimized algorithms
Real-time updates via WebSockets
Memory optimization with chunking strategies
Response streaming for large results

7.2 Scalability

Distributed processing with BullMQ
Resource management with worker pools
Concurrent operations with rate limiting
Load balancing with Redis
Cache optimization for repeated queries

8. Security and Monitoring

8.1 Security Features

API key management with environment variables
Request validation with NestJS pipes
Rate limiting with configurable thresholds
Data encryption for sensitive information
Access control with JWT authentication

8.2 System Monitoring

Performance metrics with Prometheus
Error tracking with structured logging
Resource utilization monitoring
API health checks
Usage analytics

Conclusion

Qwello represents a significant advancement in document analysis and knowledge processing technology. The current implementation demonstrates robust PDF processing capabilities through the NestJS-based SPARK engine, efficient parallel processing via the BullMQ job queue architecture, and sophisticated knowledge graph generation using state-of-the-art AI models (Grok and Claude) through Cloudflare AI integration.

The system's modular architecture allows for flexible deployment and scaling, while the MongoDB database provides efficient storage and retrieval of knowledge graph data. The interactive visualization capabilities with vis-network.js enable users to explore and interact with complex knowledge graphs in an intuitive way.

While the system is currently focused on PDF processing and knowledge graph generation, the planned integration of MeTTa, MORK, and Hyperon will expand its capabilities into advanced reasoning and knowledge synthesis. With a clear development roadmap and strong technical foundation, Qwello is well-positioned to evolve into a comprehensive platform for document analysis, knowledge processing, and research enhancement.

PreviousProduct Whitepaper NextIntroduction

Last updated 2 months ago