Technical Whitepaper

Advanced Knowledge and Intelligent Search

Version 1.0 | March 2025

---

Abstract

Qwello represents a significant advancement in document analysis, knowledge processing, and intelligent search, combining sophisticated PDF analysis with advanced AI capabilities. The platform implements a NestJS-based architecture with Cloudflare AI integration, supporting multiple AI models including Grok and Claude. The system features a robust PDF processing pipeline with parallel worker architecture, sophisticated knowledge graph generation with entity resolution, and interactive visualization capabilities. This unique combination enables deep research capabilities, interactive knowledge exploration, and intelligent search through a unified system.

Current Implementation Status

Completed Features

• Core Infrastructure:

  • ✓ NestJS-based modular architecture

  • ✓ MongoDB integration for data persistence

  • ✓ BullMQ for job queue processing

  • ✓ AWS S3 integration for file storage

  • ✓ Redis for caching and job queue management

• Knowledge Graph Foundation:

  • ✓ KG report generation with chunking for large graphs

  • ✓ DeepSeek integration for analysis

  • ✓ Entity relationship mapping with JSON structure

  • ✓ Interactive visualization with vis-network.js

• PDF Processing Pipeline:

  • ✓ SPARK Engine (SingularityNET PDF Analysis & Reasoning for Knowledge)

  • ✓ Worker-based parallel processing with BullMQ

  • ✓ Multi-model AI approach with fallback mechanisms

  • ✓ PDF to image conversion with optimization

  • ✓ Rate limiting and error handling with exponential backoff

• AI Integration:

  • ✓ Cloudflare AI provider implementation

  • ✓ Primary: x-ai/grok-2-vision-1212 (Vision)

  • ✓ Primary: x-ai/grok-2-1212 (Language)

  • ✓ Fallback: anthropic/claude-3.7-sonnet

  • ✓ Large context: deepseek/deepseek-r1

• API Services:

  • ✓ NestJS controllers with versioning

  • ✓ Rate-limited endpoints with configurable limits

  • ✓ Worker pool with CPU-based scaling

  • ✓ WebSocket-based progress tracking

• Frontend Interface:

  • ✓ React-based UI with TypeScript migration in progress

  • ✓ Interactive knowledge graph visualization with vis-network.js

  • ✓ Entity filtering and search capabilities

  • ✓ Document library management

In Development

• TypeScript Migration (In Progress)

  • Frontend: React components being converted

  • Backend: NestJS implementation with TypeScript

• MeTTa Integration (Planned for April 2025)

• MORK Engine Integration

• Hyperon Runtime Integration

• SERP API Integration

1. Introduction

1.1 The Knowledge Processing Challenge

Organizations face increasing challenges in research, document analysis, and knowledge discovery:

• Complex information retrieval needs

• Deep semantic understanding requirements

• Integration of multiple knowledge sources

• Domain-specific research demands

• Real-time knowledge synthesis needs

1.2 The Qwello Solution

Qwello addresses these challenges through a sophisticated integration of technologies:

• NestJS-based modular architecture

• Advanced PDF processing pipeline

• Multi-model AI analysis with Cloudflare integration

• Deep research capabilities

• Knowledge graph generation with entity resolution

• Interactive visualization with vis-network.js

2. Core Architecture

2.1 System Components

2.2 NestJS Module Structure

src/
├── app.module.ts            # Main application module
├── common/                  # Common utilities and helpers
├── config/                  # Configuration management
├── interfaces/              # TypeScript interfaces
├── modules/
│   ├── ai/                  # AI integration module
│   │   ├── ai.module.ts
│   │   ├── ai.config.ts
│   │   ├── types.ts
│   │   ├── prompts.ts
│   │   ├── ai-providers/    # AI provider implementations
│   │   ├── controllers/     # API endpoints
│   │   ├── enum/           # Enumerations
│   │   └── services/       # AI services
│   ├── auth/               # Authentication module
│   ├── chats/              # Chat functionality
│   ├── pdf/                # PDF processing module
│   │   ├── pdf.module.ts
│   │   ├── pdf.config.ts
│   │   ├── types.ts
│   │   ├── controllers/    # API endpoints
│   │   ├── entities/       # MongoDB schemas
│   │   ├── enum/          # Enumerations
│   │   ├── processors/    # BullMQ processors
│   │   ├── repositories/  # Data access layer
│   │   ├── services/      # Business logic
│   │   └── tools/         # Utility tools
│   ├── search/            # Search functionality
│   └── user/              # User management
└── telegram/              # Telegram bot integration

2.3 Processing Pipeline

1. PDF Ingestion

   • PDF upload handling through NestJS controller

   • File validation and storage in AWS S3

   • Metadata extraction and job creation

   • BullMQ job queue entry creation

2. SPARK Processing

   • Worker-based parallel processing with BullMQ

   • PDF to image conversion with optimization

   • Image processing with Grok Vision model

   • Text extraction with fallback to Claude

   • Knowledge graph generation with Grok Language model

   • Entity resolution and graph merging

3. Knowledge Graph Creation

   • Entity extraction and typing

   • Relationship identification

   • Entity resolution across pages

   • Graph merging and optimization

   • Metadata enrichment

   • MongoDB storage of results

4. Report Generation

   • Large graph chunking with token-based splitting

   • DeepSeek-based analysis for large contexts

   • Context-aware processing

   • Markdown report creation

   • Progress tracking and error handling

2.4 AI Model Integration

• Vision Processing:

  • Primary: x-ai/grok-2-vision-1212

  • Capabilities: PDF image analysis, text extraction, structure preservation

  • Fallback: anthropic/claude-3.7-sonnet

  • Error handling with retry and exponential backoff

• Language Processing:

  • Primary: x-ai/grok-2-1212

  • Capabilities: Content analysis, entity extraction, relationship identification

  • Fallback: anthropic/claude-3.7-sonnet

  • Error handling with retry and exponential backoff

• Large Context Processing:

  • Model: deepseek/deepseek-r1

  • Capabilities: Processing up to 128,000 tokens

  • Use case: Knowledge graph analysis and report generation

3. Deep Research Capabilities

3.1 Prompt Engineering System

Core Prompt Templates

• Document Analysis Templates

  // Image to Markdown prompt
  const markdownPrompt = (pageNum) => `
    You are an expert document analyzer and formatter. Extract all text from these images and convert them to clean, structured markdown format.
  
  1. EXTRACTION AND STRUCTURE:
     - Extract all text accurately while maintaining each document's logical structure
     - Identify and properly format headings using markdown # syntax:
       * Main title: # Title (H1)
       * Main sections: ## Section (H2)
       * Subsections: ### Subsection (H3)
       * Further subsections: #### Subsubsection (H4)
     - Remove any numbering schemes from headings (like "1.2.3", "I.A.1") but keep the text
     - Preserve the hierarchical relationship between sections
     - Begin each image's content with a marker in this format: "{{{${pageNum}}}}" (where ${pageNum} is the page number)
  
  2. FORMATTING AND SPECIAL ELEMENTS:
     - Convert tables to proper markdown table syntax with aligned columns
     - Format lists as proper markdown bulleted or numbered lists
     - Format code blocks and technical snippets with appropriate syntax
     - Use *italics* and **bold** where appropriate in the original
     - Format footnotes properly (author affiliations with asterisks, other footnotes with [^1] notation)
     - Preserve mathematical formulas and equations accurately using LaTeX syntax when needed
  
  3. CONTENT ACCURACY:
     - Transcribe all text, numbers, and symbols precisely
     - Maintain exact terminology, technical jargon, and specialized vocabulary
     - Keep proper nouns, names, and titles with correct capitalization
     - Preserve the exact structure of tables, including column alignments
     - Maintain the integrity of diagrams and figures by describing their content
  
  4. CLEANUP AND CLARITY:
     - Remove any PDF artifacts or format remnants
     - Remove any duplicated text from layout issues
     - Clean up any OCR errors that are obviously incorrect
     - Ensure consistent spacing between sections
     - Maintain proper paragraph breaks and section divisions
  
  5. DO NOT:
     - Add any commentary, analysis, or explanations about the content
     - Include watermarks, headers, footers, or page numbers
     - Add any text that isn't from the original document
     - Modify, summarize, or paraphrase the original content
     - Merge content between different images unless they are clearly part of the same section
  `;
  
  // Knowledge Graph generation prompt
  const kgPrompt = `
    You are an expert knowledge graph creator. Convert the provided markdown text from a single page into a structured knowledge graph by identifying key entities, relationships, and concepts.
  
  1. ENTITY RECOGNITION:
     - Identify key entities (people, organizations, concepts, technologies, methods)
     - Extract attributes and properties of these entities
     - Recognize specialized terminology and technical concepts
     - Identify numerical data, statistics, and measurements
     - Be aware that some entities may be referenced but defined on other pages
  
  2. RELATIONSHIP EXTRACTION:
     - Identify relationships between entities
     - Determine the nature of these relationships (e.g., "is part of", "causes", "implements")
     - Capture hierarchical relationships between concepts
     - Identify temporal relationships and sequences
  
  3. KNOWLEDGE STRUCTURING:
     - Organize extracted information into a coherent knowledge structure
     - Maintain the logical flow and connections between concepts
     - Preserve the context in which entities and relationships appear
     - Identify overarching themes and categories
  
  4. COREFERENCE AND REFERENCES:
     - Identify when the text refers to entities that might be defined elsewhere
     - Include these references even if the full entity definition is not on this page
     - Use the most specific name or identifier available on this page
  
  5. OUTPUT FORMAT:
     - Provide a JSON object representing the knowledge graph with entities and relationships
     - The JSON should follow this structure:
       {
         "entities": [
           {"id": "e1", "type": "concept", "name": "Entity Name", "attributes": {"key": "value"}},
           ...
         ],
         "relationships": [
           {"source": "e1", "target": "e2", "type": "relationship_type", "attributes": {"key": "value"}},
           ...
         ]
       }
  
  Your response should ONLY contain the JSON knowledge graph without any additional text or explanation.
  `;

Dynamic Prompt Generation

• Context-aware prompts based on document type

• Token limit management for large documents

• Error handling and recovery prompts

Chain-of-Thought Implementation

• Multi-stage processing for complex documents

• Context preservation between processing stages

• Entity resolution across document pages

3.2 AI Model Integration

• Vision Processing (Grok Vision)

  • Model: x-ai/grok-2-vision-1212

  • Input: Base64-encoded images

  • Output: Structured markdown text

  • Features: Layout preservation, table recognition, hierarchical structure

• Language Processing (Grok Language)

  • Model: x-ai/grok-2-1212

  • Input: Markdown text

  • Output: JSON knowledge graph

  • Features: Entity extraction, relationship identification, attribute assignment

• Fallback Processing (Claude)

  • Model: anthropic/claude-3.7-sonnet

  • Activation: Rate limit errors or processing failures

  • Features: Compatible input/output format with primary models

• Large Context Processing (DeepSeek)

  • Model: deepseek/deepseek-r1

  • Input: Large knowledge graphs (up to 128,000 tokens)

  • Output: Analytical reports and insights

  • Features: Context preservation across large documents

3.3 Knowledge Graph Generation

• Entity Extraction

  • Type classification (concept, person, organization, etc.)

  • Attribute identification

  • Cross-reference with existing entities

• Relationship Identification

  • Source and target entity mapping

  • Relationship type classification

  • Attribute assignment

• Entity Resolution

  • Cross-page entity matching

  • Acronym resolution

  • Partial name matching

  • Attribute-based disambiguation

• Graph Merging

  • ID mapping between page graphs

  • Attribute consolidation

  • Relationship deduplication

  • Page reference tracking

4. Technical Implementation

4.1 NestJS Backend Implementation

// app.module.ts - Main application module
@Module({
  imports: [
    ConfigModule.forRoot({
      isGlobal: true,
    }),
    ScheduleModule.forRoot(),
    MongooseModule.forRootAsync({
      useFactory: (configService: ConfigService) => ({
        uri: configService.get('MONGODB_URI'),
        dbName: configService.get('MONGODB_DB', 'my'),
      }),
      inject: [ConfigService],
    }),
    AwsSdkModule.forRootAsync({
      defaultServiceOptions: {
        useFactory: (configService: ConfigService) => ({
          accessKeyId: configService.get('AWS_ACCESS_KEY_ID'),
          secretAccessKey: configService.get('AWS_SECRET_ACCESS_KEY'),
          region: configService.get('AWS_REGION'),
        }),
        inject: [ConfigService],
      },
      services: [S3],
    }),
    BullModule.forRootAsync({
      useFactory: (configService: ConfigService) => ({
        connection: {
          host: configService.get('REDIS_HOST'),
          port: configService.get('REDIS_PORT'),
        },
      }),
      inject: [ConfigService],
    }),
    AuthModule,
    UserModule,
    PdfModule,
    SearchModule,
    ChatsModule,
  ],
  controllers: [AppController],
  providers: [],
})
export class AppModule {}

4.2 Cloudflare AI Provider Implementation

// cloudflare.ai-provider.ts - AI provider implementation
@Injectable()
export class CloudflareAIProvider implements IAIProvider {
  constructor(
    protected readonly configService: ConfigService,
    protected readonly httpService: HttpService,
  ) {}

  service(): Provider {
    return Provider.Cloudflare;
  }

  async query(message: ModelRequest, fallback?: ModelRequest): Promise<string> {
    try {
      const response = await this._apiCall<CloudflareAIResponse>([
        {
          provider: message.provider,
          endpoint: message.route,
          headers: {
            Authorization: `Bearer ${this.configService.getOrThrow('ai.apiKey')}`,
            'Content-Type': 'application/json',
          },
          query: {
            model: message.model,
            messages: message.messages,
            temperature: 0.0,
            max_tokens:
              message.model === 'deepseek/deepseek-r1' ? 30000 : 16_000,
          },
        },
      ]);
      
      if ((response as any).success === false) {
        throw { response };
      }
      
      return response.choices[0].message.content;
    } catch (exception) {
      throw exception;
    }
  }

  protected async _apiCall<T>(data?: any): Promise<T> {
    const timeoutMs = 120_000;
    const controller = new AbortController();
    const timeoutId = setTimeout(() => {
      controller.abort();
    }, timeoutMs);

    try {
      const axiosConfig = {
        method: 'POST',
        url: this.configService.getOrThrow('ai.apiEndpoint'),
        data,
        headers: {
          Authorization: `Bearer ${this.configService.getOrThrow('ai.apiKey')}`,
          'HTTP-Referer': 'https://qwello.ai',
          'X-Title': 'SPARK: SingularityNET PDF Analysis & Reasoning for Knowledge',
          'Content-Type': 'application/json',
        },
        signal: controller.signal,
      };

      const response = await this.httpService.axiosRef.request<T>(axiosConfig);
      return response.data;
    } catch (error: any) {
      if (axios.isCancel(error)) {
        throw new Error(`Request aborted after timeout of ${timeoutMs} ms`);
      }
      throw error;
    } finally {
      clearTimeout(timeoutId);
    }
  }
}

4.3 PDF Processing Service

// pdf.processing.service.ts - PDF processing implementation
@Injectable()
export class PdfProcessingService {
  private readonly logger = new Logger(PdfProcessingService.name);

  // Configuration for retries and rate limiting
  private readonly retries = 1;
  private readonly retryDelay = 3000;
  private readonly useFallbackModel = true;
  private readonly retryBeforeFallback = 2;

  constructor(
    protected readonly pdfService: PdfService,
    protected readonly aiService: AIService,
    @InjectQueue('pdf') protected readonly pdfQueue: Queue,
    protected readonly pdfKnowledgeGraphRepository: PdfKnowledgeGraphRepository,
    protected readonly pdfKnowledgeGraphResultRepository: PdfKnowledgeGraphResultRepository,
  ) {}

  public async processPdf(
    pdfBuffer: Buffer,
    user: User,
  ): Promise<PdfKnowledgeGraph> {
    const pageBuffers = await this.pdfService.pdfToCompressedImages(pdfBuffer);
    const totalPages = pageBuffers.length;
    const pdfPages: PdfPage[] = [];

    for (let i = 0; i < totalPages; i++) {
      const pageBuffer = pageBuffers[i];
      const pageBase64 = pageBuffer.toString('base64');
      pdfPages.push({ page: i + 1, data: pageBase64 });
    }

    const knowledgeGraph = await this.createKnowledgeGraph(totalPages, user);
    const chunks = chunkArray(pdfPages, 10);
    const jobs = [];
    
    for (let i = 0; i < chunks.length; i++) {
      const chunk = chunks[i];
      const pdfProcessingJob = await this.pdfQueue.add('pdf', {
        pdfId: knowledgeGraph.id,
        pages: chunk,
      });

      this.logger.log(
        `Added job ${pdfProcessingJob.id} for chunk ${i + 1}/${chunks.length}`,
      );
      jobs.push(pdfProcessingJob.id);
    }

    knowledgeGraph.jobs = jobs.map((jobId) => {
      return { jobId, status: KgStatus.Processing } as PdfJob;
    });

    await this.pdfKnowledgeGraphRepository.update(knowledgeGraph);
    return knowledgeGraph;
  }
}

4.4 Knowledge Graph Visualization

// GraphVisualization.tsx - Interactive graph visualization
const GraphVisualization = React.forwardRef<
  GraphVisualizationRef,
  GraphVisualizationProps
>(
  (
    { data, onNodeSelect, onNodeHover, className = "" },
    ref: ForwardedRef<GraphVisualizationRef>
  ) => {
    const containerRef = useRef<HTMLDivElement>(null);
    const networkRef = useRef<Network | null>(null);
    const nodesRef = useRef<DataSet<NetworkNode> | null>(null);
    const edgesRef = useRef<DataSet<NetworkEdge> | null>(null);

    // Get color for entity type
    const getTypeColor = (type?: string): string =>
      type
        ? typeColors[type as keyof typeof typeColors] || typeColors.default
        : typeColors.default;

    // Create or update network
    const updateNetwork = useCallback((): void => {
      if (!data?.entities || !data?.relationships || !containerRef.current)
        return;

      try {
        // Create nodes array for vis.js
        const nodes = data.entities.map((entity) => {
          const type = entity.type || "default";
          const color = getTypeColor(type);

          return {
            id: entity.id,
            label: entity.name || entity.id,
            color: color,
            font: {
              color: getContrastColor(color),
            },
            originalData: entity,
          };
        });

        // Create edges array for vis.js
        const edges = data.relationships.map((rel, index) => ({
          id: `e${index}`,
          from: rel.source,
          to: rel.target,
          label: rel.type || "",
          originalData: rel,
        }));

        const options = {
          nodes: {
            shape: "box",
            margin: { top: 10, right: 10, bottom: 10, left: 10 },
            font: { size: 14 },
          },
          edges: {
            width: 1.5,
            length: 200,
            smooth: {
              enabled: true,
              type: "continuous",
              roundness: 0.3,
            },
            font: {
              size: 12,
              align: "middle",
            },
          },
          physics: {
            solver: "forceAtlas2Based",
            enabled: true,
            forceAtlas2Based: {
              gravitationalConstant: -50,
              centralGravity: 0.01,
              springLength: 100,
              springConstant: 0.08,
              damping: 0.4,
            },
          },
        };

        // Update existing datasets or create new ones
        if (nodesRef.current && edgesRef.current) {
          nodesRef.current.clear();
          edgesRef.current.clear();
          nodesRef.current.add(nodes);
          edgesRef.current.add(edges);
        } else {
          nodesRef.current = new DataSet(nodes);
          edgesRef.current = new DataSet(edges);
          
          // Create network
          networkRef.current = new Network(
            containerRef.current,
            {
              nodes: nodesRef.current,
              edges: edgesRef.current,
            },
            options
          );
          
          setupEventListeners();
        }
      } catch (error) {
        console.warn("Error initializing network:", error);
      }
    }, [data]);

    return (
      <div
        ref={containerRef}
        className={`w-full bg-black800 ${className}`}
        style={{
          height: "calc(100vh - 164px)",
          position: "relative",
          overflow: "hidden",
        }}
      />
    );
  }
);

5. System Features

5.1 PDF Processing

• Multi-model AI approach: Uses Grok Vision for image-to-text conversion and Grok Language for knowledge graph generation

• Fallback mechanisms: Automatically switches to Claude when primary models encounter rate limits or errors

• Parallel processing: Worker-based architecture with BullMQ for efficient processing of large documents

• Progress tracking: Real-time updates via WebSockets during document processing

• Error handling: Sophisticated retry logic with exponential backoff and fallback models

5.2 Knowledge Graph Generation

• Entity extraction: Identifies key entities, their types, and attributes

• Relationship mapping: Determines connections between entities with relationship types

• Entity resolution: Cross-page entity resolution to maintain consistency

• Metadata enrichment: Adds document metadata, page references, and contextual information

• MongoDB storage: Efficient storage and retrieval of graph data

5.3 Visualization and Interaction

• Interactive graph: Force-directed layout with zoom, pan, and filtering capabilities

• Entity filtering: Filter by entity type, connection count, and search terms

• Entity details: View detailed information about selected entities

• Report integration: View AI-generated reports about the knowledge graph content

• TypeScript implementation: Enhanced type safety and developer experience

6. Development Roadmap (Q2 2025)

6.1 Phase 1: Core Enhancement (March 2025 - Current)

Week 1-2: TypeScript Migration

• Backend TypeScript Migration

  • Set up TypeScript configuration

  • Migrate Node.js modules to NestJS

  • Add type definitions

• Frontend TypeScript Migration

  • Update React components

  • Add component prop types

  • Add API interface definitions

• Type System Implementation

  • Core type definitions

  • API interfaces

  • Utility types

Week 3-4: Query Pipeline

• SPARK Engine implementation with NestJS

• Worker Pool architecture with BullMQ

• PDF processing pipeline

• Knowledge Graph generation

6.2 Phase 2: MeTTa Integration (April 2025)

Week 1-2: Core MeTTa Setup

• Atomspace System Implementation

  • Symbol implementation

  • Expression handling

  • Variable system

  • Grounded atoms

• MORK & Hyperon Integration

  • WASM bridge setup

  • Node.js native modules

  • Type system integration

Week 3-4: MeTTa-Motto Integration

• LLM Agent Implementation

  • ChatGPT integration

  • Claude integration

  • OpenRouter support

• Dialog System Creation

  • History management

  • Context handling

  • Response synthesis

• Retrieval Capabilities

  • Document chunking

  • Embedding generation

  • Vector search

6.3 Phase 3: Advanced Features (May 2025)

Week 1-2: Knowledge Processing

• MeTTa Script Implementation

  • Script execution engine

  • Template system

  • Function calls

• LangChain Integration

  • Model support

  • Tool integration

  • Agent execution

Week 3-4: Query Enhancement

• Knowledge Graph Queries

  • Pattern matching

  • Inference rules

  • Result synthesis

• Streaming Support

  • Real-time responses

  • Progress tracking

  • Error handling

6.4 Phase 4: Production Readiness (June 2025)

Week 1-2: System Integration

• Document processing

• Knowledge extraction

• Graph generation

• Query system

Week 3-4: Final Release

• Performance optimization

• Production deployment

• Documentation

• Monitoring setup

7. Technical Capabilities

7.1 Performance

• Parallel query processing with BullMQ

• Efficient graph operations with optimized algorithms

• Real-time updates via WebSockets

• Memory optimization with chunking strategies

• Response streaming for large results

7.2 Scalability

• Distributed processing with BullMQ

• Resource management with worker pools

• Concurrent operations with rate limiting

• Load balancing with Redis

• Cache optimization for repeated queries

8. Security and Monitoring

8.1 Security Features

• API key management with environment variables

• Request validation with NestJS pipes

• Rate limiting with configurable thresholds

• Data encryption for sensitive information

• Access control with JWT authentication

8.2 System Monitoring

• Performance metrics with Prometheus

• Error tracking with structured logging

• Resource utilization monitoring

• API health checks

• Usage analytics

Conclusion

Qwello represents a significant advancement in document analysis and knowledge processing technology. The current implementation demonstrates robust PDF processing capabilities through the NestJS-based SPARK engine, efficient parallel processing via the BullMQ job queue architecture, and sophisticated knowledge graph generation using state-of-the-art AI models (Grok and Claude) through Cloudflare AI integration.

The system's modular architecture allows for flexible deployment and scaling, while the MongoDB database provides efficient storage and retrieval of knowledge graph data. The interactive visualization capabilities with vis-network.js enable users to explore and interact with complex knowledge graphs in an intuitive way.

While the system is currently focused on PDF processing and knowledge graph generation, the planned integration of MeTTa, MORK, and Hyperon will expand its capabilities into advanced reasoning and knowledge synthesis. With a clear development roadmap and strong technical foundation, Qwello is well-positioned to evolve into a comprehensive platform for document analysis, knowledge processing, and research enhancement.