espiritismo-rag-system/docs/superpowers/plans/2025-11-01-espiritismo-rag-system.md

Sistema RAG Espiritismo - Implementação

Para workers agenticos: SUB-SKILL OBRIGATÓRIA: Use superpowers:subagent-driven-development (recomendado) ou superpowers:executing-plans para implementar este plano tarefa por tarefa. Os passos usam sintaxe de checkbox (- [ ]) para rastreamento.

Objetivo: Desenvolver um sistema RAG (Retrieval-Augmented Generation) completo para consulta em linguagem natural aos livros de espiritismo em PDF, com interface web rica e integração com Telegram para agendamento de mensagens de conteúdo diário.

Arquitetura: Stack minimalista com GPT4All (Llama 3.2 3B) + LangChain + Qdrant para vectores, FastAPI para backend, HTML/CSS/JS para frontend, e pyTelegramBotAPI para bot com Celery para agendamento.

Stack Tecnológica: GPT4All, LangChain, Qdrant, FastAPI, pyTelegramBotAPI, Celery, Redis, Docker Compose

Spec: docs/superpowers/specs/2025-11-01-espiritismo-rag-system-design.md

Restrições Globais

• CPU-only, sem GPU (máximo 8GB RAM)
• Armazenamento máximo: ~5GB (modelo 2GB + vetores 2GB + aplicação)
• Todos os serviços em Docker
• Interface web simples mas rica, responsiva e amigável
• Bot Telegram funcional com comandos de agendamento
• Suporte para upload e agregação futura de livros PDF

Foco de Revisão

1. Consulta web: Usuário faz pergunta → sistema retorna resposta com citações e fontes
2. Upload PDF: Interface web deve permitir upload de novos PDFs → sistema indexa automaticamente
3. Agendamento Telegram: Bot deve permitir configurar horários de envio → mensagens enviadas corretamente
4. Indexação RAG: Sistema deve processar 6 livros existentes → gerar embeddings e estrutura vetorial
5. Integração web-bot: Configurações feitas no web devem ser acessíveis via bot

---

Estrutura de Arquivos

espiritismo-rag/
├── backend/
│   ├── app/
│   │   ├── main.py                 # FastAPI application
│   │   ├── models.py               # Pydantic models
│   │   ├── services/
│   │   │   ├── rag_service.py      # RAG pipeline
│   │   │   ├── pdf_processor.py    # PDF processing
│   │   │   └── bot_service.py      # Telegram bot integration
│   │   ├── utils/
│   │   │   ├── embeddings.py       # Embedding generation
│   │   │   └── schedule_manager.py # Schedule management
│   │   ├── templates/
│   │   │   └── index.html          # Frontend interface
│   │   └── tasks/
│   │       └── celery_tasks.py     # Celery tasks
│   ├── scripts/
│   │   └── index_books.py          # Initial index script
│   ├── Dockerfile
│   ├── requirements.txt
│   └── .env.example
├── frontend/
│   └── static/
│       ├── css/
│       │   └── style.css
│       └── js/
│           └── script.js
├── docker-compose.yml
├── .env
├── celerybeat-schedule.json
├── schedules.json
└── README.md

---

Tarefa 1: Setup Inicial e Estrutura do Projeto

Arquivos:

• Criar: backend/app/main.py
• Criar: backend/app/models.py
• Criar: backend/services/rag_service.py
• Criar: backend/services/pdf_processor.py
• Criar: backend/services/bot_service.py
• Criar: backend/utils/embeddings.py
• Criar: backend/utils/schedule_manager.py
• Criar: backend/templates/index.html
• Criar: backend/tasks/celery_tasks.py
• Criar: backend/scripts/index_books.py
• Criar: backend/Dockerfile
• Criar: backend/requirements.txt
• Criar: docker-compose.yml
• Criar: frontend/static/css/style.css
• Criar: frontend/static/js/script.js
• Criar: backend/.env.example
• Criar: .env
• Criar: README.md

Interfaces:

• Consumes: N/A (setup inicial)

• Produces: Estrutura completa do projeto, Docker Compose configurado

• Passo 1: Criar estrutura de diretórios e arquivos básicos

mkdir -p backend/app/{models,services,utils,templates,tasks}
mkdir -p backend/scripts
mkdir -p frontend/static/{css,js}

• Passo 2: Criar docker-compose.yml

version: '3.8'

services:
  qdrant:
    image: qdrant/qdrant:latest
    container_name: espiritismo-qdrant
    ports:
      - "6333:6333"
    volumes:
      - qdrant_data:/qdrant/storage
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    container_name: espiritismo-redis
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    restart: unless-stopped

  web:
    build: ./backend
    container_name: espiritismo-web
    ports:
      - "8000:8000"
    volumes:
      - ./PDF:/app/uploads:ro
      - ./backend/schedules.json:/app/schedules.json
    environment:
      - QDRANT_URL=http://qdrant:6333
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
      - TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN}
      - TELEGRAM_CHAT_ID=${TELEGRAM_CHAT_ID}
    depends_on:
      - qdrant
      - redis
    restart: unless-stopped

  celery:
    build: ./backend
    container_name: espiritismo-celery
    command: celery -A app.tasks.celery_tasks worker --loglevel=info
    volumes:
      - ./PDF:/app/uploads:ro
      - ./backend/schedules.json:/app/schedules.json
    environment:
      - QDRANT_URL=http://qdrant:6333
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
      - TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN}
      - TELEGRAM_CHAT_ID=${TELEGRAM_CHAT_ID}
    depends_on:
      - qdrant
      - redis
    restart: unless-stopped

  celery-beat:
    build: ./backend
    command: celery -A app.tasks.celery_tasks beat --loglevel=info
    volumes:
      - ./backend/schedules.json:/app/schedules.json
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
      - TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN}
      - TELEGRAM_CHAT_ID=${TELEGRAM_CHAT_ID}
    depends_on:
      - redis
    restart: unless-stopped

volumes:
  qdrant_data:
  redis_data:

• Passo 3: Criar requirements.txt

fastapi==0.104.1
uvicorn[standard]==0.24.0
pydantic==2.5.0
python-multipart==0.0.6
gpt4all==2.2.7
langchain==0.1.0
langchain-community==0.0.10
langchain-experimental==0.0.52
sentence-transformers==2.2.2
pytelegrambotapi==4.15.1
celery==5.3.4
redis==5.0.1
python-dotenv==1.0.0
python-dateutil==2.8.2
aiofiles==23.2.1

• Passo 4: Criar .env.example

GPT4ALL_MODEL=llama-3.2-3b-instruct-q4_0.gguf
QDRANT_URL=http://localhost:6333
QDRANT_API_KEY=
TELEGRAM_BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrsTUVwxyz
TELEGRAM_CHAT_ID=123456789
CELERY_BROKER_URL=redis://localhost:6379/0
CELERY_RESULT_BACKEND=redis://localhost:6379/0
UPLOAD_DIR=/app/uploads
INDEX_DIR=/app/index

• Passo 5: Criar README.md

# Sistema RAG Espiritismo

Sistema de consulta assistida por IA para livros de espiritismo em PDF.

## Stack

- GPT4All + Llama 3.2 3B (CPU-only)
- LangChain para RAG
- Qdrant para vectores
- FastAPI + HTML/CSS/JS

## Quick Start

1. Clone o repositório
2. Configure o .env
3. Baixe o modelo Llama 3.2 3B quantized
4. `docker-compose up --build`

## Comandos

- `docker-compose up`: Start all services
- `docker-compose run --rm web python backend/scripts/index_books.py`: Indexar livros

---

Tarefa 2: Indexação dos Livros PDF

Arquivos:

• Criar: backend/app/utils/embeddings.py
• Criar: backend/app/services/pdf_processor.py
• Criar: backend/scripts/index_books.py

Interfaces:

• Consumes: N/A (setup inicial)

• Produces: Livros indexados no Qdrant

• Passo 1: Criar embeddings.py

from sentence_transformers import SentenceTransformer
import numpy as np

class EmbeddingService:
    def __init__(self, model_name="all-MiniLM-L6-v2"):
        self.model = SentenceTransformer(model_name)
        self.dimension = 384

    def embed(self, texts: list[str]) -> np.ndarray:
        """Generate embeddings for a list of texts."""
        return self.model.encode(texts, convert_to_numpy=True)

    def embed_query(self, text: str) -> np.ndarray:
        """Generate embedding for a single query."""
        return self.model.encode([text], convert_to_numpy=True)[0]

    def get_dimension(self) -> int:
        """Return embedding dimension."""
        return self.dimension

• Passo 2: Criar pdf_processor.py

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
import os
from typing import List, Dict
from .embeddings import EmbeddingService

class PDFProcessor:
    def __init__(self, embedding_service: EmbeddingService):
        self.embedding_service = embedding_service
        self.text_splitter = RecursiveCharacterTextSplitter(
            chunk_size=500,
            chunk_overlap=50,
            length_function=len,
        )

    def load_and_split(self, file_path: str, metadata: Dict) -> List[Dict]:
        """Load and split a PDF file."""
        loader = PyPDFLoader(file_path)
        documents = loader.load()

        chunks = self.text_splitter.split_documents(documents)

        # Convert to dictionaries with metadata
        chunk_data = []
        for i, doc in enumerate(chunks):
            chunk_data.append({
                "content": doc.page_content,
                "metadata": {
                    **metadata,
                    "chunk_id": i,
                    "source": file_path,
                }
            })

        return chunk_data

    def embed_chunks(self, chunks: List[Dict]) -> List[Dict]:
        """Add embeddings to chunks."""
        texts = [chunk["content"] for chunk in chunks]
        embeddings = self.embedding_service.embed(texts)

        for chunk, embedding in zip(chunks, embeddings):
            chunk["embedding"] = embedding.tolist()

        return chunks

• Passo 3: Criar index_books.py

from app.services.pdf_processor import PDFProcessor
from app.utils.embeddings import EmbeddingService
from app.utils.schedule_manager import ScheduleManager
import os
import json

def main():
    # Setup services
    embedding_service = EmbeddingService()
    pdf_processor = PDFProcessor(embedding_service)

    # Load books from PDF directory
    pdf_dir = "/app/uploads"
    books = os.listdir(pdf_dir)
    books = [f for f in books if f.endswith('.pdf')]

    print(f"Found {len(books)} books")

    # Schedule manager
    schedule_manager = ScheduleManager("/app/schedules.json")

    # Process each book
    for book_name in books:
        book_path = os.path.join(pdf_dir, book_name)
        book_metadata = {
            "title": book_name.replace(".pdf", "").replace("-", " ").replace("_", " "),
            "author": "Desconhecido",
            "type": "book",
        }

        print(f"Processing: {book_name}")

        # Load and split PDF
        chunks = pdf_processor.load_and_split(book_path, book_metadata)

        # Embed chunks
        chunks_with_embeddings = pdf_processor.embed_chunks(chunks)

        # Save chunks to JSON for reference
        chunks_file = f"/app/index/{book_name}.json"
        os.makedirs("/app/index", exist_ok=True)

        with open(chunks_file, 'w') as f:
            json.dump(chunks_with_embeddings, f, indent=2)

        print(f"Saved {len(chunks)} chunks to {chunks_file}")

    print("Indexing complete!")

if __name__ == "__main__":
    main()

---

Tarefa 3: Configuração do Qdrant e RAG Service

Arquivos:

• Criar: backend/app/services/rag_service.py
• Criar: backend/app/utils/__init__.py

Interfaces:

• Consumes: PDFProcessor, EmbeddingService

• Produces: Qdrant collection configurada, service RAG funcional

• Passo 1: Criar __init__.py

from .embeddings import EmbeddingService
from .schedule_manager import ScheduleManager

__all__ = ["EmbeddingService", "ScheduleManager"]

• Passo 2: Criar rag_service.py

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct, Filter, FieldCondition, MatchValue
import numpy as np
from typing import List, Dict, Optional
import os
import json

class QdrantService:
    def __init__(self, url: str, collection_name: str = "espiritismo_books"):
        self.client = QdrantClient(url=url, prefer_grpc=False)
        self.collection_name = collection_name
        self.embedding_dim = 384

        self._ensure_collection()

    def _ensure_collection(self):
        """Create collection if it doesn't exist."""
        collections = self.client.get_collections().collections

        collection_exists = any(
            c.name == self.collection_name for c in collections
        )

        if not collection_exists:
            self.client.create_collection(
                collection_name=self.collection_name,
                vectors_config=VectorParams(
                    size=self.embedding_dim,
                    distance=Distance.COSINE,
                ),
            )
            print(f"Created collection: {self.collection_name}")
        else:
            print(f"Collection {self.collection_name} already exists")

    def add_chunks(self, chunks: List[Dict]):
        """Add chunks to the collection."""
        points = []

        for i, chunk in enumerate(chunks):
            point = PointStruct(
                id=i,
                vector=chunk["embedding"],
                payload={
                    "content": chunk["content"],
                    "metadata": chunk["metadata"],
                }
            )
            points.append(point)

        if points:
            self.client.upsert(
                collection_name=self.collection_name,
                points=points,
            )
            print(f"Added {len(points)} chunks to Qdrant")

    def search(self, query_embedding: List[float], top_k: int = 3) -> List[Dict]:
        """Search for similar chunks."""
        results = self.client.search(
            collection_name=self.collection_name,
            query_vector=query_embedding,
            limit=top_k,
        )

        return [
            {
                "content": hit.payload["content"],
                "metadata": hit.payload["metadata"],
                "score": hit.score,
            }
            for hit in results
        ]

    def get_books(self) -> List[str]:
        """Get list of books in the collection."""
        points = self.client.scroll(
            collection_name=self.collection_name,
            limit=10000,
        )[0]

        books = set()
        for point in points:
            metadata = point.payload["metadata"]
            books.add(metadata.get("title", "Unknown"))

        return sorted(list(books))

    def get_collection_info(self) -> Dict:
        """Get information about the collection."""
        info = self.client.get_collection(self.collection_name)
        return {
            "status": info.status,
            "vectors_count": info.vectors_count,
            "points_count": info.points_count,
        }


class RAGService:
    def __init__(self, qdrant_service: QdrantService, model_name: str = "llama-3.2-3b-instruct-q4_0.gguf"):
        self.qdrant = qdrant_service
        self.model_name = model_name

        # Setup GPT4All
        from gpt4all import GPT4All

        self.llm = GPT4All(model_name=model_name, device="cpu")

    def generate_response(self, query: str, context: List[str], top_k: int = 3) -> str:
        """Generate a response based on the query and context."""
        context_text = "\n\n".join(context)

        system_prompt = f"""Você é um assistente especializado em espiritismo. Use as informações abaixo para responder à pergunta do usuário.

Informações:
{context_text}

Instruções:
- Responda de forma clara e concisa
- Use o contexto fornecido
- Quando citar passagens, mencione o livro e capítulo (se disponível)
- Se não souber a resposta, diga que não encontrou na base de dados"""

        response = self.llm.generate(
            system_prompt,
            query,
            max_tokens=500,
        )

        return response

    def query(self, query: str, top_k: int = 3) -> Dict:
        """Query the RAG system."""
        # Get embeddings
        from app.utils.embeddings import EmbeddingService
        embedding_service = EmbeddingService()

        query_embedding = embedding_service.embed_query(query)

        # Search
        chunks = self.qdrant.search(query_embedding, top_k)

        # Generate response
        context = [chunk["content"] for chunk in chunks]

        response = self.generate_response(query, context, top_k)

        return {
            "query": query,
            "response": response,
            "context": context,
            "sources": [chunk["metadata"] for chunk in chunks],
        }

---

Tarefa 4: API Web com Chat

Arquivos:

• Criar: backend/app/main.py
• Criar: backend/app/models.py

Interfaces:

• Consumes: RAGService

• Produces: API web com endpoint de chat e listagem de livros

• Passo 1: Criar models.py

from pydantic import BaseModel
from typing import Optional, List, Dict

class ChatRequest(BaseModel):
    message: str
    top_k: int = 3

class ChatResponse(BaseModel):
    query: str
    response: str
    sources: List[Dict]

class BookInfo(BaseModel):
    title: str
    source: str

class ChatRequestResponse(BaseModel):
    message: str
    response: str

class UploadResponse(BaseModel):
    message: str
    filename: str

class ScheduleInfo(BaseModel):
    id: int
    time: str
    book: Optional[str]
    enabled: bool
    next_run: Optional[str]

class ScheduleRequest(BaseModel):
    time: str
    book: Optional[str] = None
    enabled: bool = True

• Passo 2: Criar main.py

from fastapi import FastAPI, HTTPException
from fastapi.responses import HTMLResponse
from app.services.rag_service import QdrantService, RAGService
from app.models import ChatRequest, ChatResponse, BookInfo, ChatRequestResponse, UploadResponse, ScheduleInfo, ScheduleRequest
import os
from app.utils.schedule_manager import ScheduleManager

app = FastAPI(title="Sistema RAG Espiritismo")

# Initialize services
qdrant_service = QdrantService(os.getenv("QDRANT_URL", "http://qdrant:6333"))
rag_service = RAGService(qdrant_service, os.getenv("GPT4ALL_MODEL", "llama-3.2-3b-instruct-q4_0.gguf"))
schedule_manager = ScheduleManager("/app/schedules.json")

@app.get("/", response_class=HTMLResponse)
async def index():
    """Serve the frontend interface."""
    with open("app/templates/index.html", "r") as f:
        return f.read()

@app.post("/api/chat", response_model=ChatResponse)
async def chat(request: ChatRequest):
    """Chat endpoint - Query the RAG system."""
    try:
        result = rag_service.query(request.message, request.top_k)
        return ChatResponse(**result)
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

@app.get("/api/books", response_model=List[BookInfo])
async def get_books():
    """Get list of indexed books."""
    try:
        books = qdrant_service.get_books()
        return [BookInfo(title=book, source=f"Qdrant collection") for book in books]
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

@app.get("/api/schedules", response_model=List[ScheduleInfo])
async def get_schedules():
    """Get list of schedules."""
    try:
        schedules = schedule_manager.get_all()
        return schedules
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

@app.post("/api/schedules", response_model=ScheduleInfo)
async def create_schedule(request: ScheduleRequest):
    """Create a new schedule."""
    try:
        schedule = schedule_manager.create(
            time=request.time,
            book=request.book,
            enabled=request.enabled
        )
        return schedule
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

@app.delete("/api/schedules/{schedule_id}")
async def delete_schedule(schedule_id: int):
    """Delete a schedule."""
    try:
        schedule_manager.delete(schedule_id)
        return {"message": "Schedule deleted"}
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

@app.post("/api/upload")
async def upload_pdf(file):
    """Upload a new PDF file."""
    try:
        from app.services.pdf_processor import PDFProcessor
        from app.utils.embeddings import EmbeddingService

        file_path = f"/app/uploads/{file.filename}"

        with open(file_path, "wb") as buffer:
            buffer.write(await file.read())

        # Process the PDF
        embedding_service = EmbeddingService()
        pdf_processor = PDFProcessor(embedding_service)

        book_metadata = {
            "title": file.filename.replace(".pdf", "").replace("-", " ").replace("_", " "),
            "author": "Uploaded",
            "type": "book",
        }

        chunks = pdf_processor.load_and_split(file_path, book_metadata)
        chunks_with_embeddings = pdf_processor.embed_chunks(chunks)

        # Add to Qdrant
        qdrant_service.add_chunks(chunks_with_embeddings)

        # Save chunks to JSON
        chunks_file = f"/app/index/{file.filename}.json"
        os.makedirs("/app/index", exist_ok=True)

        with open(chunks_file, 'w') as f:
            json.dump(chunks_with_embeddings, f, indent=2)

        return UploadResponse(
            message="PDF uploaded and indexed successfully",
            filename=file.filename
        )
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

---

Tarefa 5: Frontend Interface

Arquivos:

• Criar: backend/app/templates/index.html
• Criar: frontend/static/css/style.css
• Criar: frontend/static/js/script.js

Interfaces:

• Consumes: API web endpoints

• Produces: Interface web funcional para chat, upload e configuração

• Passo 1: Criar style.css

* {
    margin: 0;
    padding: 0;
    box-sizing: border-box;
}

body {
    font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
    background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
    min-height: 100vh;
    padding: 20px;
}

.container {
    max-width: 1200px;
    margin: 0 auto;
}

header {
    background: white;
    padding: 20px;
    border-radius: 10px;
    margin-bottom: 20px;
    box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);
}

header h1 {
    color: #667eea;
    font-size: 2rem;
}

header p {
    color: #666;
    margin-top: 5px;
}

.grid {
    display: grid;
    grid-template-columns: repeat(auto-fit, minmax(350px, 1fr));
    gap: 20px;
}

.card {
    background: white;
    padding: 20px;
    border-radius: 10px;
    box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);
}

.card h2 {
    color: #667eea;
    margin-bottom: 15px;
    font-size: 1.25rem;
}

.chat-container {
    height: 500px;
    display: flex;
    flex-direction: column;
}

.chat-messages {
    flex: 1;
    overflow-y: auto;
    padding: 15px;
    background: #f5f5f5;
    border-radius: 5px;
    margin-bottom: 15px;
}

.message {
    padding: 10px;
    margin-bottom: 10px;
    border-radius: 5px;
}

.message.user {
    background: #667eea;
    color: white;
}

.message.bot {
    background: #e3f2fd;
}

.message-content {
    line-height: 1.6;
}

.message-source {
    font-size: 0.8rem;
    color: #666;
    margin-top: 5px;
    font-style: italic;
}

input[type="text"],
input[type="datetime-local"],
input[type="number"] {
    width: 100%;
    padding: 10px;
    border: 1px solid #ddd;
    border-radius: 5px;
    margin-bottom: 10px;
    font-size: 1rem;
}

button {
    width: 100%;
    padding: 12px;
    background: #667eea;
    color: white;
    border: none;
    border-radius: 5px;
    font-size: 1rem;
    cursor: pointer;
    transition: background 0.3s;
}

button:hover {
    background: #5568d3;
}

button:disabled {
    background: #ccc;
    cursor: not-allowed;
}

table {
    width: 100%;
    border-collapse: collapse;
}

th, td {
    padding: 10px;
    text-align: left;
    border-bottom: 1px solid #ddd;
}

th {
    background: #f5f5f5;
}

.status-badge {
    padding: 5px 10px;
    border-radius: 5px;
    font-size: 0.8rem;
    display: inline-block;
}

.status-badge.active {
    background: #4caf50;
    color: white;
}

.status-badge.inactive {
    background: #f44336;
    color: white;
}

.file-input {
    border: 2px dashed #ddd;
    padding: 20px;
    text-align: center;
    border-radius: 5px;
    cursor: pointer;
}

.file-input:hover {
    border-color: #667eea;
    background: #f9f9f9;
}

• Passo 2: Criar script.js

document.addEventListener('DOMContentLoaded', function() {
    const chatMessages = document.getElementById('chatMessages');
    const messageInput = document.getElementById('messageInput');
    const sendMessageBtn = document.getElementById('sendMessageBtn');
    const bookSelect = document.getElementById('bookSelect');
    const schedulesTable = document.getElementById('schedulesTable');

    // Load books
    loadBooks();

    // Load schedules
    loadSchedules();

    // Chat functionality
    sendMessageBtn.addEventListener('click', sendMessage);
    messageInput.addEventListener('keypress', function(e) {
        if (e.key === 'Enter') sendMessage();
    });

    async function sendMessage() {
        const message = messageInput.value.trim();
        if (!message) return;

        // Add user message
        addMessage(message, 'user');

        messageInput.value = '';

        try {
            const response = await fetch('/api/chat', {
                method: 'POST',
                headers: {
                    'Content-Type': 'application/json',
                },
                body: JSON.stringify({
                    message: message,
                    top_k: 3
                })
            });

            if (!response.ok) throw new Error('Failed to send message');

            const data = await response.json();

            // Add bot response
            addMessage(data.response, 'bot');

            // Add sources
            data.sources.forEach(source => {
                addSource(`📖 ${source.title}`);
            });

        } catch (error) {
            addMessage('Erro ao enviar mensagem. Por favor, tente novamente.', 'bot');
            console.error(error);
        }
    }

    function addMessage(text, type) {
        const messageDiv = document.createElement('div');
        messageDiv.className = `message ${type}`;
        messageDiv.innerHTML = `<div class="message-content">${text}</div>`;
        chatMessages.appendChild(messageDiv);
        chatMessages.scrollTop = chatMessages.scrollHeight;
    }

    function addSource(text) {
        const messageDiv = document.createElement('div');
        messageDiv.className = 'message-source';
        messageDiv.textContent = text;
        chatMessages.appendChild(messageDiv);
    }

    async function loadBooks() {
        try {
            const response = await fetch('/api/books');
            const books = await response.json();

            bookSelect.innerHTML = '<option value="">Todos os livros</option>';
            books.forEach(book => {
                const option = document.createElement('option');
                option.value = book.title;
                option.textContent = book.title;
                bookSelect.appendChild(option);
            });
        } catch (error) {
            console.error('Error loading books:', error);
        }
    }

    async function loadSchedules() {
        try {
            const response = await fetch('/api/schedules');
            const schedules = await response.json();

            schedulesTable.innerHTML = '';
            schedules.forEach(schedule => {
                const row = document.createElement('tr');
                row.innerHTML = `
                    <td>${schedule.id}</td>
                    <td>${schedule.time}</td>
                    <td>${schedule.book || 'Todos os livros'}</td>
                    <td><span class="status-badge ${schedule.enabled ? 'active' : 'inactive'}">${schedule.enabled ? 'Ativo' : 'Inativo'}</span></td>
                    <td>
                        <button onclick="deleteSchedule(${schedule.id})" style="width: auto; padding: 5px 10px; font-size: 0.8rem;">
                            Remover
                        </button>
                    </td>
                `;
                schedulesTable.appendChild(row);
            });
        } catch (error) {
            console.error('Error loading schedules:', error);
        }
    }

    async function deleteSchedule(id) {
        if (!confirm('Tem certeza que deseja remover este agendamento?')) return;

        try {
            const response = await fetch(`/api/schedules/${id}`, {
                method: 'DELETE'
            });

            if (!response.ok) throw new Error('Failed to delete schedule');

            loadSchedules();
        } catch (error) {
            alert('Erro ao remover agendamento');
            console.error(error);
        }
    }

    // Upload file
    const fileInput = document.getElementById('fileInput');
    fileInput.addEventListener('change', async function(e) {
        const file = e.target.files[0];
        if (!file) return;

        const formData = new FormData();
        formData.append('file', file);

        try {
            const response = await fetch('/api/upload', {
                method: 'POST',
                body: formData
            });

            if (!response.ok) throw new Error('Failed to upload file');

            const data = await response.json();
            alert(`Sucesso: ${data.message}`);

            // Reload books
            loadBooks();
        } catch (error) {
            alert('Erro ao fazer upload do arquivo');
            console.error(error);
        }
    });
});

• Passo 3: Criar index.html

<!DOCTYPE html>
<html lang="pt-BR">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Sistema RAG Espiritismo</title>
    <link rel="stylesheet" href="/static/css/style.css">
</head>
<body>
    <div class="container">
        <header>
            <h1>📚 Sistema RAG Espiritismo</h1>
            <p>Consulte livros de espiritismo em linguagem natural</p>
        </header>

        <div class="grid">
            <div class="card">
                <h2>💬 Chat com os Livros</h2>
                <div class="chat-container">
                    <div class="chat-messages" id="chatMessages">
                        <div class="message bot">
                            <div class="message-content">
                                Olá! Sou um assistente especializado em espiritismo. 
                                Pergunte-me sobre os livros que você tem na base de dados.
                            </div>
                        </div>
                    </div>
                    <input type="text" id="messageInput" placeholder="Digite sua pergunta..." autocomplete="off">
                    <button id="sendMessageBtn">Enviar Mensagem</button>
                </div>
            </div>

            <div class="card">
                <h2>📂 Adicionar Novo Livro</h2>
                <input type="file" id="fileInput" accept=".pdf" class="file-input">
                <p style="margin-top: 10px; color: #666; font-size: 0.9rem;">
                    Faça upload de um PDF para adicionar novos livros à base de dados.
                </p>
            </div>

            <div class="card">
                <h2>⏰ Agendar Mensagens</h2>
                <label>Filtrar por livro:</label>
                <select id="bookSelect"></select>

                <label>Horário (HH:MM):</label>
                <input type="time" id="scheduleTime">

                <button id="addScheduleBtn">Adicionar Agendamento</button>

                <table id="schedulesTable">
                    <thead>
                        <tr>
                            <th>ID</th>
                            <th>Horário</th>
                            <th>Livro</th>
                            <th>Status</th>
                            <th>Ações</th>
                        </tr>
                    </thead>
                    <tbody>
                        <tr>
                            <td colspan="5" style="text-align: center; color: #999;">
                                Carregando agendamentos...
                            </td>
                        </tr>
                    </tbody>
                </table>
            </div>
        </div>
    </div>

    <script src="/static/js/script.js"></script>
</body>
</html>

---

Tarefa 6: Serviço de Telegram Bot

Arquivos:

• Criar: backend/app/services/bot_service.py
• Criar: backend/app/tasks/celery_tasks.py

Interfaces:

• Consumes: RAGService, ScheduleManager

• Produces: Bot Telegram funcional com comandos de agendamento

• Passo 1: Criar bot_service.py

import logging
import random
from typing import Dict
from pyTelegramBotAPI import telebot
import os

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class TelegramBot:
    def __init__(self, token: str, chat_id: str):
        self.bot = telebot.TeleBot(token)
        self.chat_id = chat_id

        self._register_handlers()

    def _register_handlers(self):
        """Register bot command handlers."""

        @self.bot.message_handler(commands=['start'])
        def send_welcome(message):
            self.bot.reply_to(
                message,
                "📚 *Bem-vindo ao Sistema RAG Espiritismo!*\n\n"
                "Comandos disponíveis:\n"
                "/start - Iniciar bot\n"
                "/help - Ajuda\n"
                "/list - Listar livros disponíveis\n"
                "/schedule - Configurar horário de mensagem\n"
                "/schedule list - Ver agendamentos\n"
                "/schedule remove <id> - Remover agendamento\n"
                "/content - Receber conteúdo agora\n\n"
                "Use o sistema web para gerenciar melhor os agendamentos!"
            )

        @self.bot.message_handler(commands=['help'])
        def send_help(message):
            self.bot.reply_to(
                message,
                "📖 *Comandos do Bot Telegram*\n\n"
                "/start - Iniciar\n"
                "/help - Esta mensagem\n"
                "/list - Ver livros disponíveis\n"
                "/schedule <horário> [livro] - Agendar mensagem\n"
                "  Ex: /schedule 08:00\n"
                "  Ex: /schedule 08:00 \"Livro dos Espíritos\"\n"
                "/schedule list - Ver todos os agendamentos\n"
                "/schedule remove <id> - Remover agendamento\n"
                "/content - Receber conteúdo agora"
            )

        @self.bot.message_handler(commands=['list'])
        def list_books(message):
            books = self.rag_service.qdrant_service.get_books()

            if books:
                books_text = "\n".join([f"{i+1}. {book}" for i, book in enumerate(books)])
                self.bot.reply_to(
                    message,
                    f"📚 *Livros disponíveis:*\n\n{books_text}"
                )
            else:
                self.bot.reply_to(message, "Não há livros na base de dados.")

        @self.bot.message_handler(commands=['content'])
        def send_content(message):
            try:
                book = self.rag_service.query("Conteúdo aleatório sobre espiritismo", top_k=1)
                self.bot.reply_to(message, book['response'])
            except Exception as e:
                logger.error(f"Error sending content: {e}")
                self.bot.reply_to(message, "Erro ao gerar conteúdo.")

        @self.bot.message_handler(commands=['schedule'])
        def schedule_message(message):
            args = message.text.split()[1:]

            if not args:
                self.bot.reply_to(message, "Uso: /schedule <horário> [livro]")
                return

            time = args[0]

            if len(args) > 1:
                book = " ".join(args[1:])
            else:
                book = None

            try:
                self.schedule_manager.create(time=time, book=book)
                self.bot.reply_to(
                    message,
                    f"✅ Agendamento criado para {time}\n"
                    f"📝 Livro: {book or 'Todos os livros'}"
                )
            except Exception as e:
                logger.error(f"Error creating schedule: {e}")
                self.bot.reply_to(message, f"Erro ao criar agendamento: {e}")

        @self.bot.message_handler(commands=['schedule', 'list'])
        def list_schedules(message):
            schedules = self.schedule_manager.get_all()

            if not schedules:
                self.bot.reply_to(message, "Não há agendamentos.")
                return

            schedules_text = "\n".join([
                f"{s.id}. {s.time} - {s.book or 'Todos'} - {'✅' if s.enabled else '❌'}"
                for s in schedules
            ])

            self.bot.reply_to(message, f"📅 *Agendamentos:*\n\n{schedules_text}")

        @self.bot.message_handler(commands=['schedule'])
        def remove_schedule(message):
            try:
                args = message.text.split()
                if len(args) < 3:
                    self.bot.reply_to(message, "Uso: /schedule remove <id>")
                    return

                schedule_id = int(args[2])
                self.schedule_manager.delete(schedule_id)

                self.bot.reply_to(message, f"✅ Agendamento {schedule_id} removido.")
            except ValueError:
                self.bot.reply_to(message, "ID deve ser um número.")
            except Exception as e:
                logger.error(f"Error removing schedule: {e}")
                self.bot.reply_to(message, f"Erro ao remover agendamento: {e}")

    def start(self):
        """Start the bot."""
        logger.info("Starting Telegram bot...")
        self.bot.infinity_polling(timeout=10, long_polling_timeout=1)

• Passo 2: Criar celery_tasks.py

from celery import Celery
from celery.schedules import crontab
from app.services.bot_service import TelegramBot
from app.services.rag_service import RAGService
from app.utils.schedule_manager import ScheduleManager
import os
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

celery = Celery(
    "espiritismo_tasks",
    broker=os.getenv("CELERY_BROKER_URL", "redis://redis:6379/0"),
    backend=os.getenv("CELERY_RESULT_BACKEND", "redis://redis:6379/0"),
)

# Initialize services
qdrant_service = qdrant_client.QdrantClient(
    url=os.getenv("QDRANT_URL", "http://qdrant:6333")
)
rag_service = RAGService(qdrant_service, os.getenv("GPT4ALL_MODEL", "llama-3.2-3b-instruct-q4_0.gguf"))
schedule_manager = ScheduleManager("/app/schedules.json")
bot = TelegramBot(os.getenv("TELEGRAM_BOT_TOKEN", ""), os.getenv("TELEGRAM_CHAT_ID", ""))


@celery.task
def send_scheduled_message():
    """Send a random message from the books."""
    logger.info("Sending scheduled message...")

    try:
        # Get all books
        books = qdrant_service.get_books()

        if not books:
            logger.warning("No books found in database")
            return

        # Choose a random book
        book = random.choice(books)

        # Generate random query
        queries = [
            "O que diz o livro sobre amor?",
            "Qual é a mensagem principal deste livro?",
            "Dê uma reflexão sobre este livro",
            "Conte uma história importante deste livro"
        ]

        query = random.choice(queries)

        # Query the RAG system
        result = rag_service.query(query, top_k=1)

        # Send via Telegram
        message = f"📖 *{book}*\n\n{result['response']}"

        bot.bot.send_message(
            bot.chat_id,
            message,
            parse_mode="Markdown"
        )

        logger.info(f"Message sent successfully: {book}")

    except Exception as e:
        logger.error(f"Error sending scheduled message: {e}")
        raise


@celery.periodic_task(
    run_every=crontab(minute="*"),
)
def check_schedules():
    """Check and execute scheduled messages."""
    logger.info("Checking schedules...")

    schedules = schedule_manager.get_all()

    for schedule in schedules:
        if not schedule.enabled:
            continue

        # Check if this is the scheduled time
        current_time = datetime.now().strftime("%H:%M")
        if current_time == schedule.time:
            send_scheduled_message.delay()

    logger.info("Schedule check completed")

---

Tarefa 7: Utilitários Adicionais

Arquivos:

• Criar: backend/app/utils/schedule_manager.py
• Criar: backend/app/__init__.py

Interfaces:

• Consumes: N/A

• Produces: Gerenciamento de agendamentos funcional

• Passo 1: Criar __init__.py

from .models import ChatRequest, ChatResponse, BookInfo, ChatRequestResponse, UploadResponse, ScheduleInfo, ScheduleRequest

__all__ = [
    "ChatRequest",
    "ChatResponse",
    "BookInfo",
    "ChatRequestResponse",
    "UploadResponse",
    "ScheduleInfo",
    "ScheduleRequest",
]

• Passo 2: Criar schedule_manager.py

import json
import os
from datetime import datetime
from typing import List, Dict
from dataclasses import dataclass, asdict, field
import pytz

@dataclass
class ScheduleInfo:
    id: int
    time: str
    book: Optional[str]
    enabled: bool
    created_at: str
    next_run: Optional[str] = None

class ScheduleManager:
    def __init__(self, file_path: str = "schedules.json"):
        self.file_path = file_path
        self.schedules: List[Dict] = []
        self.next_id: int = 1

        if os.path.exists(file_path):
            self._load()

    def _load(self):
        """Load schedules from file."""
        with open(self.file_path, 'r') as f:
            data = json.load(f)
            self.schedules = data.get("schedules", [])
            self.next_id = data.get("next_id", 1)

    def _save(self):
        """Save schedules to file."""
        data = {
            "schedules": self.schedules,
            "next_id": self.next_id
        }
        with open(self.file_path, 'w') as f:
            json.dump(data, f, indent=2)

    def get_all(self) -> List[Dict]:
        """Get all schedules."""
        return self.schedules

    def get(self, schedule_id: int) -> Dict:
        """Get a specific schedule."""
        for schedule in self.schedules:
            if schedule["id"] == schedule_id:
                return schedule
        return None

    def create(self, time: str, book: str = None, enabled: bool = True) -> Dict:
        """Create a new schedule."""
        schedule = {
            "id": self.next_id,
            "time": time,
            "book": book,
            "enabled": enabled,
            "created_at": datetime.now().isoformat(),
            "next_run": None
        }

        self.schedules.append(schedule)
        self.next_id += 1
        self._save()

        return schedule

    def delete(self, schedule_id: int):
        """Delete a schedule."""
        self.schedules = [s for s in self.schedules if s["id"] != schedule_id]
        self._save()

    def update(self, schedule_id: int, enabled: bool = None, book: str = None):
        """Update a schedule."""
        for schedule in self.schedules:
            if schedule["id"] == schedule_id:
                if enabled is not None:
                    schedule["enabled"] = enabled
                if book is not None:
                    schedule["book"] = book
                self._save()
                break

---

Tarefa 8: Dockerfile e Build

Arquivos:

• Criar: backend/Dockerfile

Interfaces:

• Consumes: N/A

• Produces: Imagem Docker funcional para todos os serviços

• Passo 1: Criar Dockerfile

FROM python:3.11-slim

WORKDIR /app

# Install system dependencies
RUN apt-get update && apt-get install -y \
    build-essential \
    && rm -rf /var/lib/apt/lists/*

# Copy requirements
COPY requirements.txt .

# Install Python dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Create directories
RUN mkdir -p /app/uploads /app/index /app/schedules.json

# Copy application
COPY app/ ./app/
COPY frontend/ ./frontend/

# Expose ports
EXPOSE 8000

# Run the application
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

---

Tarefa 9: Build e Testes

Arquivos:

• N/A

Interfaces:

• Consumes: Todo o código implementado

• Produces: Sistema funcional testado

• Passo 1: Build da aplicação

cd backend
docker-compose up --build -d

• Passo 2: Indexar livros

docker-compose run --rm web python scripts/index_books.py

• Passo 3: Verificar logs

docker-compose logs -f web
docker-compose logs -f celery
docker-compose logs -f celery-beat

• Passo 4: Testar API

curl http://localhost:8000/api/books
curl -X POST http://localhost:8000/api/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "O que é espiritismo?"}'

• Passo 5: Testar index.html

# Acessar: http://localhost:8000

• Passo 6: Iniciar bot Telegram

# Iniciar bot separadamente
docker-compose run --rm web python -m app.services.bot_service

---

Review Focus

1. Consulta web: Sistema retorna resposta com contexto e fontes
2. Upload PDF: PDF uploadado é indexado automaticamente
3. Agendamento Telegram: Mensagens enviadas no horário correto
4. Indexação RAG: Livros processados corretamente em chunks
5. Integração web-bot: Agendamentos funcionam em ambas as interfaces

---

Checklist de Aceitação

• Sistema iniciado com docker-compose up
• 6 livros indexados no Qdrant
• API web responde a consultas
• Interface web funcional para chat
• Upload de novos PDFs funciona
• Agendamentos podem ser criados no web
• Agendamentos aparecem no bot Telegram
• Mensagens são enviadas no horário correto
• Sistema rodando com 0 erros