# Sistema RAG Espiritismo - Implementação
> **Para workers agenticos:** SUB-SKILL OBRIGATÓRIA: Use superpowers:subagent-driven-development (recomendado) ou superpowers:executing-plans para implementar este plano tarefa por tarefa. Os passos usam sintaxe de checkbox (`- [ ]`) para rastreamento.
**Objetivo:** Desenvolver um sistema RAG (Retrieval-Augmented Generation) completo para consulta em linguagem natural aos livros de espiritismo em PDF, com interface web rica e integração com Telegram para agendamento de mensagens de conteúdo diário.
**Arquitetura:** Stack minimalista com GPT4All (Llama 3.2 3B) + LangChain + Qdrant para vectores, FastAPI para backend, HTML/CSS/JS para frontend, e pyTelegramBotAPI para bot com Celery para agendamento.
**Stack Tecnológica:** GPT4All, LangChain, Qdrant, FastAPI, pyTelegramBotAPI, Celery, Redis, Docker Compose
**Spec:** docs/superpowers/specs/2025-11-01-espiritismo-rag-system-design.md
## Restrições Globais
- CPU-only, sem GPU (máximo 8GB RAM)
- Armazenamento máximo: ~5GB (modelo 2GB + vetores 2GB + aplicação)
- Todos os serviços em Docker
- Interface web simples mas rica, responsiva e amigável
- Bot Telegram funcional com comandos de agendamento
- Suporte para upload e agregação futura de livros PDF
## Foco de Revisão
1. **Consulta web**: Usuário faz pergunta → sistema retorna resposta com citações e fontes
2. **Upload PDF**: Interface web deve permitir upload de novos PDFs → sistema indexa automaticamente
3. **Agendamento Telegram**: Bot deve permitir configurar horários de envio → mensagens enviadas corretamente
4. **Indexação RAG**: Sistema deve processar 6 livros existentes → gerar embeddings e estrutura vetorial
5. **Integração web-bot**: Configurações feitas no web devem ser acessíveis via bot
---
## Estrutura de Arquivos
```
espiritismo-rag/
├── backend/
│ ├── app/
│ │ ├── main.py # FastAPI application
│ │ ├── models.py # Pydantic models
│ │ ├── services/
│ │ │ ├── rag_service.py # RAG pipeline
│ │ │ ├── pdf_processor.py # PDF processing
│ │ │ └── bot_service.py # Telegram bot integration
│ │ ├── utils/
│ │ │ ├── embeddings.py # Embedding generation
│ │ │ └── schedule_manager.py # Schedule management
│ │ ├── templates/
│ │ │ └── index.html # Frontend interface
│ │ └── tasks/
│ │ └── celery_tasks.py # Celery tasks
│ ├── scripts/
│ │ └── index_books.py # Initial index script
│ ├── Dockerfile
│ ├── requirements.txt
│ └── .env.example
├── frontend/
│ └── static/
│ ├── css/
│ │ └── style.css
│ └── js/
│ └── script.js
├── docker-compose.yml
├── .env
├── celerybeat-schedule.json
├── schedules.json
└── README.md
```
---
## Tarefa 1: Setup Inicial e Estrutura do Projeto
**Arquivos:**
- Criar: `backend/app/main.py`
- Criar: `backend/app/models.py`
- Criar: `backend/services/rag_service.py`
- Criar: `backend/services/pdf_processor.py`
- Criar: `backend/services/bot_service.py`
- Criar: `backend/utils/embeddings.py`
- Criar: `backend/utils/schedule_manager.py`
- Criar: `backend/templates/index.html`
- Criar: `backend/tasks/celery_tasks.py`
- Criar: `backend/scripts/index_books.py`
- Criar: `backend/Dockerfile`
- Criar: `backend/requirements.txt`
- Criar: `docker-compose.yml`
- Criar: `frontend/static/css/style.css`
- Criar: `frontend/static/js/script.js`
- Criar: `backend/.env.example`
- Criar: `.env`
- Criar: `README.md`
**Interfaces:**
- Consumes: N/A (setup inicial)
- Produces: Estrutura completa do projeto, Docker Compose configurado
- [ ] **Passo 1: Criar estrutura de diretórios e arquivos básicos**
```bash
mkdir -p backend/app/{models,services,utils,templates,tasks}
mkdir -p backend/scripts
mkdir -p frontend/static/{css,js}
```
- [ ] **Passo 2: Criar docker-compose.yml**
```yaml
version: '3.8'
services:
qdrant:
image: qdrant/qdrant:latest
container_name: espiritismo-qdrant
ports:
- "6333:6333"
volumes:
- qdrant_data:/qdrant/storage
restart: unless-stopped
redis:
image: redis:7-alpine
container_name: espiritismo-redis
ports:
- "6379:6379"
volumes:
- redis_data:/data
restart: unless-stopped
web:
build: ./backend
container_name: espiritismo-web
ports:
- "8000:8000"
volumes:
- ./PDF:/app/uploads:ro
- ./backend/schedules.json:/app/schedules.json
environment:
- QDRANT_URL=http://qdrant:6333
- CELERY_BROKER_URL=redis://redis:6379/0
- CELERY_RESULT_BACKEND=redis://redis:6379/0
- TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN}
- TELEGRAM_CHAT_ID=${TELEGRAM_CHAT_ID}
depends_on:
- qdrant
- redis
restart: unless-stopped
celery:
build: ./backend
container_name: espiritismo-celery
command: celery -A app.tasks.celery_tasks worker --loglevel=info
volumes:
- ./PDF:/app/uploads:ro
- ./backend/schedules.json:/app/schedules.json
environment:
- QDRANT_URL=http://qdrant:6333
- CELERY_BROKER_URL=redis://redis:6379/0
- CELERY_RESULT_BACKEND=redis://redis:6379/0
- TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN}
- TELEGRAM_CHAT_ID=${TELEGRAM_CHAT_ID}
depends_on:
- qdrant
- redis
restart: unless-stopped
celery-beat:
build: ./backend
command: celery -A app.tasks.celery_tasks beat --loglevel=info
volumes:
- ./backend/schedules.json:/app/schedules.json
environment:
- CELERY_BROKER_URL=redis://redis:6379/0
- CELERY_RESULT_BACKEND=redis://redis:6379/0
- TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN}
- TELEGRAM_CHAT_ID=${TELEGRAM_CHAT_ID}
depends_on:
- redis
restart: unless-stopped
volumes:
qdrant_data:
redis_data:
```
- [ ] **Passo 3: Criar requirements.txt**
```
fastapi==0.104.1
uvicorn[standard]==0.24.0
pydantic==2.5.0
python-multipart==0.0.6
gpt4all==2.2.7
langchain==0.1.0
langchain-community==0.0.10
langchain-experimental==0.0.52
sentence-transformers==2.2.2
pytelegrambotapi==4.15.1
celery==5.3.4
redis==5.0.1
python-dotenv==1.0.0
python-dateutil==2.8.2
aiofiles==23.2.1
```
- [ ] **Passo 4: Criar .env.example**
```env
GPT4ALL_MODEL=llama-3.2-3b-instruct-q4_0.gguf
QDRANT_URL=http://localhost:6333
QDRANT_API_KEY=
TELEGRAM_BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrsTUVwxyz
TELEGRAM_CHAT_ID=123456789
CELERY_BROKER_URL=redis://localhost:6379/0
CELERY_RESULT_BACKEND=redis://localhost:6379/0
UPLOAD_DIR=/app/uploads
INDEX_DIR=/app/index
```
- [ ] **Passo 5: Criar README.md**
```markdown
# Sistema RAG Espiritismo
Sistema de consulta assistida por IA para livros de espiritismo em PDF.
## Stack
- GPT4All + Llama 3.2 3B (CPU-only)
- LangChain para RAG
- Qdrant para vectores
- FastAPI + HTML/CSS/JS
## Quick Start
1. Clone o repositório
2. Configure o .env
3. Baixe o modelo Llama 3.2 3B quantized
4. `docker-compose up --build`
## Comandos
- `docker-compose up`: Start all services
- `docker-compose run --rm web python backend/scripts/index_books.py`: Indexar livros
```
---
## Tarefa 2: Indexação dos Livros PDF
**Arquivos:**
- Criar: `backend/app/utils/embeddings.py`
- Criar: `backend/app/services/pdf_processor.py`
- Criar: `backend/scripts/index_books.py`
**Interfaces:**
- Consumes: N/A (setup inicial)
- Produces: Livros indexados no Qdrant
- [ ] **Passo 1: Criar embeddings.py**
```python
from sentence_transformers import SentenceTransformer
import numpy as np
class EmbeddingService:
def __init__(self, model_name="all-MiniLM-L6-v2"):
self.model = SentenceTransformer(model_name)
self.dimension = 384
def embed(self, texts: list[str]) -> np.ndarray:
"""Generate embeddings for a list of texts."""
return self.model.encode(texts, convert_to_numpy=True)
def embed_query(self, text: str) -> np.ndarray:
"""Generate embedding for a single query."""
return self.model.encode([text], convert_to_numpy=True)[0]
def get_dimension(self) -> int:
"""Return embedding dimension."""
return self.dimension
```
- [ ] **Passo 2: Criar pdf_processor.py**
```python
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
import os
from typing import List, Dict
from .embeddings import EmbeddingService
class PDFProcessor:
def __init__(self, embedding_service: EmbeddingService):
self.embedding_service = embedding_service
self.text_splitter = RecursiveCharacterTextSplitter(
chunk_size=500,
chunk_overlap=50,
length_function=len,
)
def load_and_split(self, file_path: str, metadata: Dict) -> List[Dict]:
"""Load and split a PDF file."""
loader = PyPDFLoader(file_path)
documents = loader.load()
chunks = self.text_splitter.split_documents(documents)
# Convert to dictionaries with metadata
chunk_data = []
for i, doc in enumerate(chunks):
chunk_data.append({
"content": doc.page_content,
"metadata": {
**metadata,
"chunk_id": i,
"source": file_path,
}
})
return chunk_data
def embed_chunks(self, chunks: List[Dict]) -> List[Dict]:
"""Add embeddings to chunks."""
texts = [chunk["content"] for chunk in chunks]
embeddings = self.embedding_service.embed(texts)
for chunk, embedding in zip(chunks, embeddings):
chunk["embedding"] = embedding.tolist()
return chunks
```
- [ ] **Passo 3: Criar index_books.py**
```python
from app.services.pdf_processor import PDFProcessor
from app.utils.embeddings import EmbeddingService
from app.utils.schedule_manager import ScheduleManager
import os
import json
def main():
# Setup services
embedding_service = EmbeddingService()
pdf_processor = PDFProcessor(embedding_service)
# Load books from PDF directory
pdf_dir = "/app/uploads"
books = os.listdir(pdf_dir)
books = [f for f in books if f.endswith('.pdf')]
print(f"Found {len(books)} books")
# Schedule manager
schedule_manager = ScheduleManager("/app/schedules.json")
# Process each book
for book_name in books:
book_path = os.path.join(pdf_dir, book_name)
book_metadata = {
"title": book_name.replace(".pdf", "").replace("-", " ").replace("_", " "),
"author": "Desconhecido",
"type": "book",
}
print(f"Processing: {book_name}")
# Load and split PDF
chunks = pdf_processor.load_and_split(book_path, book_metadata)
# Embed chunks
chunks_with_embeddings = pdf_processor.embed_chunks(chunks)
# Save chunks to JSON for reference
chunks_file = f"/app/index/{book_name}.json"
os.makedirs("/app/index", exist_ok=True)
with open(chunks_file, 'w') as f:
json.dump(chunks_with_embeddings, f, indent=2)
print(f"Saved {len(chunks)} chunks to {chunks_file}")
print("Indexing complete!")
if __name__ == "__main__":
main()
```
---
## Tarefa 3: Configuração do Qdrant e RAG Service
**Arquivos:**
- Criar: `backend/app/services/rag_service.py`
- Criar: `backend/app/utils/__init__.py`
**Interfaces:**
- Consumes: PDFProcessor, EmbeddingService
- Produces: Qdrant collection configurada, service RAG funcional
- [ ] **Passo 1: Criar __init__.py**
```python
from .embeddings import EmbeddingService
from .schedule_manager import ScheduleManager
__all__ = ["EmbeddingService", "ScheduleManager"]
```
- [ ] **Passo 2: Criar rag_service.py**
```python
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct, Filter, FieldCondition, MatchValue
import numpy as np
from typing import List, Dict, Optional
import os
import json
class QdrantService:
def __init__(self, url: str, collection_name: str = "espiritismo_books"):
self.client = QdrantClient(url=url, prefer_grpc=False)
self.collection_name = collection_name
self.embedding_dim = 384
self._ensure_collection()
def _ensure_collection(self):
"""Create collection if it doesn't exist."""
collections = self.client.get_collections().collections
collection_exists = any(
c.name == self.collection_name for c in collections
)
if not collection_exists:
self.client.create_collection(
collection_name=self.collection_name,
vectors_config=VectorParams(
size=self.embedding_dim,
distance=Distance.COSINE,
),
)
print(f"Created collection: {self.collection_name}")
else:
print(f"Collection {self.collection_name} already exists")
def add_chunks(self, chunks: List[Dict]):
"""Add chunks to the collection."""
points = []
for i, chunk in enumerate(chunks):
point = PointStruct(
id=i,
vector=chunk["embedding"],
payload={
"content": chunk["content"],
"metadata": chunk["metadata"],
}
)
points.append(point)
if points:
self.client.upsert(
collection_name=self.collection_name,
points=points,
)
print(f"Added {len(points)} chunks to Qdrant")
def search(self, query_embedding: List[float], top_k: int = 3) -> List[Dict]:
"""Search for similar chunks."""
results = self.client.search(
collection_name=self.collection_name,
query_vector=query_embedding,
limit=top_k,
)
return [
{
"content": hit.payload["content"],
"metadata": hit.payload["metadata"],
"score": hit.score,
}
for hit in results
]
def get_books(self) -> List[str]:
"""Get list of books in the collection."""
points = self.client.scroll(
collection_name=self.collection_name,
limit=10000,
)[0]
books = set()
for point in points:
metadata = point.payload["metadata"]
books.add(metadata.get("title", "Unknown"))
return sorted(list(books))
def get_collection_info(self) -> Dict:
"""Get information about the collection."""
info = self.client.get_collection(self.collection_name)
return {
"status": info.status,
"vectors_count": info.vectors_count,
"points_count": info.points_count,
}
class RAGService:
def __init__(self, qdrant_service: QdrantService, model_name: str = "llama-3.2-3b-instruct-q4_0.gguf"):
self.qdrant = qdrant_service
self.model_name = model_name
# Setup GPT4All
from gpt4all import GPT4All
self.llm = GPT4All(model_name=model_name, device="cpu")
def generate_response(self, query: str, context: List[str], top_k: int = 3) -> str:
"""Generate a response based on the query and context."""
context_text = "\n\n".join(context)
system_prompt = f"""Você é um assistente especializado em espiritismo. Use as informações abaixo para responder à pergunta do usuário.
Informações:
{context_text}
Instruções:
- Responda de forma clara e concisa
- Use o contexto fornecido
- Quando citar passagens, mencione o livro e capítulo (se disponível)
- Se não souber a resposta, diga que não encontrou na base de dados"""
response = self.llm.generate(
system_prompt,
query,
max_tokens=500,
)
return response
def query(self, query: str, top_k: int = 3) -> Dict:
"""Query the RAG system."""
# Get embeddings
from app.utils.embeddings import EmbeddingService
embedding_service = EmbeddingService()
query_embedding = embedding_service.embed_query(query)
# Search
chunks = self.qdrant.search(query_embedding, top_k)
# Generate response
context = [chunk["content"] for chunk in chunks]
response = self.generate_response(query, context, top_k)
return {
"query": query,
"response": response,
"context": context,
"sources": [chunk["metadata"] for chunk in chunks],
}
```
---
## Tarefa 4: API Web com Chat
**Arquivos:**
- Criar: `backend/app/main.py`
- Criar: `backend/app/models.py`
**Interfaces:**
- Consumes: RAGService
- Produces: API web com endpoint de chat e listagem de livros
- [ ] **Passo 1: Criar models.py**
```python
from pydantic import BaseModel
from typing import Optional, List, Dict
class ChatRequest(BaseModel):
message: str
top_k: int = 3
class ChatResponse(BaseModel):
query: str
response: str
sources: List[Dict]
class BookInfo(BaseModel):
title: str
source: str
class ChatRequestResponse(BaseModel):
message: str
response: str
class UploadResponse(BaseModel):
message: str
filename: str
class ScheduleInfo(BaseModel):
id: int
time: str
book: Optional[str]
enabled: bool
next_run: Optional[str]
class ScheduleRequest(BaseModel):
time: str
book: Optional[str] = None
enabled: bool = True
```
- [ ] **Passo 2: Criar main.py**
```python
from fastapi import FastAPI, HTTPException
from fastapi.responses import HTMLResponse
from app.services.rag_service import QdrantService, RAGService
from app.models import ChatRequest, ChatResponse, BookInfo, ChatRequestResponse, UploadResponse, ScheduleInfo, ScheduleRequest
import os
from app.utils.schedule_manager import ScheduleManager
app = FastAPI(title="Sistema RAG Espiritismo")
# Initialize services
qdrant_service = QdrantService(os.getenv("QDRANT_URL", "http://qdrant:6333"))
rag_service = RAGService(qdrant_service, os.getenv("GPT4ALL_MODEL", "llama-3.2-3b-instruct-q4_0.gguf"))
schedule_manager = ScheduleManager("/app/schedules.json")
@app.get("/", response_class=HTMLResponse)
async def index():
"""Serve the frontend interface."""
with open("app/templates/index.html", "r") as f:
return f.read()
@app.post("/api/chat", response_model=ChatResponse)
async def chat(request: ChatRequest):
"""Chat endpoint - Query the RAG system."""
try:
result = rag_service.query(request.message, request.top_k)
return ChatResponse(**result)
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
@app.get("/api/books", response_model=List[BookInfo])
async def get_books():
"""Get list of indexed books."""
try:
books = qdrant_service.get_books()
return [BookInfo(title=book, source=f"Qdrant collection") for book in books]
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
@app.get("/api/schedules", response_model=List[ScheduleInfo])
async def get_schedules():
"""Get list of schedules."""
try:
schedules = schedule_manager.get_all()
return schedules
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
@app.post("/api/schedules", response_model=ScheduleInfo)
async def create_schedule(request: ScheduleRequest):
"""Create a new schedule."""
try:
schedule = schedule_manager.create(
time=request.time,
book=request.book,
enabled=request.enabled
)
return schedule
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
@app.delete("/api/schedules/{schedule_id}")
async def delete_schedule(schedule_id: int):
"""Delete a schedule."""
try:
schedule_manager.delete(schedule_id)
return {"message": "Schedule deleted"}
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
@app.post("/api/upload")
async def upload_pdf(file):
"""Upload a new PDF file."""
try:
from app.services.pdf_processor import PDFProcessor
from app.utils.embeddings import EmbeddingService
file_path = f"/app/uploads/{file.filename}"
with open(file_path, "wb") as buffer:
buffer.write(await file.read())
# Process the PDF
embedding_service = EmbeddingService()
pdf_processor = PDFProcessor(embedding_service)
book_metadata = {
"title": file.filename.replace(".pdf", "").replace("-", " ").replace("_", " "),
"author": "Uploaded",
"type": "book",
}
chunks = pdf_processor.load_and_split(file_path, book_metadata)
chunks_with_embeddings = pdf_processor.embed_chunks(chunks)
# Add to Qdrant
qdrant_service.add_chunks(chunks_with_embeddings)
# Save chunks to JSON
chunks_file = f"/app/index/{file.filename}.json"
os.makedirs("/app/index", exist_ok=True)
with open(chunks_file, 'w') as f:
json.dump(chunks_with_embeddings, f, indent=2)
return UploadResponse(
message="PDF uploaded and indexed successfully",
filename=file.filename
)
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
```
---
## Tarefa 5: Frontend Interface
**Arquivos:**
- Criar: `backend/app/templates/index.html`
- Criar: `frontend/static/css/style.css`
- Criar: `frontend/static/js/script.js`
**Interfaces:**
- Consumes: API web endpoints
- Produces: Interface web funcional para chat, upload e configuração
- [ ] **Passo 1: Criar style.css**
```css
* {
margin: 0;
padding: 0;
box-sizing: border-box;
}
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
min-height: 100vh;
padding: 20px;
}
.container {
max-width: 1200px;
margin: 0 auto;
}
header {
background: white;
padding: 20px;
border-radius: 10px;
margin-bottom: 20px;
box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);
}
header h1 {
color: #667eea;
font-size: 2rem;
}
header p {
color: #666;
margin-top: 5px;
}
.grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(350px, 1fr));
gap: 20px;
}
.card {
background: white;
padding: 20px;
border-radius: 10px;
box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);
}
.card h2 {
color: #667eea;
margin-bottom: 15px;
font-size: 1.25rem;
}
.chat-container {
height: 500px;
display: flex;
flex-direction: column;
}
.chat-messages {
flex: 1;
overflow-y: auto;
padding: 15px;
background: #f5f5f5;
border-radius: 5px;
margin-bottom: 15px;
}
.message {
padding: 10px;
margin-bottom: 10px;
border-radius: 5px;
}
.message.user {
background: #667eea;
color: white;
}
.message.bot {
background: #e3f2fd;
}
.message-content {
line-height: 1.6;
}
.message-source {
font-size: 0.8rem;
color: #666;
margin-top: 5px;
font-style: italic;
}
input[type="text"],
input[type="datetime-local"],
input[type="number"] {
width: 100%;
padding: 10px;
border: 1px solid #ddd;
border-radius: 5px;
margin-bottom: 10px;
font-size: 1rem;
}
button {
width: 100%;
padding: 12px;
background: #667eea;
color: white;
border: none;
border-radius: 5px;
font-size: 1rem;
cursor: pointer;
transition: background 0.3s;
}
button:hover {
background: #5568d3;
}
button:disabled {
background: #ccc;
cursor: not-allowed;
}
table {
width: 100%;
border-collapse: collapse;
}
th, td {
padding: 10px;
text-align: left;
border-bottom: 1px solid #ddd;
}
th {
background: #f5f5f5;
}
.status-badge {
padding: 5px 10px;
border-radius: 5px;
font-size: 0.8rem;
display: inline-block;
}
.status-badge.active {
background: #4caf50;
color: white;
}
.status-badge.inactive {
background: #f44336;
color: white;
}
.file-input {
border: 2px dashed #ddd;
padding: 20px;
text-align: center;
border-radius: 5px;
cursor: pointer;
}
.file-input:hover {
border-color: #667eea;
background: #f9f9f9;
}
```
- [ ] **Passo 2: Criar script.js**
```javascript
document.addEventListener('DOMContentLoaded', function() {
const chatMessages = document.getElementById('chatMessages');
const messageInput = document.getElementById('messageInput');
const sendMessageBtn = document.getElementById('sendMessageBtn');
const bookSelect = document.getElementById('bookSelect');
const schedulesTable = document.getElementById('schedulesTable');
// Load books
loadBooks();
// Load schedules
loadSchedules();
// Chat functionality
sendMessageBtn.addEventListener('click', sendMessage);
messageInput.addEventListener('keypress', function(e) {
if (e.key === 'Enter') sendMessage();
});
async function sendMessage() {
const message = messageInput.value.trim();
if (!message) return;
// Add user message
addMessage(message, 'user');
messageInput.value = '';
try {
const response = await fetch('/api/chat', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
message: message,
top_k: 3
})
});
if (!response.ok) throw new Error('Failed to send message');
const data = await response.json();
// Add bot response
addMessage(data.response, 'bot');
// Add sources
data.sources.forEach(source => {
addSource(`📖 ${source.title}`);
});
} catch (error) {
addMessage('Erro ao enviar mensagem. Por favor, tente novamente.', 'bot');
console.error(error);
}
}
function addMessage(text, type) {
const messageDiv = document.createElement('div');
messageDiv.className = `message ${type}`;
messageDiv.innerHTML = `${text}
`;
chatMessages.appendChild(messageDiv);
chatMessages.scrollTop = chatMessages.scrollHeight;
}
function addSource(text) {
const messageDiv = document.createElement('div');
messageDiv.className = 'message-source';
messageDiv.textContent = text;
chatMessages.appendChild(messageDiv);
}
async function loadBooks() {
try {
const response = await fetch('/api/books');
const books = await response.json();
bookSelect.innerHTML = '';
books.forEach(book => {
const option = document.createElement('option');
option.value = book.title;
option.textContent = book.title;
bookSelect.appendChild(option);
});
} catch (error) {
console.error('Error loading books:', error);
}
}
async function loadSchedules() {
try {
const response = await fetch('/api/schedules');
const schedules = await response.json();
schedulesTable.innerHTML = '';
schedules.forEach(schedule => {
const row = document.createElement('tr');
row.innerHTML = `
${schedule.id} |
${schedule.time} |
${schedule.book || 'Todos os livros'} |
${schedule.enabled ? 'Ativo' : 'Inativo'} |
|
`;
schedulesTable.appendChild(row);
});
} catch (error) {
console.error('Error loading schedules:', error);
}
}
async function deleteSchedule(id) {
if (!confirm('Tem certeza que deseja remover este agendamento?')) return;
try {
const response = await fetch(`/api/schedules/${id}`, {
method: 'DELETE'
});
if (!response.ok) throw new Error('Failed to delete schedule');
loadSchedules();
} catch (error) {
alert('Erro ao remover agendamento');
console.error(error);
}
}
// Upload file
const fileInput = document.getElementById('fileInput');
fileInput.addEventListener('change', async function(e) {
const file = e.target.files[0];
if (!file) return;
const formData = new FormData();
formData.append('file', file);
try {
const response = await fetch('/api/upload', {
method: 'POST',
body: formData
});
if (!response.ok) throw new Error('Failed to upload file');
const data = await response.json();
alert(`Sucesso: ${data.message}`);
// Reload books
loadBooks();
} catch (error) {
alert('Erro ao fazer upload do arquivo');
console.error(error);
}
});
});
```
- [ ] **Passo 3: Criar index.html**
```html
Sistema RAG Espiritismo
```
---
## Tarefa 6: Serviço de Telegram Bot
**Arquivos:**
- Criar: `backend/app/services/bot_service.py`
- Criar: `backend/app/tasks/celery_tasks.py`
**Interfaces:**
- Consumes: RAGService, ScheduleManager
- Produces: Bot Telegram funcional com comandos de agendamento
- [ ] **Passo 1: Criar bot_service.py**
```python
import logging
import random
from typing import Dict
from pyTelegramBotAPI import telebot
import os
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
class TelegramBot:
def __init__(self, token: str, chat_id: str):
self.bot = telebot.TeleBot(token)
self.chat_id = chat_id
self._register_handlers()
def _register_handlers(self):
"""Register bot command handlers."""
@self.bot.message_handler(commands=['start'])
def send_welcome(message):
self.bot.reply_to(
message,
"📚 *Bem-vindo ao Sistema RAG Espiritismo!*\n\n"
"Comandos disponíveis:\n"
"/start - Iniciar bot\n"
"/help - Ajuda\n"
"/list - Listar livros disponíveis\n"
"/schedule - Configurar horário de mensagem\n"
"/schedule list - Ver agendamentos\n"
"/schedule remove - Remover agendamento\n"
"/content - Receber conteúdo agora\n\n"
"Use o sistema web para gerenciar melhor os agendamentos!"
)
@self.bot.message_handler(commands=['help'])
def send_help(message):
self.bot.reply_to(
message,
"📖 *Comandos do Bot Telegram*\n\n"
"/start - Iniciar\n"
"/help - Esta mensagem\n"
"/list - Ver livros disponíveis\n"
"/schedule [livro] - Agendar mensagem\n"
" Ex: /schedule 08:00\n"
" Ex: /schedule 08:00 \"Livro dos Espíritos\"\n"
"/schedule list - Ver todos os agendamentos\n"
"/schedule remove - Remover agendamento\n"
"/content - Receber conteúdo agora"
)
@self.bot.message_handler(commands=['list'])
def list_books(message):
books = self.rag_service.qdrant_service.get_books()
if books:
books_text = "\n".join([f"{i+1}. {book}" for i, book in enumerate(books)])
self.bot.reply_to(
message,
f"📚 *Livros disponíveis:*\n\n{books_text}"
)
else:
self.bot.reply_to(message, "Não há livros na base de dados.")
@self.bot.message_handler(commands=['content'])
def send_content(message):
try:
book = self.rag_service.query("Conteúdo aleatório sobre espiritismo", top_k=1)
self.bot.reply_to(message, book['response'])
except Exception as e:
logger.error(f"Error sending content: {e}")
self.bot.reply_to(message, "Erro ao gerar conteúdo.")
@self.bot.message_handler(commands=['schedule'])
def schedule_message(message):
args = message.text.split()[1:]
if not args:
self.bot.reply_to(message, "Uso: /schedule [livro]")
return
time = args[0]
if len(args) > 1:
book = " ".join(args[1:])
else:
book = None
try:
self.schedule_manager.create(time=time, book=book)
self.bot.reply_to(
message,
f"✅ Agendamento criado para {time}\n"
f"📝 Livro: {book or 'Todos os livros'}"
)
except Exception as e:
logger.error(f"Error creating schedule: {e}")
self.bot.reply_to(message, f"Erro ao criar agendamento: {e}")
@self.bot.message_handler(commands=['schedule', 'list'])
def list_schedules(message):
schedules = self.schedule_manager.get_all()
if not schedules:
self.bot.reply_to(message, "Não há agendamentos.")
return
schedules_text = "\n".join([
f"{s.id}. {s.time} - {s.book or 'Todos'} - {'✅' if s.enabled else '❌'}"
for s in schedules
])
self.bot.reply_to(message, f"📅 *Agendamentos:*\n\n{schedules_text}")
@self.bot.message_handler(commands=['schedule'])
def remove_schedule(message):
try:
args = message.text.split()
if len(args) < 3:
self.bot.reply_to(message, "Uso: /schedule remove ")
return
schedule_id = int(args[2])
self.schedule_manager.delete(schedule_id)
self.bot.reply_to(message, f"✅ Agendamento {schedule_id} removido.")
except ValueError:
self.bot.reply_to(message, "ID deve ser um número.")
except Exception as e:
logger.error(f"Error removing schedule: {e}")
self.bot.reply_to(message, f"Erro ao remover agendamento: {e}")
def start(self):
"""Start the bot."""
logger.info("Starting Telegram bot...")
self.bot.infinity_polling(timeout=10, long_polling_timeout=1)
```
- [ ] **Passo 2: Criar celery_tasks.py**
```python
from celery import Celery
from celery.schedules import crontab
from app.services.bot_service import TelegramBot
from app.services.rag_service import RAGService
from app.utils.schedule_manager import ScheduleManager
import os
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
celery = Celery(
"espiritismo_tasks",
broker=os.getenv("CELERY_BROKER_URL", "redis://redis:6379/0"),
backend=os.getenv("CELERY_RESULT_BACKEND", "redis://redis:6379/0"),
)
# Initialize services
qdrant_service = qdrant_client.QdrantClient(
url=os.getenv("QDRANT_URL", "http://qdrant:6333")
)
rag_service = RAGService(qdrant_service, os.getenv("GPT4ALL_MODEL", "llama-3.2-3b-instruct-q4_0.gguf"))
schedule_manager = ScheduleManager("/app/schedules.json")
bot = TelegramBot(os.getenv("TELEGRAM_BOT_TOKEN", ""), os.getenv("TELEGRAM_CHAT_ID", ""))
@celery.task
def send_scheduled_message():
"""Send a random message from the books."""
logger.info("Sending scheduled message...")
try:
# Get all books
books = qdrant_service.get_books()
if not books:
logger.warning("No books found in database")
return
# Choose a random book
book = random.choice(books)
# Generate random query
queries = [
"O que diz o livro sobre amor?",
"Qual é a mensagem principal deste livro?",
"Dê uma reflexão sobre este livro",
"Conte uma história importante deste livro"
]
query = random.choice(queries)
# Query the RAG system
result = rag_service.query(query, top_k=1)
# Send via Telegram
message = f"📖 *{book}*\n\n{result['response']}"
bot.bot.send_message(
bot.chat_id,
message,
parse_mode="Markdown"
)
logger.info(f"Message sent successfully: {book}")
except Exception as e:
logger.error(f"Error sending scheduled message: {e}")
raise
@celery.periodic_task(
run_every=crontab(minute="*"),
)
def check_schedules():
"""Check and execute scheduled messages."""
logger.info("Checking schedules...")
schedules = schedule_manager.get_all()
for schedule in schedules:
if not schedule.enabled:
continue
# Check if this is the scheduled time
current_time = datetime.now().strftime("%H:%M")
if current_time == schedule.time:
send_scheduled_message.delay()
logger.info("Schedule check completed")
```
---
## Tarefa 7: Utilitários Adicionais
**Arquivos:**
- Criar: `backend/app/utils/schedule_manager.py`
- Criar: `backend/app/__init__.py`
**Interfaces:**
- Consumes: N/A
- Produces: Gerenciamento de agendamentos funcional
- [ ] **Passo 1: Criar __init__.py**
```python
from .models import ChatRequest, ChatResponse, BookInfo, ChatRequestResponse, UploadResponse, ScheduleInfo, ScheduleRequest
__all__ = [
"ChatRequest",
"ChatResponse",
"BookInfo",
"ChatRequestResponse",
"UploadResponse",
"ScheduleInfo",
"ScheduleRequest",
]
```
- [ ] **Passo 2: Criar schedule_manager.py**
```python
import json
import os
from datetime import datetime
from typing import List, Dict
from dataclasses import dataclass, asdict, field
import pytz
@dataclass
class ScheduleInfo:
id: int
time: str
book: Optional[str]
enabled: bool
created_at: str
next_run: Optional[str] = None
class ScheduleManager:
def __init__(self, file_path: str = "schedules.json"):
self.file_path = file_path
self.schedules: List[Dict] = []
self.next_id: int = 1
if os.path.exists(file_path):
self._load()
def _load(self):
"""Load schedules from file."""
with open(self.file_path, 'r') as f:
data = json.load(f)
self.schedules = data.get("schedules", [])
self.next_id = data.get("next_id", 1)
def _save(self):
"""Save schedules to file."""
data = {
"schedules": self.schedules,
"next_id": self.next_id
}
with open(self.file_path, 'w') as f:
json.dump(data, f, indent=2)
def get_all(self) -> List[Dict]:
"""Get all schedules."""
return self.schedules
def get(self, schedule_id: int) -> Dict:
"""Get a specific schedule."""
for schedule in self.schedules:
if schedule["id"] == schedule_id:
return schedule
return None
def create(self, time: str, book: str = None, enabled: bool = True) -> Dict:
"""Create a new schedule."""
schedule = {
"id": self.next_id,
"time": time,
"book": book,
"enabled": enabled,
"created_at": datetime.now().isoformat(),
"next_run": None
}
self.schedules.append(schedule)
self.next_id += 1
self._save()
return schedule
def delete(self, schedule_id: int):
"""Delete a schedule."""
self.schedules = [s for s in self.schedules if s["id"] != schedule_id]
self._save()
def update(self, schedule_id: int, enabled: bool = None, book: str = None):
"""Update a schedule."""
for schedule in self.schedules:
if schedule["id"] == schedule_id:
if enabled is not None:
schedule["enabled"] = enabled
if book is not None:
schedule["book"] = book
self._save()
break
```
---
## Tarefa 8: Dockerfile e Build
**Arquivos:**
- Criar: `backend/Dockerfile`
**Interfaces:**
- Consumes: N/A
- Produces: Imagem Docker funcional para todos os serviços
- [ ] **Passo 1: Criar Dockerfile**
```dockerfile
FROM python:3.11-slim
WORKDIR /app
# Install system dependencies
RUN apt-get update && apt-get install -y \
build-essential \
&& rm -rf /var/lib/apt/lists/*
# Copy requirements
COPY requirements.txt .
# Install Python dependencies
RUN pip install --no-cache-dir -r requirements.txt
# Create directories
RUN mkdir -p /app/uploads /app/index /app/schedules.json
# Copy application
COPY app/ ./app/
COPY frontend/ ./frontend/
# Expose ports
EXPOSE 8000
# Run the application
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
```
---
## Tarefa 9: Build e Testes
**Arquivos:**
- N/A
**Interfaces:**
- Consumes: Todo o código implementado
- Produces: Sistema funcional testado
- [ ] **Passo 1: Build da aplicação**
```bash
cd backend
docker-compose up --build -d
```
- [ ] **Passo 2: Indexar livros**
```bash
docker-compose run --rm web python scripts/index_books.py
```
- [ ] **Passo 3: Verificar logs**
```bash
docker-compose logs -f web
docker-compose logs -f celery
docker-compose logs -f celery-beat
```
- [ ] **Passo 4: Testar API**
```bash
curl http://localhost:8000/api/books
curl -X POST http://localhost:8000/api/chat \
-H "Content-Type: application/json" \
-d '{"message": "O que é espiritismo?"}'
```
- [ ] **Passo 5: Testar index.html**
```bash
# Acessar: http://localhost:8000
```
- [ ] **Passo 6: Iniciar bot Telegram**
```bash
# Iniciar bot separadamente
docker-compose run --rm web python -m app.services.bot_service
```
---
## Review Focus
1. **Consulta web**: Sistema retorna resposta com contexto e fontes
2. **Upload PDF**: PDF uploadado é indexado automaticamente
3. **Agendamento Telegram**: Mensagens enviadas no horário correto
4. **Indexação RAG**: Livros processados corretamente em chunks
5. **Integração web-bot**: Agendamentos funcionam em ambas as interfaces
---
## Checklist de Aceitação
- [ ] Sistema iniciado com `docker-compose up`
- [ ] 6 livros indexados no Qdrant
- [ ] API web responde a consultas
- [ ] Interface web funcional para chat
- [ ] Upload de novos PDFs funciona
- [ ] Agendamentos podem ser criados no web
- [ ] Agendamentos aparecem no bot Telegram
- [ ] Mensagens são enviadas no horário correto
- [ ] Sistema rodando com 0 erros