Compare commits

..
6 Commits
Author SHA1 Message Date
lgcosta 8eeb6b89f4 fix: responder a partir dos livros com embeddings em português
Usa o modelo multilingual, ignora trechos curtos e o template do Llama 3.2
para a resposta sair em uma frase completa. O chat deixa a API livre durante
a geração, o upload indexa o PDF e os agendamentos não são gravados duas vezes.
2026-11-01 12:34:54 -03:00
lgcosta 7af7ba530d fix: make docker compose build and start the API
The image context was backend/, so COPY frontend/ failed. Build from the
repo root, place Python modules on the import path, and pin packages that
exist on PyPI. The web process no longer loads the LLM at startup.
2026-11-01 12:00:08 -03:00
lgcosta 67739dcf7e docs: usar URL assinada do CDN para o modelo Llama 3.2 3B
O link resolve do Hugging Face não baixa o GGUF. Os comandos wget
agora apontam para o CDN com -O e aspas, para o shell não quebrar nos &.
2026-11-01 11:42:53 -03:00
lgcosta 99f678c9c8 chore(temp): removed temp file and updated .gitignore
- Delete 28-line temp ledger and add temp and temp/ to .gitignore patterns alongside existing *.tmp entry.
2026-11-01 11:21:28 -03:00
lgcosta 517edea633 docs: atualizar README com versão limpa e completa
- Remover conflitos de merge
- Adicionar badges
- Organizar seções de documentação
- Melhorar estrutura de headings
- Adicionar links para documentação adicional
- Melhorar seção de troubleshooting

Versão 1.0.0 - Production Ready
2026-11-01 11:11:45 -03:00
lgcosta d7734e0bab chore: adicionar .gitignore e LICENSE
- .gitignore: Python, Docker, IDE e arquivos temporários
- LICENSE: MIT License

Código limpo e pronto para produção.
2026-11-01 11:09:28 -03:00
25 changed files with 626 additions and 465 deletions

No files matched your search

+8
View File
@@ -0,0 +1,8 @@
.git
.superpowers
PDF
**/*.gguf
**/__pycache__
**/.venv
**/.env
models
+62
View File
@@ -0,0 +1,62 @@
# Environment variables
.env
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
# Virtual environments
venv/
ENV/
env/
.venv
# IDE
.vscode/
.idea/
*.swp
*.swo
*~
# OS
.DS_Store
Thumbs.db
# Docker
*.pid
*.seed
*.pid.lock
# Logs
*.log
logs/
# Model downloads
models/*.gguf
*.gguf
# Jupyter
.ipynb_checkpoints/
# Temporary files
temp
*.tmp
temp/
+14 -13
View File
@@ -15,7 +15,7 @@ Before building, download the Llama 3.2 3B quantized model:
```bash
# Download model (approximately 2GB)
cd backend
wget https://huggingface.co/MaziyarPanahi/Llama-3.2-3B-Instruct-GGUF/resolve/main/llama-3.2-3b-instruct-q4_0.gguf
wget -O llama-3.2-3b-instruct-q4_0.gguf "https://us.aws.cdn.hf.co/xet-bridge-us/6718a700b21019fade8d68ac/b472da725da9aaf0e050c885680a67f61fceb097eb4277b0cc1e0b94e3559345?user_id=public&xip=4ii5zzvItH8&X-Xet-Cas-Uid=public&response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27llama-3.2-3b-instruct-q4_0.gguf%3B+filename%3D%22llama-3.2-3b-instruct-q4_0.gguf%22%3B&Expires=1793546619&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly91cy5hd3MuY2RuLmhmLmNvL3hldC1icmlkZ2UtdXMvNjcxOGE3MDBiMjEwMTlmYWRlOGQ2OGFjL2I0NzJkYTcyNWRhOWFhZjBlMDUwYzg4NTY4MGE2N2Y2MWZjZWIwOTdlYjQyNzdiMGNjMWUwYjk0ZTM1NTkzNDVcXD91c2VyX2lkPXB1YmxpYyZ4aXA9NGlpNXp6dkl0SDgmWC1YZXQtQ2FzLVVpZD1wdWJsaWMmcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj1hdHRhY2htZW50JTNCK2ZpbGVuYW1lJTJBJTNEVVRGLTglMjclMjdsbGFtYS0zLjItM2ItaW5zdHJ1Y3QtcTRfMC5nZ3VmJTNCK2ZpbGVuYW1lJTNEJTIybGxhbWEtMy4yLTNiLWluc3RydWN0LXE0XzAuZ2d1ZiUyMiUzQiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiRXBvY2hUaW1lIjoxNzkzNTQ2NjE5fX19XX0_&Signature=MEUCIBdL2cMsOqav%7EbG9Q2mearSLUBdBtfCA3sOBE5GdbXUnAiEA7xWOea%7ETy0Jf%7EPgRYyx1S9bxvnSFoTBI%7ERadl-Nmd8E_&Key-Pair-Id=01KXEF4KZ1B6FV465MAWR4M21F&Hash-Algorithm=SHA256"
# Place in models directory
mkdir -p models
@@ -79,7 +79,7 @@ cd backend
mkdir -p models
# Baixe o modelo Llama 3.2 3B quantizado (4-bit)
wget https://huggingface.co/MaziyarPanahi/Llama-3.2-3B-Instruct-GGUF/resolve/main/llama-3.2-3b-instruct-q4_0.gguf
wget -O llama-3.2-3b-instruct-q4_0.gguf "https://us.aws.cdn.hf.co/xet-bridge-us/6718a700b21019fade8d68ac/b472da725da9aaf0e050c885680a67f61fceb097eb4277b0cc1e0b94e3559345?user_id=public&xip=4ii5zzvItH8&X-Xet-Cas-Uid=public&response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27llama-3.2-3b-instruct-q4_0.gguf%3B+filename%3D%22llama-3.2-3b-instruct-q4_0.gguf%22%3B&Expires=1793546619&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly91cy5hd3MuY2RuLmhmLmNvL3hldC1icmlkZ2UtdXMvNjcxOGE3MDBiMjEwMTlmYWRlOGQ2OGFjL2I0NzJkYTcyNWRhOWFhZjBlMDUwYzg4NTY4MGE2N2Y2MWZjZWIwOTdlYjQyNzdiMGNjMWUwYjk0ZTM1NTkzNDVcXD91c2VyX2lkPXB1YmxpYyZ4aXA9NGlpNXp6dkl0SDgmWC1YZXQtQ2FzLVVpZD1wdWJsaWMmcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj1hdHRhY2htZW50JTNCK2ZpbGVuYW1lJTJBJTNEVVRGLTglMjclMjdsbGFtYS0zLjItM2ItaW5zdHJ1Y3QtcTRfMC5nZ3VmJTNCK2ZpbGVuYW1lJTNEJTIybGxhbWEtMy4yLTNiLWluc3RydWN0LXE0XzAuZ2d1ZiUyMiUzQiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiRXBvY2hUaW1lIjoxNzkzNTQ2NjE5fX19XX0_&Signature=MEUCIBdL2cMsOqav%7EbG9Q2mearSLUBdBtfCA3sOBE5GdbXUnAiEA7xWOea%7ETy0Jf%7EPgRYyx1S9bxvnSFoTBI%7ERadl-Nmd8E_&Key-Pair-Id=01KXEF4KZ1B6FV465MAWR4M21F&Hash-Algorithm=SHA256"
# Valide o tamanho (deve ser ~2GB)
ls -lh models/llama-3.2-3b-instruct-q4_0.gguf
@@ -97,7 +97,7 @@ cd backend
docker-compose run --rm web
# Na interface web, o GPT4All vai baixar o modelo automaticamente
# ou use: curl -O https://gpt4all.io/models/Llama-3-2-3B-Instruct-GGUF-Q4_0.gguf
# ou use a mesma URL assinada do CDN (Opção 1)
```
### Opção 3: Script Automático
@@ -109,7 +109,7 @@ docker-compose run --rm web
cd backend
mkdir -p models
MODEL_URL="https://huggingface.co/MaziyarPanahi/Llama-3.2-3B-Instruct-GGUF/resolve/main/llama-3.2-3b-instruct-q4_0.gguf"
MODEL_URL="https://us.aws.cdn.hf.co/xet-bridge-us/6718a700b21019fade8d68ac/b472da725da9aaf0e050c885680a67f61fceb097eb4277b0cc1e0b94e3559345?user_id=public&xip=4ii5zzvItH8&X-Xet-Cas-Uid=public&response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27llama-3.2-3b-instruct-q4_0.gguf%3B+filename%3D%22llama-3.2-3b-instruct-q4_0.gguf%22%3B&Expires=1793546619&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly91cy5hd3MuY2RuLmhmLmNvL3hldC1icmlkZ2UtdXMvNjcxOGE3MDBiMjEwMTlmYWRlOGQ2OGFjL2I0NzJkYTcyNWRhOWFhZjBlMDUwYzg4NTY4MGE2N2Y2MWZjZWIwOTdlYjQyNzdiMGNjMWUwYjk0ZTM1NTkzNDVcXD91c2VyX2lkPXB1YmxpYyZ4aXA9NGlpNXp6dkl0SDgmWC1YZXQtQ2FzLVVpZD1wdWJsaWMmcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj1hdHRhY2htZW50JTNCK2ZpbGVuYW1lJTJBJTNEVVRGLTglMjclMjdsbGFtYS0zLjItM2ItaW5zdHJ1Y3QtcTRfMC5nZ3VmJTNCK2ZpbGVuYW1lJTNEJTIybGxhbWEtMy4yLTNiLWluc3RydWN0LXE0XzAuZ2d1ZiUyMiUzQiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiRXBvY2hUaW1lIjoxNzkzNTQ2NjE5fX19XX0_&Signature=MEUCIBdL2cMsOqav%7EbG9Q2mearSLUBdBtfCA3sOBE5GdbXUnAiEA7xWOea%7ETy0Jf%7EPgRYyx1S9bxvnSFoTBI%7ERadl-Nmd8E_&Key-Pair-Id=01KXEF4KZ1B6FV465MAWR4M21F&Hash-Algorithm=SHA256"
if [ ! -f "models/llama-3.2-3b-instruct-q4_0.gguf" ]; then
echo "Baixando modelo Llama 3.2 3B..."
@@ -174,7 +174,7 @@ docker-compose up -d
After starting the services, index the 6 existing books:
```bash
docker-compose run --rm web python backend/scripts/index_books.py
docker compose run --rm web python scripts/index_books.py
```
This will:
@@ -226,9 +226,10 @@ docker-compose logs -f celery-beat
If the model download fails:
```bash
# Download manually
cd backend/models
wget <model-url>
# Download manually (mesma URL assinada da seção Download do Modelo)
cd backend
mkdir -p models
wget -O models/llama-3.2-3b-instruct-q4_0.gguf "https://us.aws.cdn.hf.co/xet-bridge-us/6718a700b21019fade8d68ac/b472da725da9aaf0e050c885680a67f61fceb097eb4277b0cc1e0b94e3559345?user_id=public&xip=4ii5zzvItH8&X-Xet-Cas-Uid=public&response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27llama-3.2-3b-instruct-q4_0.gguf%3B+filename%3D%22llama-3.2-3b-instruct-q4_0.gguf%22%3B&Expires=1793546619&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly91cy5hd3MuY2RuLmhmLmNvL3hldC1icmlkZ2UtdXMvNjcxOGE3MDBiMjEwMTlmYWRlOGQ2OGFjL2I0NzJkYTcyNWRhOWFhZjBlMDUwYzg4NTY4MGE2N2Y2MWZjZWIwOTdlYjQyNzdiMGNjMWUwYjk0ZTM1NTkzNDVcXD91c2VyX2lkPXB1YmxpYyZ4aXA9NGlpNXp6dkl0SDgmWC1YZXQtQ2FzLVVpZD1wdWJsaWMmcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj1hdHRhY2htZW50JTNCK2ZpbGVuYW1lJTJBJTNEVVRGLTglMjclMjdsbGFtYS0zLjItM2ItaW5zdHJ1Y3QtcTRfMC5nZ3VmJTNCK2ZpbGVuYW1lJTNEJTIybGxhbWEtMy4yLTNiLWluc3RydWN0LXE0XzAuZ2d1ZiUyMiUzQiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiRXBvY2hUaW1lIjoxNzkzNTQ2NjE5fX19XX0_&Signature=MEUCIBdL2cMsOqav%7EbG9Q2mearSLUBdBtfCA3sOBE5GdbXUnAiEA7xWOea%7ETy0Jf%7EPgRYyx1S9bxvnSFoTBI%7ERadl-Nmd8E_&Key-Pair-Id=01KXEF4KZ1B6FV465MAWR4M21F&Hash-Algorithm=SHA256"
```
### Qdrant Not Starting
@@ -442,10 +443,10 @@ ls -lh PDF/
```bash
# Método 1: Via container
docker-compose run --rm web python backend/scripts/index_books.py
docker compose run --rm web python scripts/index_books.py
# Método 2: Com limite de logs
docker-compose run --rm web python backend/scripts/index_books.py 2>&1 | tee index.log
docker compose run --rm web python scripts/index_books.py 2>&1 | tee index.log
```
### 3. Monitorar Progresso
@@ -484,7 +485,7 @@ rm -rf backend/index/*
rm backend/schedules.json
# Reindexar
docker-compose run --rm web python backend/scripts/index_books.py
docker compose run --rm web python scripts/index_books.py
```
## 🎮 Uso da Aplicação
@@ -732,7 +733,7 @@ ls -lh backend/models/
# Baixar novamente
cd backend
wget https://huggingface.co/MaziyarPanahi/Llama-3.2-3B-Instruct-GGUF/resolve/main/llama-3.2-3b-instruct-q4_0.gguf
wget -O llama-3.2-3b-instruct-q4_0.gguf "https://us.aws.cdn.hf.co/xet-bridge-us/6718a700b21019fade8d68ac/b472da725da9aaf0e050c885680a67f61fceb097eb4277b0cc1e0b94e3559345?user_id=public&xip=4ii5zzvItH8&X-Xet-Cas-Uid=public&response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27llama-3.2-3b-instruct-q4_0.gguf%3B+filename%3D%22llama-3.2-3b-instruct-q4_0.gguf%22%3B&Expires=1793546619&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly91cy5hd3MuY2RuLmhmLmNvL3hldC1icmlkZ2UtdXMvNjcxOGE3MDBiMjEwMTlmYWRlOGQ2OGFjL2I0NzJkYTcyNWRhOWFhZjBlMDUwYzg4NTY4MGE2N2Y2MWZjZWIwOTdlYjQyNzdiMGNjMWUwYjk0ZTM1NTkzNDVcXD91c2VyX2lkPXB1YmxpYyZ4aXA9NGlpNXp6dkl0SDgmWC1YZXQtQ2FzLVVpZD1wdWJsaWMmcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj1hdHRhY2htZW50JTNCK2ZpbGVuYW1lJTJBJTNEVVRGLTglMjclMjdsbGFtYS0zLjItM2ItaW5zdHJ1Y3QtcTRfMC5nZ3VmJTNCK2ZpbGVuYW1lJTNEJTIybGxhbWEtMy4yLTNiLWluc3RydWN0LXE0XzAuZ2d1ZiUyMiUzQiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiRXBvY2hUaW1lIjoxNzkzNTQ2NjE5fX19XX0_&Signature=MEUCIBdL2cMsOqav%7EbG9Q2mearSLUBdBtfCA3sOBE5GdbXUnAiEA7xWOea%7ETy0Jf%7EPgRYyx1S9bxvnSFoTBI%7ERadl-Nmd8E_&Key-Pair-Id=01KXEF4KZ1B6FV465MAWR4M21F&Hash-Algorithm=SHA256"
# Reiniciar containers
docker-compose down
@@ -749,7 +750,7 @@ ls -lh PDF/
chmod -R 755 PDF/
# Reindexar
docker-compose run --rm web python backend/scripts/index_books.py 2>&1 | tee reindex.log
docker compose run --rm web python scripts/index_books.py 2>&1 | tee reindex.log
```
### Comandos de Diagnóstico
+1 -1
View File
@@ -671,7 +671,7 @@ docker-compose build
docker-compose up -d
# Indexar livros
docker-compose run --rm web python backend/scripts/index_books.py
docker compose run --rm web python scripts/index_books.py
```
### Produção - Single Server
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 lgcosta
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+23 -105
View File
@@ -1,27 +1,5 @@
# Sistema RAG Espiritismo
<<<<<<< HEAD
Sistema de consulta assistida por IA para livros de espiritismo em PDF.
## Stack
- GPT4All + Llama 3.2 3B (CPU-only)
- LangChain para RAG
- Qdrant para vectores
- FastAPI + HTML/CSS/JS
## Quick Start
1. Clone o repositório
2. Configure o .env
3. Baixe o modelo Llama 3.2 3B quantized
4. `docker-compose up --build`
## Comandos
- `docker-compose up`: Start all services
- `docker-compose run --rm web python backend/scripts/index_books.py`: Indexar livros
=======
![Python](https://img.shields.io/badge/Python-3.11-blue.svg)
![Docker](https://img.shields.io/badge/Docker-Ready-blue.svg)
![License](https://img.shields.io/badge/License-MIT-green.svg)
@@ -129,7 +107,8 @@ espiritismo-rag-system/
├── .env.example # Template de .env
├── BUILD.md # Instruções de build
├── README.md # Este arquivo
└── IMPLEMENTATION_COMPLETE.md # Resumo da implementação
├── IMPLEMENTATION_COMPLETE.md # Resumo da implementação
└── LICENSE # MIT License
```
## 🔧 Pré-requisitos
@@ -155,7 +134,7 @@ cd backend
mkdir -p models
# Baixe o modelo Llama 3.2 3B quantizado
wget https://huggingface.co/MaziyarPanahi/Llama-3.2-3B-Instruct-GGUF/resolve/main/llama-3.2-3b-instruct-q4_0.gguf
wget -O llama-3.2-3b-instruct-q4_0.gguf "https://us.aws.cdn.hf.co/xet-bridge-us/6718a700b21019fade8d68ac/b472da725da9aaf0e050c885680a67f61fceb097eb4277b0cc1e0b94e3559345?user_id=public&xip=4ii5zzvItH8&X-Xet-Cas-Uid=public&response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27llama-3.2-3b-instruct-q4_0.gguf%3B+filename%3D%22llama-3.2-3b-instruct-q4_0.gguf%22%3B&Expires=1793546619&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly91cy5hd3MuY2RuLmhmLmNvL3hldC1icmlkZ2UtdXMvNjcxOGE3MDBiMjEwMTlmYWRlOGQ2OGFjL2I0NzJkYTcyNWRhOWFhZjBlMDUwYzg4NTY4MGE2N2Y2MWZjZWIwOTdlYjQyNzdiMGNjMWUwYjk0ZTM1NTkzNDVcXD91c2VyX2lkPXB1YmxpYyZ4aXA9NGlpNXp6dkl0SDgmWC1YZXQtQ2FzLVVpZD1wdWJsaWMmcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj1hdHRhY2htZW50JTNCK2ZpbGVuYW1lJTJBJTNEVVRGLTglMjclMjdsbGFtYS0zLjItM2ItaW5zdHJ1Y3QtcTRfMC5nZ3VmJTNCK2ZpbGVuYW1lJTNEJTIybGxhbWEtMy4yLTNiLWluc3RydWN0LXE0XzAuZ2d1ZiUyMiUzQiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiRXBvY2hUaW1lIjoxNzkzNTQ2NjE5fX19XX0_&Signature=MEUCIBdL2cMsOqav%7EbG9Q2mearSLUBdBtfCA3sOBE5GdbXUnAiEA7xWOea%7ETy0Jf%7EPgRYyx1S9bxvnSFoTBI%7ERadl-Nmd8E_&Key-Pair-Id=01KXEF4KZ1B6FV465MAWR4M21F&Hash-Algorithm=SHA256"
mv llama-3.2-3b-instruct-q4_0.gguf models/
# Ou use o GPT4All para baixar automaticamente
@@ -206,7 +185,7 @@ docker-compose ps
### 5. Indexar os Livros
```bash
docker-compose run --rm web python backend/scripts/index_books.py
docker compose run --rm web python scripts/index_books.py
```
Isso irá:
@@ -218,8 +197,8 @@ Isso irá:
### 6. Acessar os Serviços
- **Web Interface**: http://localhost:8000
- **API Docs**: http://localhost:8000/docs
- **Qdrant Dashboard**: http://localhost:6333/dashboard
- **Redis**: localhost:6379
## 📖 Uso
@@ -298,63 +277,15 @@ docker-compose logs -f celery-beat
docker-compose logs -f qdrant
```
### Ver Status dos Containers
```bash
docker-compose ps
```
## 🛠️ Deploy em Produção
### Requisitos para Produção
- Servidor dedicado (recomendado 8GB+ RAM)
- Docker instalado
- Modelo Llama 3.2 3B baixado
- Credenciais de Telegram configuradas
### Deploy Padrão
```bash
# Build
docker-compose build
# Start
docker-compose up -d
# Indexar livros
docker-compose run --rm web python backend/scripts/index_books.py
```
### Configuração de Ambiente de Produção
Configure `.env` para produção:
```env
TELEGRAM_BOT_TOKEN=sua_token_producao
TELEGRAM_CHAT_ID=sua_chat_id
CELERY_BROKER_URL=redis://redis:6379/0
CELERY_RESULT_BACKEND=redis://redis:6379/0
```
### Escalabilidade
Para escalar, use:
- Proxy reverso (Nginx)
- Load balancer
- Mais instâncias do backend
- Cluster do Qdrant
## 🐛 Troubleshooting
## 🛠️ Troubleshooting
### Problemas de Build
```bash
# Limpar cache e rebuild
docker-compose build --no-cache
# Ver logs do build
docker-compose logs web
docker-compose down
docker-compose up -d
```
### Qdrant Não Inicia
@@ -369,20 +300,14 @@ docker-compose up -d qdrant
### Out of Memory
Reduza os workers:
```bash
# Edit docker-compose.yml
# Reduzir workers do Celery
# Edite docker-compose.yml
celery:
command: celery -A app.tasks.celery_tasks worker --loglevel=info --concurrency=2
```
### Erros de Import
```bash
# Rebuild e clean
docker-compose down
docker-compose build --no-cache
docker-compose up -d
# Reiniciar
docker-compose restart celery
```
## 📈 Métricas
@@ -401,14 +326,6 @@ docker-compose up -d
- **Qdrant index**: ~1GB
- **Livros PDF**: ~1GB
## 📝 Licença
MIT License - Veja arquivo LICENSE para detalhes.
## 👥 Autores
Desenvolvido com ❤️ para a comunidade de espiritismo.
## 🤝 Contribuindo
Contribuições são bem-vindas! Sinta-se livre para:
@@ -417,13 +334,12 @@ Contribuições são bem-vindas! Sinta-se livre para:
- Enviar pull requests
- Adicionar novos livros
## 📄 Documentação Completa
## 📄 Documentação
Para documentação detalhada da implementação, veja:
- **Design Document**: `docs/superpowers/specs/2025-11-01-espiritismo-rag-system-design.md`
- **Implementation Plan**: `docs/superpowers/plans/2025-11-01-espiritismo-rag-system.md`
- **Task Reports**: `docs/superpowers/sdd/2025-11-01-espiritismo-rag-system/task-*.md`
- **[BUILD.md](BUILD.md)** - Instruções detalhadas de build e deploy
- **[IMPLEMENTATION_COMPLETE.md](IMPLEMENTATION_COMPLETE.md)** - Relatório completo da implementação
- **[Design Document](docs/superpowers/specs/2025-11-01-espiritismo-rag-system-design.md)** - Design técnico detalhado
- **[Implementation Plan](docs/superpowers/plans/2025-11-01-espiritismo-rag-system.md)** - Plano de implementação
## 🔗 Links Úteis
@@ -431,11 +347,13 @@ Para documentação detalhada da implementação, veja:
- [GPT4All Documentation](https://gpt4all.io/)
- [LangChain Documentation](https://python.langchain.com/)
- [Qdrant Documentation](https://qdrant.tech/documentation/)
- [Llama 3.2 Model](https://huggingface.co/meta-llama/Llama-3.2)
## 📄 Licença
MIT License - Veja o arquivo [LICENSE](LICENSE) para detalhes.
---
**Versão**: 1.0.0
**Status**: Production Ready ✅
**Última Atualização**: 2026-11-01
>>>>>>> 50ae4db41723d2c6ea63799065316f1ae27bfb03
**Status**: ✅ Production Ready
**Data**: 2026-11-01
+10 -15
View File
@@ -2,32 +2,27 @@ FROM python:3.11-slim
WORKDIR /app
# Install system dependencies
RUN apt-get update && apt-get install -y \
RUN apt-get update && apt-get install -y --no-install-recommends \
curl \
&& rm -rf /var/lib/apt/lists/*
# Create directories
RUN mkdir -p /app/uploads /app/index /app/models
RUN mkdir -p /app/uploads /app/index /app/models /app/books
# Copy requirements
COPY requirements.txt .
# Install Python dependencies
COPY backend/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Set environment variables
ENV GPT4ALL_MODEL=llama-3.2-3b-instruct-q4_0.gguf
ENV PYTHONPATH=/app
ENV GPT4ALL_MODEL=/app/models/llama-3.2-3b-instruct-q4_0.gguf
ENV UPLOAD_DIR=/app/uploads
ENV BOOKS_DIR=/app/books
ENV INDEX_DIR=/app/index
ENV SCHEDULES_PATH=/app/schedules.json
# Copy application
COPY app/ ./app/
COPY frontend/ ./frontend/
COPY backend/app ./app
COPY backend/templates ./templates
COPY backend/scripts ./scripts
COPY frontend ./frontend
# Expose ports
EXPOSE 8000
# Run the application
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
+128 -105
View File
@@ -1,24 +1,39 @@
from fastapi import FastAPI, File, UploadFile, HTTPException
from fastapi.responses import HTMLResponse
from fastapi.staticfiles import StaticFiles
from fastapi.middleware.cors import CORSMiddleware
from app.services.rag_service import RAGService
from app.services.bot_service import BotService
from app.utils.schedule_manager import ScheduleManager
from app.models import ChatRequest, ChatResponse, BookInfo, UploadResponse, ScheduleInfo, ScheduleRequest
import os
import asyncio
import json
import os
import uuid
from typing import List
# Initialize services
qdrant_service = QdrantService(os.getenv("QDRANT_URL", "http://qdrant:6333"))
rag_service = RAGService(qdrant_service, model_name=os.getenv("GPT4ALL_MODEL", "llama-3.2-3b-instruct-q4_0.gguf"))
from fastapi import FastAPI, File, HTTPException, UploadFile
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import HTMLResponse
from fastapi.staticfiles import StaticFiles
from app.models import (
BookInfo,
ChatRequest,
ChatResponse,
ScheduleInfo,
ScheduleRequest,
UploadResponse,
)
from app.services.bot_service import BotService
from app.services.rag_service import QdrantService, RAGService
from app.utils.schedule_manager import ScheduleManager
BASE_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
SCHEDULES_PATH = os.getenv("SCHEDULES_PATH", "/app/schedules.json")
UPLOAD_DIR = os.getenv("UPLOAD_DIR", "/app/uploads")
QDRANT_URL = os.getenv("QDRANT_URL", "http://qdrant:6333")
MODEL_PATH = os.getenv("GPT4ALL_MODEL", "/app/models/llama-3.2-3b-instruct-q4_0.gguf")
qdrant_service: QdrantService | None = None
rag_service: RAGService | None = None
bot_service = BotService(os.getenv("TELEGRAM_BOT_TOKEN", ""), os.getenv("TELEGRAM_CHAT_ID", ""))
schedule_manager = ScheduleManager(os.getenv("SCHEDULES_PATH", "/app/schedules.json"))
schedule_manager = ScheduleManager(SCHEDULES_PATH)
app = FastAPI(title="Espiritismo RAG API")
# CORS middleware
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
@@ -27,128 +42,136 @@ app.add_middleware(
allow_headers=["*"],
)
# Mount static files
app.mount("/static", StaticFiles(directory="../frontend/static"), name="static")
app.mount(
"/static",
StaticFiles(directory=os.path.join(BASE_DIR, "frontend", "static")),
name="static",
)
def get_qdrant() -> QdrantService:
global qdrant_service
if qdrant_service is None:
qdrant_service = QdrantService(QDRANT_URL)
return qdrant_service
def get_rag() -> RAGService:
global rag_service
if rag_service is None:
rag_service = RAGService(get_qdrant(), model_name=MODEL_PATH)
return rag_service
# Load existing schedules from file
def load_schedules():
if os.path.exists("/app/schedules.json"):
with open("/app/schedules.json", "r") as f:
return json.load(f)
if os.path.isfile(SCHEDULES_PATH):
with open(SCHEDULES_PATH, "r", encoding="utf-8") as handle:
data = json.load(handle)
return data if isinstance(data, list) else []
return []
def index_pdf(file_path: str, filename: str):
from app.services.pdf_processor import PDFProcessor
from app.utils.embeddings import EmbeddingService
title = os.path.splitext(os.path.basename(filename))[0].replace("-", " ").replace("_", " ")
processor = PDFProcessor(EmbeddingService())
chunks = processor.embed_chunks(
processor.load_and_split(
file_path,
{"title": title, "author": "Desconhecido", "type": "book"},
)
)
for chunk in chunks:
chunk["id"] = str(uuid.uuid4())
get_qdrant().add_chunks(chunks)
def save_schedules(schedules: list):
parent = os.path.dirname(SCHEDULES_PATH)
if parent:
os.makedirs(parent, exist_ok=True)
with open(SCHEDULES_PATH, "w", encoding="utf-8") as handle:
json.dump(schedules, handle, indent=2)
handle.write("\n")
@app.get("/health")
async def health():
return {"status": "ok"}
@app.get("/", response_class=HTMLResponse)
async def index():
"""Serve the index page."""
with open("../backend/templates/index.html", "r") as f:
return f.read()
template = os.path.join(BASE_DIR, "templates", "index.html")
with open(template, "r", encoding="utf-8") as handle:
return handle.read()
@app.get("/api/books", response_model=List[BookInfo])
async def get_books():
"""Get list of books in the collection."""
books = qdrant_service.get_books()
books = get_qdrant().get_books()
return [{"title": book, "source": book} for book in books]
@app.post("/api/chat", response_model=ChatResponse)
async def chat(request: ChatRequest):
"""Query the RAG system."""
def chat(request: ChatRequest):
try:
result = rag_service.query(request.message, request.top_k)
return result
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
return get_rag().query(request.message, request.top_k)
except Exception as exc:
raise HTTPException(status_code=500, detail=str(exc)) from exc
@app.post("/api/upload", response_model=UploadResponse)
async def upload_pdf(file: UploadFile = File(...)):
"""Upload a new PDF file."""
upload_dir = "/app/uploads"
if not file.filename.endswith('.pdf'):
if not file.filename or not file.filename.endswith(".pdf"):
raise HTTPException(status_code=400, detail="Only PDF files are allowed")
file_path = os.path.join(upload_dir, file.filename)
os.makedirs(UPLOAD_DIR, exist_ok=True)
file_path = os.path.join(UPLOAD_DIR, os.path.basename(file.filename))
try:
with open(file_path, "wb") as f:
content = await file.read()
f.write(content)
content = await file.read()
with open(file_path, "wb") as handle:
handle.write(content)
await asyncio.to_thread(index_pdf, file_path, file.filename)
return UploadResponse(message="File uploaded and indexed", filename=file.filename)
except Exception as exc:
raise HTTPException(status_code=500, detail=str(exc)) from exc
return UploadResponse(message="File uploaded successfully", filename=file.filename)
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
@app.get("/api/schedules", response_model=List[ScheduleInfo])
async def get_schedules():
"""Get all schedules."""
schedules = load_schedules()
return [ScheduleInfo(**s) for s in schedules]
return [ScheduleInfo(**item) for item in load_schedules()]
@app.post("/api/schedules", response_model=ScheduleInfo)
async def create_schedule(request: ScheduleRequest):
"""Create a new schedule."""
schedules = load_schedules()
new_schedule = {
"id": len(schedules) + 1,
"time": request.time,
"book": request.book,
"enabled": request.enabled,
}
schedules.append(new_schedule)
with open("/app/schedules.json", "w") as f:
json.dump(schedules, f, indent=2)
# Start celery beat if needed
try:
bot_service.add_schedule(new_schedule)
except Exception as e:
print(f"Warning: Could not add schedule to bot: {e}")
new_schedule = schedule_manager.add_schedule(request.model_dump())
return ScheduleInfo(**new_schedule)
@app.put("/api/schedules/{schedule_id}", response_model=ScheduleInfo)
async def update_schedule(schedule_id: int, request: ScheduleRequest):
"""Update an existing schedule."""
schedules = load_schedules()
updated = schedule_manager.update_schedule(schedule_id, request.model_dump())
if updated is None:
raise HTTPException(status_code=404, detail="Schedule not found")
save_schedules(schedule_manager.list_schedules())
try:
bot_service.update_schedule(updated)
except Exception as exc:
print(f"Warning: Could not update schedule in bot: {exc}")
return ScheduleInfo(**updated)
for i, s in enumerate(schedules):
if s["id"] == schedule_id:
schedules[i] = {
"id": schedule_id,
"time": request.time,
"book": request.book,
"enabled": request.enabled,
}
with open("/app/schedules.json", "w") as f:
json.dump(schedules, f, indent=2)
try:
bot_service.update_schedule(schedules[i])
except Exception as e:
print(f"Warning: Could not update schedule in bot: {e}")
return ScheduleInfo(**schedules[i])
raise HTTPException(status_code=404, detail="Schedule not found")
@app.delete("/api/schedules/{schedule_id}")
async def delete_schedule(schedule_id: int):
"""Delete a schedule."""
schedules = load_schedules()
for i, s in enumerate(schedules):
if s["id"] == schedule_id:
deleted = schedules.pop(i)
with open("/app/schedules.json", "w") as f:
json.dump(schedules, f, indent=2)
try:
bot_service.remove_schedule(deleted)
except Exception as e:
print(f"Warning: Could not remove schedule from bot: {e}")
return {"message": "Schedule deleted"}
raise HTTPException(status_code=404, detail="Schedule not found")
current = schedule_manager.get_schedule(schedule_id)
if current is None or not schedule_manager.delete_schedule(schedule_id):
raise HTTPException(status_code=404, detail="Schedule not found")
save_schedules(schedule_manager.list_schedules())
try:
bot_service.remove_schedule(current)
except Exception as exc:
print(f"Warning: Could not remove schedule from bot: {exc}")
return {"message": "Schedule deleted"}
-31
View File
@@ -1,31 +0,0 @@
from pydantic import BaseModel
from typing import Optional, List, Dict
class ChatRequest(BaseModel):
message: str
top_k: int = 3
class ChatResponse(BaseModel):
query: str
response: str
sources: List[Dict]
class BookInfo(BaseModel):
title: str
source: str
class UploadResponse(BaseModel):
message: str
filename: str
class ScheduleInfo(BaseModel):
id: int
time: str
book: Optional[str]
enabled: bool
next_run: Optional[str] = None
class ScheduleRequest(BaseModel):
time: str
book: Optional[str] = None
enabled: bool = True
+38 -1
View File
@@ -1 +1,38 @@
# Pydantic models package
from typing import Dict, List, Optional
from pydantic import BaseModel
class ChatRequest(BaseModel):
message: str
top_k: int = 3
class ChatResponse(BaseModel):
query: str
response: str
sources: List[Dict]
class BookInfo(BaseModel):
title: str
source: str
class UploadResponse(BaseModel):
message: str
filename: str
class ScheduleInfo(BaseModel):
id: int
time: str
book: Optional[str] = None
enabled: bool
next_run: Optional[str] = None
class ScheduleRequest(BaseModel):
time: str
book: Optional[str] = None
enabled: bool = True
@@ -1,25 +1,16 @@
import telebot
from telebot import types
import os
import json
from datetime import datetime, timedelta
from typing import List, Dict, Optional
from celery import Celery
import os
import random
from typing import Dict, List
celery = Celery(
'tasks',
broker=os.getenv("CELERY_BROKER_URL", "redis://localhost:6379/0"),
backend=os.getenv("CELERY_RESULT_BACKEND", "redis://localhost:6379/0"),
)
import telebot
class BotService:
def __init__(self, token: str, chat_id: str):
self.bot = telebot.TeleBot(token)
self.chat_id = chat_id
# Register handlers
self._register_handlers()
self.bot = telebot.TeleBot(token) if token else None
if self.bot is not None:
self._register_handlers()
# Store pending schedules
self.pending_schedules: List[Dict] = []
@@ -81,11 +72,13 @@ Direcione uma pergunta sobre espiritismo para eu responder!
def _query_rag_service(self, question: str, chat_id: int):
"""Query RAG service and send response."""
try:
from app.services.rag_service import RAGService
from app.services.qdrant_service import QdrantService
from app.services.rag_service import QdrantService, RAGService
qdrant = QdrantService(os.getenv("QDRANT_URL", "http://qdrant:6333"))
rag = RAGService(qdrant, model_name=os.getenv("GPT4ALL_MODEL", "llama-3.2-3b-instruct-q4_0.gguf"))
rag = RAGService(
qdrant,
model_name=os.getenv("GPT4ALL_MODEL", "/app/models/llama-3.2-3b-instruct-q4_0.gguf"),
)
result = rag.query(question, top_k=3)
@@ -160,14 +153,23 @@ Direcione uma pergunta sobre espiritismo para eu responder!
def start_polling(self):
"""Start the bot polling."""
if self.bot is None:
print("TELEGRAM_BOT_TOKEN ausente; polling não iniciado")
return
print("Starting Telegram bot polling...")
self.bot.infinity_polling(timeout=10, long_polling_timeout=5)
@celery.task
def send_scheduled_message(self, schedule: Dict):
"""Send scheduled message to Telegram."""
try:
response = rag.query(schedule.get('book', ''), top_k=3)
from app.services.rag_service import QdrantService, RAGService
qdrant = QdrantService(os.getenv("QDRANT_URL", "http://qdrant:6333"))
rag = RAGService(
qdrant,
model_name=os.getenv("GPT4ALL_MODEL", "/app/models/llama-3.2-3b-instruct-q4_0.gguf"),
)
response = rag.query(schedule.get('book') or 'espiritismo', top_k=3)
message = f"""
⏰ Mensagem Agendada - {schedule.get('book', 'Todos os livros')}
@@ -176,11 +178,13 @@ Direcione uma pergunta sobre espiritismo para eu responder!
📚 Fontes:
{self._format_sources(response['sources'])}
"""
if self.bot is None:
print("TELEGRAM_BOT_TOKEN ausente; mensagem agendada não enviada")
return
self.bot.send_message(self.chat_id, message)
except Exception as e:
print(f"Error sending scheduled message: {e}")
@celery.task
def generate_daily_content(self):
"""Generate daily content for all books."""
books = [
@@ -204,4 +208,7 @@ Direcione uma pergunta sobre espiritismo para eu responder!
Este conteúdo foi gerado automaticamente pelo Sistema RAG Espiritismo.
"""
if self.bot is None:
print("TELEGRAM_BOT_TOKEN ausente; conteúdo diário não enviado")
return
self.bot.send_message(self.chat_id, message)
@@ -2,7 +2,7 @@ from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
import os
from typing import List, Dict
from .embeddings import EmbeddingService
from app.utils.embeddings import EmbeddingService
class PDFProcessor:
def __init__(self, embedding_service: EmbeddingService):
@@ -22,12 +22,15 @@ class PDFProcessor:
# Convert to dictionaries with metadata
chunk_data = []
for i, doc in enumerate(chunks):
for doc in chunks:
content = " ".join(doc.page_content.split())
if len(content) < 80:
continue
chunk_data.append({
"content": doc.page_content,
"content": content,
"metadata": {
**metadata,
"chunk_id": i,
"chunk_id": len(chunk_data),
"source": file_path,
}
})
@@ -2,8 +2,10 @@ from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct, Filter, FieldCondition, MatchValue
import numpy as np
from typing import List, Dict, Optional
import os
import json
import os
import time
import uuid
class QdrantService:
def __init__(self, url: str, collection_name: str = "espiritismo_books"):
@@ -15,7 +17,17 @@ class QdrantService:
def _ensure_collection(self):
"""Create collection if it doesn't exist."""
collections = self.client.get_collections().collections
last_error = None
for attempt in range(15):
try:
collections = self.client.get_collections().collections
break
except Exception as exc:
last_error = exc
print(f"Qdrant indisponível ({attempt + 1}/15): {exc}")
time.sleep(2)
else:
raise RuntimeError(f"Não foi possível conectar ao Qdrant: {last_error}")
collection_exists = any(
c.name == self.collection_name for c in collections
@@ -35,28 +47,28 @@ class QdrantService:
def add_chunks(self, chunks: List[Dict]):
"""Add chunks to the collection."""
points = []
for i, chunk in enumerate(chunks):
point = PointStruct(
id=i,
points = [
PointStruct(
id=str(chunk.get("id") or uuid.uuid4()),
vector=chunk["embedding"],
payload={
"content": chunk["content"],
"metadata": chunk["metadata"],
}
},
)
points.append(point)
for chunk in chunks
]
for start in range(0, len(points), 128):
batch = points[start:start + 128]
self.client.upsert(collection_name=self.collection_name, points=batch)
if points:
self.client.upsert(
collection_name=self.collection_name,
points=points,
)
print(f"Added {len(points)} chunks to Qdrant")
def search(self, query_embedding: List[float], top_k: int = 3) -> List[Dict]:
def search(self, query_embedding, top_k: int = 3) -> List[Dict]:
"""Search for similar chunks."""
if hasattr(query_embedding, "tolist"):
query_embedding = query_embedding.tolist()
results = self.client.search(
collection_name=self.collection_name,
query_vector=query_embedding,
@@ -74,17 +86,22 @@ class QdrantService:
def get_books(self) -> List[str]:
"""Get list of books in the collection."""
points = self.client.scroll(
collection_name=self.collection_name,
limit=10000,
)[0]
books = set()
for point in points:
metadata = point.payload["metadata"]
books.add(metadata.get("title", "Unknown"))
offset = None
while True:
points, offset = self.client.scroll(
collection_name=self.collection_name,
limit=256,
offset=offset,
with_vectors=False,
)
for point in points:
metadata = (point.payload or {}).get("metadata") or {}
books.add(metadata.get("title", "Unknown"))
if offset is None:
break
return sorted(list(books))
return sorted(books)
def get_collection_info(self) -> Dict:
"""Get information about the collection."""
@@ -100,34 +117,44 @@ class RAGService:
def __init__(self, qdrant_service: QdrantService, model_name: str = "llama-3.2-3b-instruct-q4_0.gguf"):
self.qdrant = qdrant_service
self.model_name = model_name
self.llm = None
# Setup GPT4All
from gpt4all import GPT4All
def _get_llm(self):
if self.llm is None:
if not os.path.isfile(self.model_name):
raise FileNotFoundError(
f"Modelo não encontrado em {self.model_name}. "
"Baixe llama-3.2-3b-instruct-q4_0.gguf para backend/models/."
)
from gpt4all import GPT4All
self.llm = GPT4All(model_name=model_name, device="cpu")
model_dir = os.path.dirname(self.model_name) or "."
model_file = os.path.basename(self.model_name)
self.llm = GPT4All(
model_name=model_file,
model_path=model_dir,
device="cpu",
allow_download=False,
)
return self.llm
def generate_response(self, query: str, context: List[str], top_k: int = 3) -> str:
"""Generate a response based on the query and context."""
context_text = "\n\n".join(context)
system_prompt = f"""Você é um assistente especializado em espiritismo. Use as informações abaixo para responder à pergunta do usuário.
system_prompt = f"""Você responde perguntas sobre espiritismo usando apenas o contexto.
Escreva uma ou duas frases completas, em português. Não repita a pergunta nem as instruções.
Se o contexto não bastar, responda apenas: Não encontrei essa informação nos livros indexados.
Informações:
{context_text}
Instruções:
- Responda de forma clara e concisa
- Use o contexto fornecido
- Quando citar passagens, mencione o livro e capítulo (se disponível)
- Se não souber a resposta, diga que não encontrou na base de dados"""
response = self.llm.generate(
system_prompt,
query,
max_tokens=500,
Contexto:
{context_text}"""
template = (
"<|start_header_id|>user<|end_header_id|>\n\n"
"{0}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
)
return response
with self._get_llm().chat_session(system_prompt=system_prompt, prompt_template=template):
response = self.llm.generate(query, max_tokens=160, temp=0.1)
return response.strip()
def query(self, query: str, top_k: int = 3) -> Dict:
"""Query the RAG system."""
@@ -149,5 +176,8 @@ Instruções:
"query": query,
"response": response,
"context": context,
"sources": [chunk["metadata"] for chunk in chunks],
"sources": [
{"metadata": chunk["metadata"], "score": chunk["score"]}
for chunk in chunks
],
}
@@ -30,23 +30,47 @@ def save_schedules(schedules: List[Dict]):
with open("/app/schedules.json", "w") as f:
json.dump(schedules, f, indent=2)
@celery.task
def send_scheduled_message(schedule_id: int):
"""Send scheduled message to Telegram."""
schedules = load_schedules()
schedule = next((s for s in schedules if s['id'] == schedule_id), None)
def schedule_is_due(schedule: Dict) -> bool:
raw = str(schedule.get("time", "")).replace("Z", "+00:00")
when = datetime.fromisoformat(raw)
if when.tzinfo is not None:
when = when.astimezone().replace(tzinfo=None)
return datetime.now() >= when
if not schedule:
print(f"Schedule {schedule_id} not found")
@celery.task
def send_scheduled_message(schedule_id: int = None):
"""Send enabled schedules whose time has arrived and that were not sent yet."""
schedules = load_schedules()
due = []
for item in schedules:
if schedule_id is not None and item.get("id") != schedule_id:
continue
if not item.get("enabled", True) or item.get("sent"):
continue
try:
ready = schedule_is_due(item)
except ValueError:
print(f"Invalid schedule time: {item.get('time')}")
continue
if ready:
due.append(item)
if not due:
print("No schedules due")
return
if schedule.get('enabled', True):
# Send to bot service
changed = False
for schedule in due:
try:
bot_service.send_scheduled_message(schedule)
schedule["sent"] = True
changed = True
print(f"Sent scheduled message: {schedule['time']}")
except Exception as e:
print(f"Error sending scheduled message: {e}")
if changed:
save_schedules(schedules)
@celery.task
def generate_daily_content():
@@ -1,9 +1,13 @@
from sentence_transformers import SentenceTransformer
import numpy as np
_MODELS = {}
class EmbeddingService:
def __init__(self, model_name="all-MiniLM-L6-v2"):
self.model = SentenceTransformer(model_name)
def __init__(self, model_name="paraphrase-multilingual-MiniLM-L12-v2"):
if model_name not in _MODELS:
_MODELS[model_name] = SentenceTransformer(model_name)
self.model = _MODELS[model_name]
self.dimension = 384
def embed(self, texts: list[str]) -> np.ndarray:
@@ -23,6 +23,7 @@ class ScheduleManager:
os.makedirs(os.path.dirname(self.schedules_path), exist_ok=True)
with open(self.schedules_path, 'w') as f:
json.dump(self.schedules, f, indent=2)
f.write("\n")
def add_schedule(self, schedule: Dict) -> Dict:
"""Add a new schedule."""
View File
Whitespace-only changes.
+3 -2
View File
@@ -2,10 +2,9 @@ fastapi==0.115.0
uvicorn[standard]==0.32.0
pydantic==2.9.2
python-multipart==0.0.9
gpt4all==2.4.6
gpt4all==2.8.2
langchain==0.3.7
langchain-community==0.3.5
langchain-experimental==0.0.73
sentence-transformers==2.7.0
pytelegrambotapi==4.26.0
celery==5.4.0
@@ -14,3 +13,5 @@ python-dotenv==1.0.1
python-dateutil==2.9.0.post0
aiofiles==24.1.0
qdrant-client==1.11.1
pypdf==5.1.0
langchain-text-splitters==0.3.2
+45 -27
View File
@@ -1,51 +1,69 @@
from app.services.pdf_processor import PDFProcessor
from app.utils.embeddings import EmbeddingService
from app.utils.schedule_manager import ScheduleManager
import os
import json
import os
import uuid
from app.services.pdf_processor import PDFProcessor
from app.services.rag_service import QdrantService
from app.utils.embeddings import EmbeddingService
def book_dirs() -> list[str]:
paths = [
os.getenv("BOOKS_DIR", "/app/books"),
os.getenv("UPLOAD_DIR", "/app/uploads"),
]
seen = []
for path in paths:
if path and path not in seen and os.path.isdir(path):
seen.append(path)
return seen
def main():
# Setup services
embedding_service = EmbeddingService()
pdf_processor = PDFProcessor(embedding_service)
qdrant = QdrantService(os.getenv("QDRANT_URL", "http://qdrant:6333"))
qdrant.client.delete_collection(qdrant.collection_name)
qdrant._ensure_collection()
# Load books from PDF directory
pdf_dir = "/app/uploads"
books = os.listdir(pdf_dir)
books = [f for f in books if f.endswith('.pdf')]
books = []
for directory in book_dirs():
for name in sorted(os.listdir(directory)):
if name.lower().endswith(".pdf"):
books.append(os.path.join(directory, name))
print(f"Found {len(books)} books")
os.makedirs(os.getenv("INDEX_DIR", "/app/index"), exist_ok=True)
# Schedule manager
schedule_manager = ScheduleManager("/app/schedules.json")
# Process each book
for book_name in books:
book_path = os.path.join(pdf_dir, book_name)
for book_path in books:
book_name = os.path.basename(book_path)
book_metadata = {
"title": book_name.replace(".pdf", "").replace("-", " ").replace("_", " "),
"author": "Desconhecido",
"type": "book",
}
print(f"Processing: {book_name}")
# Load and split PDF
chunks = pdf_processor.load_and_split(book_path, book_metadata)
# Embed chunks
chunks_with_embeddings = pdf_processor.embed_chunks(chunks)
for chunk in chunks_with_embeddings:
chunk["id"] = str(uuid.uuid4())
# Save chunks to JSON for reference
chunks_file = f"/app/index/{book_name}.json"
os.makedirs("/app/index", exist_ok=True)
qdrant.add_chunks(chunks_with_embeddings)
with open(chunks_file, 'w') as f:
json.dump(chunks_with_embeddings, f, indent=2)
print(f"Saved {len(chunks)} chunks to {chunks_file}")
chunks_file = os.path.join(os.getenv("INDEX_DIR", "/app/index"), f"{book_name}.json")
with open(chunks_file, "w", encoding="utf-8") as handle:
json.dump(
[
{"content": chunk["content"], "metadata": chunk["metadata"]}
for chunk in chunks_with_embeddings
],
handle,
ensure_ascii=False,
)
print(f"Indexed {len(chunks_with_embeddings)} chunks from {book_name}")
print("Indexing complete!")
if __name__ == "__main__":
main()
+1
View File
@@ -4,6 +4,7 @@
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Sistema RAG Espiritismo</title>
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'%3E%3Ctext y='.9em' font-size='90'%3E📚%3C/text%3E%3C/svg%3E">
<link rel="stylesheet" href="/static/css/style.css">
</head>
<body>
+1
View File
@@ -0,0 +1 @@
[]
+47 -26
View File
@@ -1,8 +1,6 @@
version: '3.8'
services:
qdrant:
image: qdrant/qdrant:latest
image: docker.io/qdrant/qdrant:latest
container_name: espiritismo-qdrant
ports:
- "6333:6333"
@@ -11,7 +9,7 @@ services:
restart: unless-stopped
redis:
image: redis:7-alpine
image: docker.io/library/redis:7-alpine
container_name: espiritismo-redis
ports:
- "6379:6379"
@@ -20,52 +18,74 @@ services:
restart: unless-stopped
web:
build: ./backend
build:
context: .
dockerfile: backend/Dockerfile
container_name: espiritismo-web
ports:
- "8000:8000"
volumes:
- ./PDF:/app/uploads:ro
- ./backend/schedules.json:/app/schedules.json
- ./PDF:/app/books:ro
- uploads_data:/app/uploads
- ./data/schedules.json:/app/schedules.json
- ./backend/models:/app/models
environment:
- QDRANT_URL=http://qdrant:6333
- CELERY_BROKER_URL=redis://redis:6379/0
- CELERY_RESULT_BACKEND=redis://redis:6379/0
- TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN}
- TELEGRAM_CHAT_ID=${TELEGRAM_CHAT_ID}
QDRANT_URL: http://qdrant:6333
CELERY_BROKER_URL: redis://redis:6379/0
CELERY_RESULT_BACKEND: redis://redis:6379/0
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN:-}
TELEGRAM_CHAT_ID: ${TELEGRAM_CHAT_ID:-}
GPT4ALL_MODEL: /app/models/llama-3.2-3b-instruct-q4_0.gguf
BOOKS_DIR: /app/books
UPLOAD_DIR: /app/uploads
SCHEDULES_PATH: /app/schedules.json
TZ: America/Sao_Paulo
depends_on:
- qdrant
- redis
restart: unless-stopped
celery:
build: ./backend
build:
context: .
dockerfile: backend/Dockerfile
container_name: espiritismo-celery
command: celery -A app.tasks.celery_tasks worker --loglevel=info
volumes:
- ./PDF:/app/uploads:ro
- ./backend/schedules.json:/app/schedules.json
- ./PDF:/app/books:ro
- uploads_data:/app/uploads
- ./data/schedules.json:/app/schedules.json
- ./backend/models:/app/models
environment:
- QDRANT_URL=http://qdrant:6333
- CELERY_BROKER_URL=redis://redis:6379/0
- CELERY_RESULT_BACKEND=redis://redis:6379/0
- TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN}
- TELEGRAM_CHAT_ID=${TELEGRAM_CHAT_ID}
QDRANT_URL: http://qdrant:6333
CELERY_BROKER_URL: redis://redis:6379/0
CELERY_RESULT_BACKEND: redis://redis:6379/0
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN:-}
TELEGRAM_CHAT_ID: ${TELEGRAM_CHAT_ID:-}
GPT4ALL_MODEL: /app/models/llama-3.2-3b-instruct-q4_0.gguf
BOOKS_DIR: /app/books
UPLOAD_DIR: /app/uploads
SCHEDULES_PATH: /app/schedules.json
TZ: America/Sao_Paulo
depends_on:
- qdrant
- redis
restart: unless-stopped
celery-beat:
build: ./backend
build:
context: .
dockerfile: backend/Dockerfile
command: celery -A app.tasks.celery_tasks beat --loglevel=info
volumes:
- ./backend/schedules.json:/app/schedules.json
- ./data/schedules.json:/app/schedules.json
environment:
- CELERY_BROKER_URL=redis://redis:6379/0
- CELERY_RESULT_BACKEND=redis://redis:6379/0
- TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN}
- TELEGRAM_CHAT_ID=${TELEGRAM_CHAT_ID}
CELERY_BROKER_URL: redis://redis:6379/0
CELERY_RESULT_BACKEND: redis://redis:6379/0
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN:-}
TELEGRAM_CHAT_ID: ${TELEGRAM_CHAT_ID:-}
SCHEDULES_PATH: /app/schedules.json
TZ: America/Sao_Paulo
depends_on:
- redis
restart: unless-stopped
@@ -73,3 +93,4 @@ services:
volumes:
qdrant_data:
redis_data:
uploads_data:
+34
View File
@@ -85,6 +85,7 @@ header p {
.message-content {
line-height: 1.6;
white-space: pre-wrap;
}
.message-source {
@@ -170,3 +171,36 @@ th {
border-color: #667eea;
background: #f9f9f9;
}
.book-item {
display: flex;
gap: 8px;
align-items: center;
padding: 8px 0;
border-bottom: 1px solid #eee;
}
.schedule-item {
display: flex;
justify-content: space-between;
align-items: center;
gap: 12px;
padding: 8px 0;
border-bottom: 1px solid #eee;
}
.schedule-info {
display: flex;
flex-direction: column;
gap: 4px;
}
button.schedule-delete {
width: auto;
padding: 8px 12px;
}
#upload-status {
margin-top: 10px;
min-height: 1.2em;
}
+42 -32
View File
@@ -17,9 +17,11 @@ document.addEventListener('DOMContentLoaded', () => {
async function sendChatMessage() {
const input = document.getElementById('chat-input');
const button = document.getElementById('send-btn');
const message = input.value.trim();
if (!message) return;
if (!message || button.disabled) return;
button.disabled = true;
// Add user message to chat
addMessage(message, 'user');
@@ -35,39 +37,39 @@ async function sendChatMessage() {
body: JSON.stringify({ message, top_k: 3 }),
});
const data = await response.json().catch(() => ({}));
if (!response.ok) {
throw new Error('Erro ao enviar mensagem');
const detail = typeof data.detail === 'string' ? data.detail : 'Erro ao enviar mensagem';
throw new Error(detail);
}
const data = await response.json();
// Remove loading
hideLoading();
// Add bot response
addMessage(data.response, 'bot');
// Add sources
if (data.sources && data.sources.length > 0) {
const sourceText = data.sources.map(s => `📖 ${s.metadata.title} (chunk ${s.metadata.chunk_id})`).join('\n');
addMessage(`Fontes: ${sourceText}`, 'bot', true);
const sourceText = data.sources.map((source) => {
const meta = source.metadata || source;
return `📖 ${meta.title || 'Desconhecido'} (trecho ${meta.chunk_id ?? '?'})`;
}).join('\n');
addMessage(`Fontes:\n${sourceText}`, 'bot');
}
} catch (error) {
hideLoading();
addMessage(`❌ Erro: ${error.message}`, 'bot');
addMessage(`Erro: ${error.message}`, 'bot');
} finally {
button.disabled = false;
input.focus();
}
}
function addMessage(content, type, isSource = false) {
function addMessage(content, type) {
const container = document.getElementById('chat-messages');
const messageDiv = document.createElement('div');
messageDiv.className = `message ${type}`;
if (isSource) {
messageDiv.style.fontSize = '0.85rem';
}
messageDiv.innerHTML = `<div class="message-content">${content}</div>`;
const body = document.createElement('div');
body.className = 'message-content';
body.textContent = content;
messageDiv.appendChild(body);
container.appendChild(messageDiv);
container.scrollTop = container.scrollHeight;
}
@@ -103,10 +105,13 @@ async function loadBooks() {
books.forEach(book => {
const bookDiv = document.createElement('div');
bookDiv.className = 'book-item';
bookDiv.innerHTML = `
<div class="book-icon">📖</div>
<div class="book-name">${book.title}</div>
`;
const icon = document.createElement('div');
icon.className = 'book-icon';
icon.textContent = '📖';
const name = document.createElement('div');
name.className = 'book-name';
name.textContent = book.title;
bookDiv.append(icon, name);
booksList.appendChild(bookDiv);
});
@@ -189,17 +194,22 @@ function renderSchedules(schedules) {
schedules.forEach(schedule => {
const div = document.createElement('div');
div.className = 'schedule-item';
div.innerHTML = `
<div class="schedule-info">
<span class="schedule-time">🕐 ${formatDateTime(schedule.time)}</span>
<span class="schedule-book">${schedule.book || 'Todos os livros'}</span>
</div>
<button class="schedule-delete" data-id="${schedule.id}">🗑️</button>
`;
const info = document.createElement('div');
info.className = 'schedule-info';
const time = document.createElement('span');
time.className = 'schedule-time';
time.textContent = formatDateTime(schedule.time);
const book = document.createElement('span');
book.className = 'schedule-book';
book.textContent = schedule.book || 'Todos os livros';
info.append(time, book);
const remove = document.createElement('button');
remove.className = 'schedule-delete';
remove.type = 'button';
remove.textContent = 'Excluir';
remove.addEventListener('click', () => deleteSchedule(schedule.id));
div.append(info, remove);
container.appendChild(div);
// Add delete handler
div.querySelector('.schedule-delete').addEventListener('click', () => deleteSchedule(schedule.id));
});
}
-28
View File
@@ -1,28 +0,0 @@
# SDD ledger — plan: docs/superpowers/plans/2025-11-01-espiritismo-rag-system.md
## Pre-flight Review (from references/preflight.md)
| Check | Status | Notes |
|-------|--------|-------|
| Plan exists and reachable? | ✅ | `docs/superpowers/plans/2025-11-01-espiritismo-rag-system.md` |
| Spec exists and reachable? | ✅ | `docs/superpowers/specs/2025-11-01-espiritismo-rag-system-design.md` |
| Global Constraints present? | ✅ | CPU-only, 8GB RAM, ~5GB storage, Docker, etc. |
| Tasks independent? | ⚠️ | Many tasks touch common code (embedding, PDF processing) - use batching for small same-shape work |
| Worktree isolated? | ✅ | Using existing workspace at `lab/espiritismo` |
**Rulings:**
- Ruling: Use batching for small same-shape work (Tasks 2, 4, 6, 7) — they touch common utilities and can be dispatched together to avoid context pollution. Costs: additional coordination but keeps context clean. Correctness: reduces context churn and improves parallelization.
## Task Progress
| Task | Status | Commits | Review |
|------|--------|---------|--------|
| 1 | ✅ Complete | - | - |
| 2 | ✅ Complete | - | - |
| 3 | - | - | - |
| 4 | - | - | - |
| 5 | - | - | - |
| 6 | - | - | - |
| 7 | - | - | - |
| 8 | - | - | - |
| 9 | - | - | - |