Compare commits
6
Commits
fe06e46cdd
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8eeb6b89f4 | ||
|
|
7af7ba530d | ||
|
|
67739dcf7e | ||
|
|
99f678c9c8 | ||
|
|
517edea633 | ||
|
|
d7734e0bab |
No files matched your search
@@ -0,0 +1,8 @@
|
||||
.git
|
||||
.superpowers
|
||||
PDF
|
||||
**/*.gguf
|
||||
**/__pycache__
|
||||
**/.venv
|
||||
**/.env
|
||||
models
|
||||
+62
@@ -0,0 +1,62 @@
|
||||
# Environment variables
|
||||
.env
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
|
||||
# Virtual environments
|
||||
venv/
|
||||
ENV/
|
||||
env/
|
||||
.venv
|
||||
|
||||
# IDE
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Docker
|
||||
*.pid
|
||||
*.seed
|
||||
*.pid.lock
|
||||
|
||||
# Logs
|
||||
*.log
|
||||
logs/
|
||||
|
||||
# Model downloads
|
||||
models/*.gguf
|
||||
*.gguf
|
||||
|
||||
# Jupyter
|
||||
.ipynb_checkpoints/
|
||||
|
||||
# Temporary files
|
||||
temp
|
||||
*.tmp
|
||||
temp/
|
||||
@@ -15,7 +15,7 @@ Before building, download the Llama 3.2 3B quantized model:
|
||||
```bash
|
||||
# Download model (approximately 2GB)
|
||||
cd backend
|
||||
wget https://huggingface.co/MaziyarPanahi/Llama-3.2-3B-Instruct-GGUF/resolve/main/llama-3.2-3b-instruct-q4_0.gguf
|
||||
wget -O llama-3.2-3b-instruct-q4_0.gguf "https://us.aws.cdn.hf.co/xet-bridge-us/6718a700b21019fade8d68ac/b472da725da9aaf0e050c885680a67f61fceb097eb4277b0cc1e0b94e3559345?user_id=public&xip=4ii5zzvItH8&X-Xet-Cas-Uid=public&response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27llama-3.2-3b-instruct-q4_0.gguf%3B+filename%3D%22llama-3.2-3b-instruct-q4_0.gguf%22%3B&Expires=1793546619&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly91cy5hd3MuY2RuLmhmLmNvL3hldC1icmlkZ2UtdXMvNjcxOGE3MDBiMjEwMTlmYWRlOGQ2OGFjL2I0NzJkYTcyNWRhOWFhZjBlMDUwYzg4NTY4MGE2N2Y2MWZjZWIwOTdlYjQyNzdiMGNjMWUwYjk0ZTM1NTkzNDVcXD91c2VyX2lkPXB1YmxpYyZ4aXA9NGlpNXp6dkl0SDgmWC1YZXQtQ2FzLVVpZD1wdWJsaWMmcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj1hdHRhY2htZW50JTNCK2ZpbGVuYW1lJTJBJTNEVVRGLTglMjclMjdsbGFtYS0zLjItM2ItaW5zdHJ1Y3QtcTRfMC5nZ3VmJTNCK2ZpbGVuYW1lJTNEJTIybGxhbWEtMy4yLTNiLWluc3RydWN0LXE0XzAuZ2d1ZiUyMiUzQiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiRXBvY2hUaW1lIjoxNzkzNTQ2NjE5fX19XX0_&Signature=MEUCIBdL2cMsOqav%7EbG9Q2mearSLUBdBtfCA3sOBE5GdbXUnAiEA7xWOea%7ETy0Jf%7EPgRYyx1S9bxvnSFoTBI%7ERadl-Nmd8E_&Key-Pair-Id=01KXEF4KZ1B6FV465MAWR4M21F&Hash-Algorithm=SHA256"
|
||||
|
||||
# Place in models directory
|
||||
mkdir -p models
|
||||
@@ -79,7 +79,7 @@ cd backend
|
||||
mkdir -p models
|
||||
|
||||
# Baixe o modelo Llama 3.2 3B quantizado (4-bit)
|
||||
wget https://huggingface.co/MaziyarPanahi/Llama-3.2-3B-Instruct-GGUF/resolve/main/llama-3.2-3b-instruct-q4_0.gguf
|
||||
wget -O llama-3.2-3b-instruct-q4_0.gguf "https://us.aws.cdn.hf.co/xet-bridge-us/6718a700b21019fade8d68ac/b472da725da9aaf0e050c885680a67f61fceb097eb4277b0cc1e0b94e3559345?user_id=public&xip=4ii5zzvItH8&X-Xet-Cas-Uid=public&response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27llama-3.2-3b-instruct-q4_0.gguf%3B+filename%3D%22llama-3.2-3b-instruct-q4_0.gguf%22%3B&Expires=1793546619&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly91cy5hd3MuY2RuLmhmLmNvL3hldC1icmlkZ2UtdXMvNjcxOGE3MDBiMjEwMTlmYWRlOGQ2OGFjL2I0NzJkYTcyNWRhOWFhZjBlMDUwYzg4NTY4MGE2N2Y2MWZjZWIwOTdlYjQyNzdiMGNjMWUwYjk0ZTM1NTkzNDVcXD91c2VyX2lkPXB1YmxpYyZ4aXA9NGlpNXp6dkl0SDgmWC1YZXQtQ2FzLVVpZD1wdWJsaWMmcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj1hdHRhY2htZW50JTNCK2ZpbGVuYW1lJTJBJTNEVVRGLTglMjclMjdsbGFtYS0zLjItM2ItaW5zdHJ1Y3QtcTRfMC5nZ3VmJTNCK2ZpbGVuYW1lJTNEJTIybGxhbWEtMy4yLTNiLWluc3RydWN0LXE0XzAuZ2d1ZiUyMiUzQiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiRXBvY2hUaW1lIjoxNzkzNTQ2NjE5fX19XX0_&Signature=MEUCIBdL2cMsOqav%7EbG9Q2mearSLUBdBtfCA3sOBE5GdbXUnAiEA7xWOea%7ETy0Jf%7EPgRYyx1S9bxvnSFoTBI%7ERadl-Nmd8E_&Key-Pair-Id=01KXEF4KZ1B6FV465MAWR4M21F&Hash-Algorithm=SHA256"
|
||||
|
||||
# Valide o tamanho (deve ser ~2GB)
|
||||
ls -lh models/llama-3.2-3b-instruct-q4_0.gguf
|
||||
@@ -97,7 +97,7 @@ cd backend
|
||||
docker-compose run --rm web
|
||||
|
||||
# Na interface web, o GPT4All vai baixar o modelo automaticamente
|
||||
# ou use: curl -O https://gpt4all.io/models/Llama-3-2-3B-Instruct-GGUF-Q4_0.gguf
|
||||
# ou use a mesma URL assinada do CDN (Opção 1)
|
||||
```
|
||||
|
||||
### Opção 3: Script Automático
|
||||
@@ -109,7 +109,7 @@ docker-compose run --rm web
|
||||
cd backend
|
||||
mkdir -p models
|
||||
|
||||
MODEL_URL="https://huggingface.co/MaziyarPanahi/Llama-3.2-3B-Instruct-GGUF/resolve/main/llama-3.2-3b-instruct-q4_0.gguf"
|
||||
MODEL_URL="https://us.aws.cdn.hf.co/xet-bridge-us/6718a700b21019fade8d68ac/b472da725da9aaf0e050c885680a67f61fceb097eb4277b0cc1e0b94e3559345?user_id=public&xip=4ii5zzvItH8&X-Xet-Cas-Uid=public&response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27llama-3.2-3b-instruct-q4_0.gguf%3B+filename%3D%22llama-3.2-3b-instruct-q4_0.gguf%22%3B&Expires=1793546619&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly91cy5hd3MuY2RuLmhmLmNvL3hldC1icmlkZ2UtdXMvNjcxOGE3MDBiMjEwMTlmYWRlOGQ2OGFjL2I0NzJkYTcyNWRhOWFhZjBlMDUwYzg4NTY4MGE2N2Y2MWZjZWIwOTdlYjQyNzdiMGNjMWUwYjk0ZTM1NTkzNDVcXD91c2VyX2lkPXB1YmxpYyZ4aXA9NGlpNXp6dkl0SDgmWC1YZXQtQ2FzLVVpZD1wdWJsaWMmcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj1hdHRhY2htZW50JTNCK2ZpbGVuYW1lJTJBJTNEVVRGLTglMjclMjdsbGFtYS0zLjItM2ItaW5zdHJ1Y3QtcTRfMC5nZ3VmJTNCK2ZpbGVuYW1lJTNEJTIybGxhbWEtMy4yLTNiLWluc3RydWN0LXE0XzAuZ2d1ZiUyMiUzQiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiRXBvY2hUaW1lIjoxNzkzNTQ2NjE5fX19XX0_&Signature=MEUCIBdL2cMsOqav%7EbG9Q2mearSLUBdBtfCA3sOBE5GdbXUnAiEA7xWOea%7ETy0Jf%7EPgRYyx1S9bxvnSFoTBI%7ERadl-Nmd8E_&Key-Pair-Id=01KXEF4KZ1B6FV465MAWR4M21F&Hash-Algorithm=SHA256"
|
||||
|
||||
if [ ! -f "models/llama-3.2-3b-instruct-q4_0.gguf" ]; then
|
||||
echo "Baixando modelo Llama 3.2 3B..."
|
||||
@@ -174,7 +174,7 @@ docker-compose up -d
|
||||
After starting the services, index the 6 existing books:
|
||||
|
||||
```bash
|
||||
docker-compose run --rm web python backend/scripts/index_books.py
|
||||
docker compose run --rm web python scripts/index_books.py
|
||||
```
|
||||
|
||||
This will:
|
||||
@@ -226,9 +226,10 @@ docker-compose logs -f celery-beat
|
||||
|
||||
If the model download fails:
|
||||
```bash
|
||||
# Download manually
|
||||
cd backend/models
|
||||
wget <model-url>
|
||||
# Download manually (mesma URL assinada da seção Download do Modelo)
|
||||
cd backend
|
||||
mkdir -p models
|
||||
wget -O models/llama-3.2-3b-instruct-q4_0.gguf "https://us.aws.cdn.hf.co/xet-bridge-us/6718a700b21019fade8d68ac/b472da725da9aaf0e050c885680a67f61fceb097eb4277b0cc1e0b94e3559345?user_id=public&xip=4ii5zzvItH8&X-Xet-Cas-Uid=public&response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27llama-3.2-3b-instruct-q4_0.gguf%3B+filename%3D%22llama-3.2-3b-instruct-q4_0.gguf%22%3B&Expires=1793546619&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly91cy5hd3MuY2RuLmhmLmNvL3hldC1icmlkZ2UtdXMvNjcxOGE3MDBiMjEwMTlmYWRlOGQ2OGFjL2I0NzJkYTcyNWRhOWFhZjBlMDUwYzg4NTY4MGE2N2Y2MWZjZWIwOTdlYjQyNzdiMGNjMWUwYjk0ZTM1NTkzNDVcXD91c2VyX2lkPXB1YmxpYyZ4aXA9NGlpNXp6dkl0SDgmWC1YZXQtQ2FzLVVpZD1wdWJsaWMmcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj1hdHRhY2htZW50JTNCK2ZpbGVuYW1lJTJBJTNEVVRGLTglMjclMjdsbGFtYS0zLjItM2ItaW5zdHJ1Y3QtcTRfMC5nZ3VmJTNCK2ZpbGVuYW1lJTNEJTIybGxhbWEtMy4yLTNiLWluc3RydWN0LXE0XzAuZ2d1ZiUyMiUzQiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiRXBvY2hUaW1lIjoxNzkzNTQ2NjE5fX19XX0_&Signature=MEUCIBdL2cMsOqav%7EbG9Q2mearSLUBdBtfCA3sOBE5GdbXUnAiEA7xWOea%7ETy0Jf%7EPgRYyx1S9bxvnSFoTBI%7ERadl-Nmd8E_&Key-Pair-Id=01KXEF4KZ1B6FV465MAWR4M21F&Hash-Algorithm=SHA256"
|
||||
```
|
||||
|
||||
### Qdrant Not Starting
|
||||
@@ -442,10 +443,10 @@ ls -lh PDF/
|
||||
|
||||
```bash
|
||||
# Método 1: Via container
|
||||
docker-compose run --rm web python backend/scripts/index_books.py
|
||||
docker compose run --rm web python scripts/index_books.py
|
||||
|
||||
# Método 2: Com limite de logs
|
||||
docker-compose run --rm web python backend/scripts/index_books.py 2>&1 | tee index.log
|
||||
docker compose run --rm web python scripts/index_books.py 2>&1 | tee index.log
|
||||
```
|
||||
|
||||
### 3. Monitorar Progresso
|
||||
@@ -484,7 +485,7 @@ rm -rf backend/index/*
|
||||
rm backend/schedules.json
|
||||
|
||||
# Reindexar
|
||||
docker-compose run --rm web python backend/scripts/index_books.py
|
||||
docker compose run --rm web python scripts/index_books.py
|
||||
```
|
||||
|
||||
## 🎮 Uso da Aplicação
|
||||
@@ -732,7 +733,7 @@ ls -lh backend/models/
|
||||
|
||||
# Baixar novamente
|
||||
cd backend
|
||||
wget https://huggingface.co/MaziyarPanahi/Llama-3.2-3B-Instruct-GGUF/resolve/main/llama-3.2-3b-instruct-q4_0.gguf
|
||||
wget -O llama-3.2-3b-instruct-q4_0.gguf "https://us.aws.cdn.hf.co/xet-bridge-us/6718a700b21019fade8d68ac/b472da725da9aaf0e050c885680a67f61fceb097eb4277b0cc1e0b94e3559345?user_id=public&xip=4ii5zzvItH8&X-Xet-Cas-Uid=public&response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27llama-3.2-3b-instruct-q4_0.gguf%3B+filename%3D%22llama-3.2-3b-instruct-q4_0.gguf%22%3B&Expires=1793546619&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly91cy5hd3MuY2RuLmhmLmNvL3hldC1icmlkZ2UtdXMvNjcxOGE3MDBiMjEwMTlmYWRlOGQ2OGFjL2I0NzJkYTcyNWRhOWFhZjBlMDUwYzg4NTY4MGE2N2Y2MWZjZWIwOTdlYjQyNzdiMGNjMWUwYjk0ZTM1NTkzNDVcXD91c2VyX2lkPXB1YmxpYyZ4aXA9NGlpNXp6dkl0SDgmWC1YZXQtQ2FzLVVpZD1wdWJsaWMmcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj1hdHRhY2htZW50JTNCK2ZpbGVuYW1lJTJBJTNEVVRGLTglMjclMjdsbGFtYS0zLjItM2ItaW5zdHJ1Y3QtcTRfMC5nZ3VmJTNCK2ZpbGVuYW1lJTNEJTIybGxhbWEtMy4yLTNiLWluc3RydWN0LXE0XzAuZ2d1ZiUyMiUzQiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiRXBvY2hUaW1lIjoxNzkzNTQ2NjE5fX19XX0_&Signature=MEUCIBdL2cMsOqav%7EbG9Q2mearSLUBdBtfCA3sOBE5GdbXUnAiEA7xWOea%7ETy0Jf%7EPgRYyx1S9bxvnSFoTBI%7ERadl-Nmd8E_&Key-Pair-Id=01KXEF4KZ1B6FV465MAWR4M21F&Hash-Algorithm=SHA256"
|
||||
|
||||
# Reiniciar containers
|
||||
docker-compose down
|
||||
@@ -749,7 +750,7 @@ ls -lh PDF/
|
||||
chmod -R 755 PDF/
|
||||
|
||||
# Reindexar
|
||||
docker-compose run --rm web python backend/scripts/index_books.py 2>&1 | tee reindex.log
|
||||
docker compose run --rm web python scripts/index_books.py 2>&1 | tee reindex.log
|
||||
```
|
||||
|
||||
### Comandos de Diagnóstico
|
||||
|
||||
@@ -671,7 +671,7 @@ docker-compose build
|
||||
docker-compose up -d
|
||||
|
||||
# Indexar livros
|
||||
docker-compose run --rm web python backend/scripts/index_books.py
|
||||
docker compose run --rm web python scripts/index_books.py
|
||||
```
|
||||
|
||||
### Produção - Single Server
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 lgcosta
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -1,27 +1,5 @@
|
||||
# Sistema RAG Espiritismo
|
||||
|
||||
<<<<<<< HEAD
|
||||
Sistema de consulta assistida por IA para livros de espiritismo em PDF.
|
||||
|
||||
## Stack
|
||||
|
||||
- GPT4All + Llama 3.2 3B (CPU-only)
|
||||
- LangChain para RAG
|
||||
- Qdrant para vectores
|
||||
- FastAPI + HTML/CSS/JS
|
||||
|
||||
## Quick Start
|
||||
|
||||
1. Clone o repositório
|
||||
2. Configure o .env
|
||||
3. Baixe o modelo Llama 3.2 3B quantized
|
||||
4. `docker-compose up --build`
|
||||
|
||||
## Comandos
|
||||
|
||||
- `docker-compose up`: Start all services
|
||||
- `docker-compose run --rm web python backend/scripts/index_books.py`: Indexar livros
|
||||
=======
|
||||

|
||||

|
||||

|
||||
@@ -129,7 +107,8 @@ espiritismo-rag-system/
|
||||
├── .env.example # Template de .env
|
||||
├── BUILD.md # Instruções de build
|
||||
├── README.md # Este arquivo
|
||||
└── IMPLEMENTATION_COMPLETE.md # Resumo da implementação
|
||||
├── IMPLEMENTATION_COMPLETE.md # Resumo da implementação
|
||||
└── LICENSE # MIT License
|
||||
```
|
||||
|
||||
## 🔧 Pré-requisitos
|
||||
@@ -155,7 +134,7 @@ cd backend
|
||||
mkdir -p models
|
||||
|
||||
# Baixe o modelo Llama 3.2 3B quantizado
|
||||
wget https://huggingface.co/MaziyarPanahi/Llama-3.2-3B-Instruct-GGUF/resolve/main/llama-3.2-3b-instruct-q4_0.gguf
|
||||
wget -O llama-3.2-3b-instruct-q4_0.gguf "https://us.aws.cdn.hf.co/xet-bridge-us/6718a700b21019fade8d68ac/b472da725da9aaf0e050c885680a67f61fceb097eb4277b0cc1e0b94e3559345?user_id=public&xip=4ii5zzvItH8&X-Xet-Cas-Uid=public&response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27llama-3.2-3b-instruct-q4_0.gguf%3B+filename%3D%22llama-3.2-3b-instruct-q4_0.gguf%22%3B&Expires=1793546619&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly91cy5hd3MuY2RuLmhmLmNvL3hldC1icmlkZ2UtdXMvNjcxOGE3MDBiMjEwMTlmYWRlOGQ2OGFjL2I0NzJkYTcyNWRhOWFhZjBlMDUwYzg4NTY4MGE2N2Y2MWZjZWIwOTdlYjQyNzdiMGNjMWUwYjk0ZTM1NTkzNDVcXD91c2VyX2lkPXB1YmxpYyZ4aXA9NGlpNXp6dkl0SDgmWC1YZXQtQ2FzLVVpZD1wdWJsaWMmcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj1hdHRhY2htZW50JTNCK2ZpbGVuYW1lJTJBJTNEVVRGLTglMjclMjdsbGFtYS0zLjItM2ItaW5zdHJ1Y3QtcTRfMC5nZ3VmJTNCK2ZpbGVuYW1lJTNEJTIybGxhbWEtMy4yLTNiLWluc3RydWN0LXE0XzAuZ2d1ZiUyMiUzQiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiRXBvY2hUaW1lIjoxNzkzNTQ2NjE5fX19XX0_&Signature=MEUCIBdL2cMsOqav%7EbG9Q2mearSLUBdBtfCA3sOBE5GdbXUnAiEA7xWOea%7ETy0Jf%7EPgRYyx1S9bxvnSFoTBI%7ERadl-Nmd8E_&Key-Pair-Id=01KXEF4KZ1B6FV465MAWR4M21F&Hash-Algorithm=SHA256"
|
||||
mv llama-3.2-3b-instruct-q4_0.gguf models/
|
||||
|
||||
# Ou use o GPT4All para baixar automaticamente
|
||||
@@ -206,7 +185,7 @@ docker-compose ps
|
||||
### 5. Indexar os Livros
|
||||
|
||||
```bash
|
||||
docker-compose run --rm web python backend/scripts/index_books.py
|
||||
docker compose run --rm web python scripts/index_books.py
|
||||
```
|
||||
|
||||
Isso irá:
|
||||
@@ -218,8 +197,8 @@ Isso irá:
|
||||
### 6. Acessar os Serviços
|
||||
|
||||
- **Web Interface**: http://localhost:8000
|
||||
- **API Docs**: http://localhost:8000/docs
|
||||
- **Qdrant Dashboard**: http://localhost:6333/dashboard
|
||||
- **Redis**: localhost:6379
|
||||
|
||||
## 📖 Uso
|
||||
|
||||
@@ -298,63 +277,15 @@ docker-compose logs -f celery-beat
|
||||
docker-compose logs -f qdrant
|
||||
```
|
||||
|
||||
### Ver Status dos Containers
|
||||
|
||||
```bash
|
||||
docker-compose ps
|
||||
```
|
||||
|
||||
## 🛠️ Deploy em Produção
|
||||
|
||||
### Requisitos para Produção
|
||||
|
||||
- Servidor dedicado (recomendado 8GB+ RAM)
|
||||
- Docker instalado
|
||||
- Modelo Llama 3.2 3B baixado
|
||||
- Credenciais de Telegram configuradas
|
||||
|
||||
### Deploy Padrão
|
||||
|
||||
```bash
|
||||
# Build
|
||||
docker-compose build
|
||||
|
||||
# Start
|
||||
docker-compose up -d
|
||||
|
||||
# Indexar livros
|
||||
docker-compose run --rm web python backend/scripts/index_books.py
|
||||
```
|
||||
|
||||
### Configuração de Ambiente de Produção
|
||||
|
||||
Configure `.env` para produção:
|
||||
|
||||
```env
|
||||
TELEGRAM_BOT_TOKEN=sua_token_producao
|
||||
TELEGRAM_CHAT_ID=sua_chat_id
|
||||
CELERY_BROKER_URL=redis://redis:6379/0
|
||||
CELERY_RESULT_BACKEND=redis://redis:6379/0
|
||||
```
|
||||
|
||||
### Escalabilidade
|
||||
|
||||
Para escalar, use:
|
||||
- Proxy reverso (Nginx)
|
||||
- Load balancer
|
||||
- Mais instâncias do backend
|
||||
- Cluster do Qdrant
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
## 🛠️ Troubleshooting
|
||||
|
||||
### Problemas de Build
|
||||
|
||||
```bash
|
||||
# Limpar cache e rebuild
|
||||
docker-compose build --no-cache
|
||||
|
||||
# Ver logs do build
|
||||
docker-compose logs web
|
||||
docker-compose down
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
### Qdrant Não Inicia
|
||||
@@ -369,20 +300,14 @@ docker-compose up -d qdrant
|
||||
|
||||
### Out of Memory
|
||||
|
||||
Reduza os workers:
|
||||
```bash
|
||||
# Edit docker-compose.yml
|
||||
# Reduzir workers do Celery
|
||||
# Edite docker-compose.yml
|
||||
celery:
|
||||
command: celery -A app.tasks.celery_tasks worker --loglevel=info --concurrency=2
|
||||
```
|
||||
|
||||
### Erros de Import
|
||||
|
||||
```bash
|
||||
# Rebuild e clean
|
||||
docker-compose down
|
||||
docker-compose build --no-cache
|
||||
docker-compose up -d
|
||||
# Reiniciar
|
||||
docker-compose restart celery
|
||||
```
|
||||
|
||||
## 📈 Métricas
|
||||
@@ -401,14 +326,6 @@ docker-compose up -d
|
||||
- **Qdrant index**: ~1GB
|
||||
- **Livros PDF**: ~1GB
|
||||
|
||||
## 📝 Licença
|
||||
|
||||
MIT License - Veja arquivo LICENSE para detalhes.
|
||||
|
||||
## 👥 Autores
|
||||
|
||||
Desenvolvido com ❤️ para a comunidade de espiritismo.
|
||||
|
||||
## 🤝 Contribuindo
|
||||
|
||||
Contribuições são bem-vindas! Sinta-se livre para:
|
||||
@@ -417,13 +334,12 @@ Contribuições são bem-vindas! Sinta-se livre para:
|
||||
- Enviar pull requests
|
||||
- Adicionar novos livros
|
||||
|
||||
## 📄 Documentação Completa
|
||||
## 📄 Documentação
|
||||
|
||||
Para documentação detalhada da implementação, veja:
|
||||
|
||||
- **Design Document**: `docs/superpowers/specs/2025-11-01-espiritismo-rag-system-design.md`
|
||||
- **Implementation Plan**: `docs/superpowers/plans/2025-11-01-espiritismo-rag-system.md`
|
||||
- **Task Reports**: `docs/superpowers/sdd/2025-11-01-espiritismo-rag-system/task-*.md`
|
||||
- **[BUILD.md](BUILD.md)** - Instruções detalhadas de build e deploy
|
||||
- **[IMPLEMENTATION_COMPLETE.md](IMPLEMENTATION_COMPLETE.md)** - Relatório completo da implementação
|
||||
- **[Design Document](docs/superpowers/specs/2025-11-01-espiritismo-rag-system-design.md)** - Design técnico detalhado
|
||||
- **[Implementation Plan](docs/superpowers/plans/2025-11-01-espiritismo-rag-system.md)** - Plano de implementação
|
||||
|
||||
## 🔗 Links Úteis
|
||||
|
||||
@@ -431,11 +347,13 @@ Para documentação detalhada da implementação, veja:
|
||||
- [GPT4All Documentation](https://gpt4all.io/)
|
||||
- [LangChain Documentation](https://python.langchain.com/)
|
||||
- [Qdrant Documentation](https://qdrant.tech/documentation/)
|
||||
- [Llama 3.2 Model](https://huggingface.co/meta-llama/Llama-3.2)
|
||||
|
||||
## 📄 Licença
|
||||
|
||||
MIT License - Veja o arquivo [LICENSE](LICENSE) para detalhes.
|
||||
|
||||
---
|
||||
|
||||
**Versão**: 1.0.0
|
||||
**Status**: Production Ready ✅
|
||||
**Última Atualização**: 2026-11-01
|
||||
>>>>>>> 50ae4db41723d2c6ea63799065316f1ae27bfb03
|
||||
**Status**: ✅ Production Ready
|
||||
**Data**: 2026-11-01
|
||||
+10
-15
@@ -2,32 +2,27 @@ FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Create directories
|
||||
RUN mkdir -p /app/uploads /app/index /app/models
|
||||
RUN mkdir -p /app/uploads /app/index /app/models /app/books
|
||||
|
||||
# Copy requirements
|
||||
COPY requirements.txt .
|
||||
|
||||
# Install Python dependencies
|
||||
COPY backend/requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
# Set environment variables
|
||||
ENV GPT4ALL_MODEL=llama-3.2-3b-instruct-q4_0.gguf
|
||||
ENV PYTHONPATH=/app
|
||||
ENV GPT4ALL_MODEL=/app/models/llama-3.2-3b-instruct-q4_0.gguf
|
||||
ENV UPLOAD_DIR=/app/uploads
|
||||
ENV BOOKS_DIR=/app/books
|
||||
ENV INDEX_DIR=/app/index
|
||||
ENV SCHEDULES_PATH=/app/schedules.json
|
||||
|
||||
# Copy application
|
||||
COPY app/ ./app/
|
||||
COPY frontend/ ./frontend/
|
||||
COPY backend/app ./app
|
||||
COPY backend/templates ./templates
|
||||
COPY backend/scripts ./scripts
|
||||
COPY frontend ./frontend
|
||||
|
||||
# Expose ports
|
||||
EXPOSE 8000
|
||||
|
||||
# Run the application
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
+128
-105
@@ -1,24 +1,39 @@
|
||||
from fastapi import FastAPI, File, UploadFile, HTTPException
|
||||
from fastapi.responses import HTMLResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from app.services.rag_service import RAGService
|
||||
from app.services.bot_service import BotService
|
||||
from app.utils.schedule_manager import ScheduleManager
|
||||
from app.models import ChatRequest, ChatResponse, BookInfo, UploadResponse, ScheduleInfo, ScheduleRequest
|
||||
import os
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import uuid
|
||||
from typing import List
|
||||
|
||||
# Initialize services
|
||||
qdrant_service = QdrantService(os.getenv("QDRANT_URL", "http://qdrant:6333"))
|
||||
rag_service = RAGService(qdrant_service, model_name=os.getenv("GPT4ALL_MODEL", "llama-3.2-3b-instruct-q4_0.gguf"))
|
||||
from fastapi import FastAPI, File, HTTPException, UploadFile
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.responses import HTMLResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
|
||||
from app.models import (
|
||||
BookInfo,
|
||||
ChatRequest,
|
||||
ChatResponse,
|
||||
ScheduleInfo,
|
||||
ScheduleRequest,
|
||||
UploadResponse,
|
||||
)
|
||||
from app.services.bot_service import BotService
|
||||
from app.services.rag_service import QdrantService, RAGService
|
||||
from app.utils.schedule_manager import ScheduleManager
|
||||
|
||||
BASE_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
|
||||
SCHEDULES_PATH = os.getenv("SCHEDULES_PATH", "/app/schedules.json")
|
||||
UPLOAD_DIR = os.getenv("UPLOAD_DIR", "/app/uploads")
|
||||
QDRANT_URL = os.getenv("QDRANT_URL", "http://qdrant:6333")
|
||||
MODEL_PATH = os.getenv("GPT4ALL_MODEL", "/app/models/llama-3.2-3b-instruct-q4_0.gguf")
|
||||
|
||||
qdrant_service: QdrantService | None = None
|
||||
rag_service: RAGService | None = None
|
||||
bot_service = BotService(os.getenv("TELEGRAM_BOT_TOKEN", ""), os.getenv("TELEGRAM_CHAT_ID", ""))
|
||||
schedule_manager = ScheduleManager(os.getenv("SCHEDULES_PATH", "/app/schedules.json"))
|
||||
schedule_manager = ScheduleManager(SCHEDULES_PATH)
|
||||
|
||||
app = FastAPI(title="Espiritismo RAG API")
|
||||
|
||||
# CORS middleware
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
@@ -27,128 +42,136 @@ app.add_middleware(
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
# Mount static files
|
||||
app.mount("/static", StaticFiles(directory="../frontend/static"), name="static")
|
||||
app.mount(
|
||||
"/static",
|
||||
StaticFiles(directory=os.path.join(BASE_DIR, "frontend", "static")),
|
||||
name="static",
|
||||
)
|
||||
|
||||
|
||||
def get_qdrant() -> QdrantService:
|
||||
global qdrant_service
|
||||
if qdrant_service is None:
|
||||
qdrant_service = QdrantService(QDRANT_URL)
|
||||
return qdrant_service
|
||||
|
||||
|
||||
def get_rag() -> RAGService:
|
||||
global rag_service
|
||||
if rag_service is None:
|
||||
rag_service = RAGService(get_qdrant(), model_name=MODEL_PATH)
|
||||
return rag_service
|
||||
|
||||
|
||||
# Load existing schedules from file
|
||||
def load_schedules():
|
||||
if os.path.exists("/app/schedules.json"):
|
||||
with open("/app/schedules.json", "r") as f:
|
||||
return json.load(f)
|
||||
if os.path.isfile(SCHEDULES_PATH):
|
||||
with open(SCHEDULES_PATH, "r", encoding="utf-8") as handle:
|
||||
data = json.load(handle)
|
||||
return data if isinstance(data, list) else []
|
||||
return []
|
||||
|
||||
|
||||
def index_pdf(file_path: str, filename: str):
|
||||
from app.services.pdf_processor import PDFProcessor
|
||||
from app.utils.embeddings import EmbeddingService
|
||||
|
||||
title = os.path.splitext(os.path.basename(filename))[0].replace("-", " ").replace("_", " ")
|
||||
processor = PDFProcessor(EmbeddingService())
|
||||
chunks = processor.embed_chunks(
|
||||
processor.load_and_split(
|
||||
file_path,
|
||||
{"title": title, "author": "Desconhecido", "type": "book"},
|
||||
)
|
||||
)
|
||||
for chunk in chunks:
|
||||
chunk["id"] = str(uuid.uuid4())
|
||||
get_qdrant().add_chunks(chunks)
|
||||
|
||||
|
||||
def save_schedules(schedules: list):
|
||||
parent = os.path.dirname(SCHEDULES_PATH)
|
||||
if parent:
|
||||
os.makedirs(parent, exist_ok=True)
|
||||
with open(SCHEDULES_PATH, "w", encoding="utf-8") as handle:
|
||||
json.dump(schedules, handle, indent=2)
|
||||
handle.write("\n")
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health():
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@app.get("/", response_class=HTMLResponse)
|
||||
async def index():
|
||||
"""Serve the index page."""
|
||||
with open("../backend/templates/index.html", "r") as f:
|
||||
return f.read()
|
||||
template = os.path.join(BASE_DIR, "templates", "index.html")
|
||||
with open(template, "r", encoding="utf-8") as handle:
|
||||
return handle.read()
|
||||
|
||||
|
||||
@app.get("/api/books", response_model=List[BookInfo])
|
||||
async def get_books():
|
||||
"""Get list of books in the collection."""
|
||||
books = qdrant_service.get_books()
|
||||
books = get_qdrant().get_books()
|
||||
return [{"title": book, "source": book} for book in books]
|
||||
|
||||
|
||||
@app.post("/api/chat", response_model=ChatResponse)
|
||||
async def chat(request: ChatRequest):
|
||||
"""Query the RAG system."""
|
||||
def chat(request: ChatRequest):
|
||||
try:
|
||||
result = rag_service.query(request.message, request.top_k)
|
||||
return result
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
return get_rag().query(request.message, request.top_k)
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@app.post("/api/upload", response_model=UploadResponse)
|
||||
async def upload_pdf(file: UploadFile = File(...)):
|
||||
"""Upload a new PDF file."""
|
||||
upload_dir = "/app/uploads"
|
||||
|
||||
if not file.filename.endswith('.pdf'):
|
||||
if not file.filename or not file.filename.endswith(".pdf"):
|
||||
raise HTTPException(status_code=400, detail="Only PDF files are allowed")
|
||||
|
||||
file_path = os.path.join(upload_dir, file.filename)
|
||||
|
||||
os.makedirs(UPLOAD_DIR, exist_ok=True)
|
||||
file_path = os.path.join(UPLOAD_DIR, os.path.basename(file.filename))
|
||||
try:
|
||||
with open(file_path, "wb") as f:
|
||||
content = await file.read()
|
||||
f.write(content)
|
||||
content = await file.read()
|
||||
with open(file_path, "wb") as handle:
|
||||
handle.write(content)
|
||||
await asyncio.to_thread(index_pdf, file_path, file.filename)
|
||||
return UploadResponse(message="File uploaded and indexed", filename=file.filename)
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||
|
||||
return UploadResponse(message="File uploaded successfully", filename=file.filename)
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
@app.get("/api/schedules", response_model=List[ScheduleInfo])
|
||||
async def get_schedules():
|
||||
"""Get all schedules."""
|
||||
schedules = load_schedules()
|
||||
return [ScheduleInfo(**s) for s in schedules]
|
||||
return [ScheduleInfo(**item) for item in load_schedules()]
|
||||
|
||||
|
||||
@app.post("/api/schedules", response_model=ScheduleInfo)
|
||||
async def create_schedule(request: ScheduleRequest):
|
||||
"""Create a new schedule."""
|
||||
schedules = load_schedules()
|
||||
new_schedule = {
|
||||
"id": len(schedules) + 1,
|
||||
"time": request.time,
|
||||
"book": request.book,
|
||||
"enabled": request.enabled,
|
||||
}
|
||||
schedules.append(new_schedule)
|
||||
|
||||
with open("/app/schedules.json", "w") as f:
|
||||
json.dump(schedules, f, indent=2)
|
||||
|
||||
# Start celery beat if needed
|
||||
try:
|
||||
bot_service.add_schedule(new_schedule)
|
||||
except Exception as e:
|
||||
print(f"Warning: Could not add schedule to bot: {e}")
|
||||
|
||||
new_schedule = schedule_manager.add_schedule(request.model_dump())
|
||||
return ScheduleInfo(**new_schedule)
|
||||
|
||||
|
||||
@app.put("/api/schedules/{schedule_id}", response_model=ScheduleInfo)
|
||||
async def update_schedule(schedule_id: int, request: ScheduleRequest):
|
||||
"""Update an existing schedule."""
|
||||
schedules = load_schedules()
|
||||
updated = schedule_manager.update_schedule(schedule_id, request.model_dump())
|
||||
if updated is None:
|
||||
raise HTTPException(status_code=404, detail="Schedule not found")
|
||||
save_schedules(schedule_manager.list_schedules())
|
||||
try:
|
||||
bot_service.update_schedule(updated)
|
||||
except Exception as exc:
|
||||
print(f"Warning: Could not update schedule in bot: {exc}")
|
||||
return ScheduleInfo(**updated)
|
||||
|
||||
for i, s in enumerate(schedules):
|
||||
if s["id"] == schedule_id:
|
||||
schedules[i] = {
|
||||
"id": schedule_id,
|
||||
"time": request.time,
|
||||
"book": request.book,
|
||||
"enabled": request.enabled,
|
||||
}
|
||||
|
||||
with open("/app/schedules.json", "w") as f:
|
||||
json.dump(schedules, f, indent=2)
|
||||
|
||||
try:
|
||||
bot_service.update_schedule(schedules[i])
|
||||
except Exception as e:
|
||||
print(f"Warning: Could not update schedule in bot: {e}")
|
||||
|
||||
return ScheduleInfo(**schedules[i])
|
||||
|
||||
raise HTTPException(status_code=404, detail="Schedule not found")
|
||||
|
||||
@app.delete("/api/schedules/{schedule_id}")
|
||||
async def delete_schedule(schedule_id: int):
|
||||
"""Delete a schedule."""
|
||||
schedules = load_schedules()
|
||||
|
||||
for i, s in enumerate(schedules):
|
||||
if s["id"] == schedule_id:
|
||||
deleted = schedules.pop(i)
|
||||
|
||||
with open("/app/schedules.json", "w") as f:
|
||||
json.dump(schedules, f, indent=2)
|
||||
|
||||
try:
|
||||
bot_service.remove_schedule(deleted)
|
||||
except Exception as e:
|
||||
print(f"Warning: Could not remove schedule from bot: {e}")
|
||||
|
||||
return {"message": "Schedule deleted"}
|
||||
|
||||
raise HTTPException(status_code=404, detail="Schedule not found")
|
||||
current = schedule_manager.get_schedule(schedule_id)
|
||||
if current is None or not schedule_manager.delete_schedule(schedule_id):
|
||||
raise HTTPException(status_code=404, detail="Schedule not found")
|
||||
save_schedules(schedule_manager.list_schedules())
|
||||
try:
|
||||
bot_service.remove_schedule(current)
|
||||
except Exception as exc:
|
||||
print(f"Warning: Could not remove schedule from bot: {exc}")
|
||||
return {"message": "Schedule deleted"}
|
||||
@@ -1,31 +0,0 @@
|
||||
from pydantic import BaseModel
|
||||
from typing import Optional, List, Dict
|
||||
|
||||
class ChatRequest(BaseModel):
|
||||
message: str
|
||||
top_k: int = 3
|
||||
|
||||
class ChatResponse(BaseModel):
|
||||
query: str
|
||||
response: str
|
||||
sources: List[Dict]
|
||||
|
||||
class BookInfo(BaseModel):
|
||||
title: str
|
||||
source: str
|
||||
|
||||
class UploadResponse(BaseModel):
|
||||
message: str
|
||||
filename: str
|
||||
|
||||
class ScheduleInfo(BaseModel):
|
||||
id: int
|
||||
time: str
|
||||
book: Optional[str]
|
||||
enabled: bool
|
||||
next_run: Optional[str] = None
|
||||
|
||||
class ScheduleRequest(BaseModel):
|
||||
time: str
|
||||
book: Optional[str] = None
|
||||
enabled: bool = True
|
||||
@@ -1 +1,38 @@
|
||||
# Pydantic models package
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class ChatRequest(BaseModel):
|
||||
message: str
|
||||
top_k: int = 3
|
||||
|
||||
|
||||
class ChatResponse(BaseModel):
|
||||
query: str
|
||||
response: str
|
||||
sources: List[Dict]
|
||||
|
||||
|
||||
class BookInfo(BaseModel):
|
||||
title: str
|
||||
source: str
|
||||
|
||||
|
||||
class UploadResponse(BaseModel):
|
||||
message: str
|
||||
filename: str
|
||||
|
||||
|
||||
class ScheduleInfo(BaseModel):
|
||||
id: int
|
||||
time: str
|
||||
book: Optional[str] = None
|
||||
enabled: bool
|
||||
next_run: Optional[str] = None
|
||||
|
||||
|
||||
class ScheduleRequest(BaseModel):
|
||||
time: str
|
||||
book: Optional[str] = None
|
||||
enabled: bool = True
|
||||
@@ -1,25 +1,16 @@
|
||||
import telebot
|
||||
from telebot import types
|
||||
import os
|
||||
import json
|
||||
from datetime import datetime, timedelta
|
||||
from typing import List, Dict, Optional
|
||||
from celery import Celery
|
||||
import os
|
||||
import random
|
||||
from typing import Dict, List
|
||||
|
||||
celery = Celery(
|
||||
'tasks',
|
||||
broker=os.getenv("CELERY_BROKER_URL", "redis://localhost:6379/0"),
|
||||
backend=os.getenv("CELERY_RESULT_BACKEND", "redis://localhost:6379/0"),
|
||||
)
|
||||
import telebot
|
||||
|
||||
class BotService:
|
||||
def __init__(self, token: str, chat_id: str):
|
||||
self.bot = telebot.TeleBot(token)
|
||||
self.chat_id = chat_id
|
||||
|
||||
# Register handlers
|
||||
self._register_handlers()
|
||||
self.bot = telebot.TeleBot(token) if token else None
|
||||
if self.bot is not None:
|
||||
self._register_handlers()
|
||||
|
||||
# Store pending schedules
|
||||
self.pending_schedules: List[Dict] = []
|
||||
@@ -81,11 +72,13 @@ Direcione uma pergunta sobre espiritismo para eu responder!
|
||||
def _query_rag_service(self, question: str, chat_id: int):
|
||||
"""Query RAG service and send response."""
|
||||
try:
|
||||
from app.services.rag_service import RAGService
|
||||
from app.services.qdrant_service import QdrantService
|
||||
from app.services.rag_service import QdrantService, RAGService
|
||||
|
||||
qdrant = QdrantService(os.getenv("QDRANT_URL", "http://qdrant:6333"))
|
||||
rag = RAGService(qdrant, model_name=os.getenv("GPT4ALL_MODEL", "llama-3.2-3b-instruct-q4_0.gguf"))
|
||||
rag = RAGService(
|
||||
qdrant,
|
||||
model_name=os.getenv("GPT4ALL_MODEL", "/app/models/llama-3.2-3b-instruct-q4_0.gguf"),
|
||||
)
|
||||
|
||||
result = rag.query(question, top_k=3)
|
||||
|
||||
@@ -160,14 +153,23 @@ Direcione uma pergunta sobre espiritismo para eu responder!
|
||||
|
||||
def start_polling(self):
|
||||
"""Start the bot polling."""
|
||||
if self.bot is None:
|
||||
print("TELEGRAM_BOT_TOKEN ausente; polling não iniciado")
|
||||
return
|
||||
print("Starting Telegram bot polling...")
|
||||
self.bot.infinity_polling(timeout=10, long_polling_timeout=5)
|
||||
|
||||
@celery.task
|
||||
def send_scheduled_message(self, schedule: Dict):
|
||||
"""Send scheduled message to Telegram."""
|
||||
try:
|
||||
response = rag.query(schedule.get('book', ''), top_k=3)
|
||||
from app.services.rag_service import QdrantService, RAGService
|
||||
|
||||
qdrant = QdrantService(os.getenv("QDRANT_URL", "http://qdrant:6333"))
|
||||
rag = RAGService(
|
||||
qdrant,
|
||||
model_name=os.getenv("GPT4ALL_MODEL", "/app/models/llama-3.2-3b-instruct-q4_0.gguf"),
|
||||
)
|
||||
response = rag.query(schedule.get('book') or 'espiritismo', top_k=3)
|
||||
message = f"""
|
||||
⏰ Mensagem Agendada - {schedule.get('book', 'Todos os livros')}
|
||||
|
||||
@@ -176,11 +178,13 @@ Direcione uma pergunta sobre espiritismo para eu responder!
|
||||
📚 Fontes:
|
||||
{self._format_sources(response['sources'])}
|
||||
"""
|
||||
if self.bot is None:
|
||||
print("TELEGRAM_BOT_TOKEN ausente; mensagem agendada não enviada")
|
||||
return
|
||||
self.bot.send_message(self.chat_id, message)
|
||||
except Exception as e:
|
||||
print(f"Error sending scheduled message: {e}")
|
||||
|
||||
@celery.task
|
||||
def generate_daily_content(self):
|
||||
"""Generate daily content for all books."""
|
||||
books = [
|
||||
@@ -204,4 +208,7 @@ Direcione uma pergunta sobre espiritismo para eu responder!
|
||||
Este conteúdo foi gerado automaticamente pelo Sistema RAG Espiritismo.
|
||||
"""
|
||||
|
||||
if self.bot is None:
|
||||
print("TELEGRAM_BOT_TOKEN ausente; conteúdo diário não enviado")
|
||||
return
|
||||
self.bot.send_message(self.chat_id, message)
|
||||
@@ -2,7 +2,7 @@ from langchain_community.document_loaders import PyPDFLoader
|
||||
from langchain_text_splitters import RecursiveCharacterTextSplitter
|
||||
import os
|
||||
from typing import List, Dict
|
||||
from .embeddings import EmbeddingService
|
||||
from app.utils.embeddings import EmbeddingService
|
||||
|
||||
class PDFProcessor:
|
||||
def __init__(self, embedding_service: EmbeddingService):
|
||||
@@ -22,12 +22,15 @@ class PDFProcessor:
|
||||
|
||||
# Convert to dictionaries with metadata
|
||||
chunk_data = []
|
||||
for i, doc in enumerate(chunks):
|
||||
for doc in chunks:
|
||||
content = " ".join(doc.page_content.split())
|
||||
if len(content) < 80:
|
||||
continue
|
||||
chunk_data.append({
|
||||
"content": doc.page_content,
|
||||
"content": content,
|
||||
"metadata": {
|
||||
**metadata,
|
||||
"chunk_id": i,
|
||||
"chunk_id": len(chunk_data),
|
||||
"source": file_path,
|
||||
}
|
||||
})
|
||||
@@ -2,8 +2,10 @@ from qdrant_client import QdrantClient
|
||||
from qdrant_client.models import Distance, VectorParams, PointStruct, Filter, FieldCondition, MatchValue
|
||||
import numpy as np
|
||||
from typing import List, Dict, Optional
|
||||
import os
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
import uuid
|
||||
|
||||
class QdrantService:
|
||||
def __init__(self, url: str, collection_name: str = "espiritismo_books"):
|
||||
@@ -15,7 +17,17 @@ class QdrantService:
|
||||
|
||||
def _ensure_collection(self):
|
||||
"""Create collection if it doesn't exist."""
|
||||
collections = self.client.get_collections().collections
|
||||
last_error = None
|
||||
for attempt in range(15):
|
||||
try:
|
||||
collections = self.client.get_collections().collections
|
||||
break
|
||||
except Exception as exc:
|
||||
last_error = exc
|
||||
print(f"Qdrant indisponível ({attempt + 1}/15): {exc}")
|
||||
time.sleep(2)
|
||||
else:
|
||||
raise RuntimeError(f"Não foi possível conectar ao Qdrant: {last_error}")
|
||||
|
||||
collection_exists = any(
|
||||
c.name == self.collection_name for c in collections
|
||||
@@ -35,28 +47,28 @@ class QdrantService:
|
||||
|
||||
def add_chunks(self, chunks: List[Dict]):
|
||||
"""Add chunks to the collection."""
|
||||
points = []
|
||||
|
||||
for i, chunk in enumerate(chunks):
|
||||
point = PointStruct(
|
||||
id=i,
|
||||
points = [
|
||||
PointStruct(
|
||||
id=str(chunk.get("id") or uuid.uuid4()),
|
||||
vector=chunk["embedding"],
|
||||
payload={
|
||||
"content": chunk["content"],
|
||||
"metadata": chunk["metadata"],
|
||||
}
|
||||
},
|
||||
)
|
||||
points.append(point)
|
||||
for chunk in chunks
|
||||
]
|
||||
|
||||
for start in range(0, len(points), 128):
|
||||
batch = points[start:start + 128]
|
||||
self.client.upsert(collection_name=self.collection_name, points=batch)
|
||||
if points:
|
||||
self.client.upsert(
|
||||
collection_name=self.collection_name,
|
||||
points=points,
|
||||
)
|
||||
print(f"Added {len(points)} chunks to Qdrant")
|
||||
|
||||
def search(self, query_embedding: List[float], top_k: int = 3) -> List[Dict]:
|
||||
def search(self, query_embedding, top_k: int = 3) -> List[Dict]:
|
||||
"""Search for similar chunks."""
|
||||
if hasattr(query_embedding, "tolist"):
|
||||
query_embedding = query_embedding.tolist()
|
||||
results = self.client.search(
|
||||
collection_name=self.collection_name,
|
||||
query_vector=query_embedding,
|
||||
@@ -74,17 +86,22 @@ class QdrantService:
|
||||
|
||||
def get_books(self) -> List[str]:
|
||||
"""Get list of books in the collection."""
|
||||
points = self.client.scroll(
|
||||
collection_name=self.collection_name,
|
||||
limit=10000,
|
||||
)[0]
|
||||
|
||||
books = set()
|
||||
for point in points:
|
||||
metadata = point.payload["metadata"]
|
||||
books.add(metadata.get("title", "Unknown"))
|
||||
offset = None
|
||||
while True:
|
||||
points, offset = self.client.scroll(
|
||||
collection_name=self.collection_name,
|
||||
limit=256,
|
||||
offset=offset,
|
||||
with_vectors=False,
|
||||
)
|
||||
for point in points:
|
||||
metadata = (point.payload or {}).get("metadata") or {}
|
||||
books.add(metadata.get("title", "Unknown"))
|
||||
if offset is None:
|
||||
break
|
||||
|
||||
return sorted(list(books))
|
||||
return sorted(books)
|
||||
|
||||
def get_collection_info(self) -> Dict:
|
||||
"""Get information about the collection."""
|
||||
@@ -100,34 +117,44 @@ class RAGService:
|
||||
def __init__(self, qdrant_service: QdrantService, model_name: str = "llama-3.2-3b-instruct-q4_0.gguf"):
|
||||
self.qdrant = qdrant_service
|
||||
self.model_name = model_name
|
||||
self.llm = None
|
||||
|
||||
# Setup GPT4All
|
||||
from gpt4all import GPT4All
|
||||
def _get_llm(self):
|
||||
if self.llm is None:
|
||||
if not os.path.isfile(self.model_name):
|
||||
raise FileNotFoundError(
|
||||
f"Modelo não encontrado em {self.model_name}. "
|
||||
"Baixe llama-3.2-3b-instruct-q4_0.gguf para backend/models/."
|
||||
)
|
||||
from gpt4all import GPT4All
|
||||
|
||||
self.llm = GPT4All(model_name=model_name, device="cpu")
|
||||
model_dir = os.path.dirname(self.model_name) or "."
|
||||
model_file = os.path.basename(self.model_name)
|
||||
self.llm = GPT4All(
|
||||
model_name=model_file,
|
||||
model_path=model_dir,
|
||||
device="cpu",
|
||||
allow_download=False,
|
||||
)
|
||||
return self.llm
|
||||
|
||||
def generate_response(self, query: str, context: List[str], top_k: int = 3) -> str:
|
||||
"""Generate a response based on the query and context."""
|
||||
context_text = "\n\n".join(context)
|
||||
|
||||
system_prompt = f"""Você é um assistente especializado em espiritismo. Use as informações abaixo para responder à pergunta do usuário.
|
||||
system_prompt = f"""Você responde perguntas sobre espiritismo usando apenas o contexto.
|
||||
Escreva uma ou duas frases completas, em português. Não repita a pergunta nem as instruções.
|
||||
Se o contexto não bastar, responda apenas: Não encontrei essa informação nos livros indexados.
|
||||
|
||||
Informações:
|
||||
{context_text}
|
||||
|
||||
Instruções:
|
||||
- Responda de forma clara e concisa
|
||||
- Use o contexto fornecido
|
||||
- Quando citar passagens, mencione o livro e capítulo (se disponível)
|
||||
- Se não souber a resposta, diga que não encontrou na base de dados"""
|
||||
|
||||
response = self.llm.generate(
|
||||
system_prompt,
|
||||
query,
|
||||
max_tokens=500,
|
||||
Contexto:
|
||||
{context_text}"""
|
||||
template = (
|
||||
"<|start_header_id|>user<|end_header_id|>\n\n"
|
||||
"{0}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
|
||||
)
|
||||
|
||||
return response
|
||||
with self._get_llm().chat_session(system_prompt=system_prompt, prompt_template=template):
|
||||
response = self.llm.generate(query, max_tokens=160, temp=0.1)
|
||||
return response.strip()
|
||||
|
||||
def query(self, query: str, top_k: int = 3) -> Dict:
|
||||
"""Query the RAG system."""
|
||||
@@ -149,5 +176,8 @@ Instruções:
|
||||
"query": query,
|
||||
"response": response,
|
||||
"context": context,
|
||||
"sources": [chunk["metadata"] for chunk in chunks],
|
||||
"sources": [
|
||||
{"metadata": chunk["metadata"], "score": chunk["score"]}
|
||||
for chunk in chunks
|
||||
],
|
||||
}
|
||||
@@ -30,23 +30,47 @@ def save_schedules(schedules: List[Dict]):
|
||||
with open("/app/schedules.json", "w") as f:
|
||||
json.dump(schedules, f, indent=2)
|
||||
|
||||
@celery.task
|
||||
def send_scheduled_message(schedule_id: int):
|
||||
"""Send scheduled message to Telegram."""
|
||||
schedules = load_schedules()
|
||||
schedule = next((s for s in schedules if s['id'] == schedule_id), None)
|
||||
def schedule_is_due(schedule: Dict) -> bool:
|
||||
raw = str(schedule.get("time", "")).replace("Z", "+00:00")
|
||||
when = datetime.fromisoformat(raw)
|
||||
if when.tzinfo is not None:
|
||||
when = when.astimezone().replace(tzinfo=None)
|
||||
return datetime.now() >= when
|
||||
|
||||
if not schedule:
|
||||
print(f"Schedule {schedule_id} not found")
|
||||
|
||||
@celery.task
|
||||
def send_scheduled_message(schedule_id: int = None):
|
||||
"""Send enabled schedules whose time has arrived and that were not sent yet."""
|
||||
schedules = load_schedules()
|
||||
due = []
|
||||
for item in schedules:
|
||||
if schedule_id is not None and item.get("id") != schedule_id:
|
||||
continue
|
||||
if not item.get("enabled", True) or item.get("sent"):
|
||||
continue
|
||||
try:
|
||||
ready = schedule_is_due(item)
|
||||
except ValueError:
|
||||
print(f"Invalid schedule time: {item.get('time')}")
|
||||
continue
|
||||
if ready:
|
||||
due.append(item)
|
||||
|
||||
if not due:
|
||||
print("No schedules due")
|
||||
return
|
||||
|
||||
if schedule.get('enabled', True):
|
||||
# Send to bot service
|
||||
changed = False
|
||||
for schedule in due:
|
||||
try:
|
||||
bot_service.send_scheduled_message(schedule)
|
||||
schedule["sent"] = True
|
||||
changed = True
|
||||
print(f"Sent scheduled message: {schedule['time']}")
|
||||
except Exception as e:
|
||||
print(f"Error sending scheduled message: {e}")
|
||||
if changed:
|
||||
save_schedules(schedules)
|
||||
|
||||
@celery.task
|
||||
def generate_daily_content():
|
||||
@@ -1,9 +1,13 @@
|
||||
from sentence_transformers import SentenceTransformer
|
||||
import numpy as np
|
||||
|
||||
_MODELS = {}
|
||||
|
||||
class EmbeddingService:
|
||||
def __init__(self, model_name="all-MiniLM-L6-v2"):
|
||||
self.model = SentenceTransformer(model_name)
|
||||
def __init__(self, model_name="paraphrase-multilingual-MiniLM-L12-v2"):
|
||||
if model_name not in _MODELS:
|
||||
_MODELS[model_name] = SentenceTransformer(model_name)
|
||||
self.model = _MODELS[model_name]
|
||||
self.dimension = 384
|
||||
|
||||
def embed(self, texts: list[str]) -> np.ndarray:
|
||||
@@ -23,6 +23,7 @@ class ScheduleManager:
|
||||
os.makedirs(os.path.dirname(self.schedules_path), exist_ok=True)
|
||||
with open(self.schedules_path, 'w') as f:
|
||||
json.dump(self.schedules, f, indent=2)
|
||||
f.write("\n")
|
||||
|
||||
def add_schedule(self, schedule: Dict) -> Dict:
|
||||
"""Add a new schedule."""
|
||||
Whitespace-only changes.
@@ -2,10 +2,9 @@ fastapi==0.115.0
|
||||
uvicorn[standard]==0.32.0
|
||||
pydantic==2.9.2
|
||||
python-multipart==0.0.9
|
||||
gpt4all==2.4.6
|
||||
gpt4all==2.8.2
|
||||
langchain==0.3.7
|
||||
langchain-community==0.3.5
|
||||
langchain-experimental==0.0.73
|
||||
sentence-transformers==2.7.0
|
||||
pytelegrambotapi==4.26.0
|
||||
celery==5.4.0
|
||||
@@ -14,3 +13,5 @@ python-dotenv==1.0.1
|
||||
python-dateutil==2.9.0.post0
|
||||
aiofiles==24.1.0
|
||||
qdrant-client==1.11.1
|
||||
pypdf==5.1.0
|
||||
langchain-text-splitters==0.3.2
|
||||
@@ -1,51 +1,69 @@
|
||||
from app.services.pdf_processor import PDFProcessor
|
||||
from app.utils.embeddings import EmbeddingService
|
||||
from app.utils.schedule_manager import ScheduleManager
|
||||
import os
|
||||
import json
|
||||
import os
|
||||
import uuid
|
||||
|
||||
from app.services.pdf_processor import PDFProcessor
|
||||
from app.services.rag_service import QdrantService
|
||||
from app.utils.embeddings import EmbeddingService
|
||||
|
||||
|
||||
def book_dirs() -> list[str]:
|
||||
paths = [
|
||||
os.getenv("BOOKS_DIR", "/app/books"),
|
||||
os.getenv("UPLOAD_DIR", "/app/uploads"),
|
||||
]
|
||||
seen = []
|
||||
for path in paths:
|
||||
if path and path not in seen and os.path.isdir(path):
|
||||
seen.append(path)
|
||||
return seen
|
||||
|
||||
|
||||
def main():
|
||||
# Setup services
|
||||
embedding_service = EmbeddingService()
|
||||
pdf_processor = PDFProcessor(embedding_service)
|
||||
qdrant = QdrantService(os.getenv("QDRANT_URL", "http://qdrant:6333"))
|
||||
qdrant.client.delete_collection(qdrant.collection_name)
|
||||
qdrant._ensure_collection()
|
||||
|
||||
# Load books from PDF directory
|
||||
pdf_dir = "/app/uploads"
|
||||
books = os.listdir(pdf_dir)
|
||||
books = [f for f in books if f.endswith('.pdf')]
|
||||
books = []
|
||||
for directory in book_dirs():
|
||||
for name in sorted(os.listdir(directory)):
|
||||
if name.lower().endswith(".pdf"):
|
||||
books.append(os.path.join(directory, name))
|
||||
|
||||
print(f"Found {len(books)} books")
|
||||
os.makedirs(os.getenv("INDEX_DIR", "/app/index"), exist_ok=True)
|
||||
|
||||
# Schedule manager
|
||||
schedule_manager = ScheduleManager("/app/schedules.json")
|
||||
|
||||
# Process each book
|
||||
for book_name in books:
|
||||
book_path = os.path.join(pdf_dir, book_name)
|
||||
for book_path in books:
|
||||
book_name = os.path.basename(book_path)
|
||||
book_metadata = {
|
||||
"title": book_name.replace(".pdf", "").replace("-", " ").replace("_", " "),
|
||||
"author": "Desconhecido",
|
||||
"type": "book",
|
||||
}
|
||||
|
||||
print(f"Processing: {book_name}")
|
||||
|
||||
# Load and split PDF
|
||||
chunks = pdf_processor.load_and_split(book_path, book_metadata)
|
||||
|
||||
# Embed chunks
|
||||
chunks_with_embeddings = pdf_processor.embed_chunks(chunks)
|
||||
for chunk in chunks_with_embeddings:
|
||||
chunk["id"] = str(uuid.uuid4())
|
||||
|
||||
# Save chunks to JSON for reference
|
||||
chunks_file = f"/app/index/{book_name}.json"
|
||||
os.makedirs("/app/index", exist_ok=True)
|
||||
qdrant.add_chunks(chunks_with_embeddings)
|
||||
|
||||
with open(chunks_file, 'w') as f:
|
||||
json.dump(chunks_with_embeddings, f, indent=2)
|
||||
|
||||
print(f"Saved {len(chunks)} chunks to {chunks_file}")
|
||||
chunks_file = os.path.join(os.getenv("INDEX_DIR", "/app/index"), f"{book_name}.json")
|
||||
with open(chunks_file, "w", encoding="utf-8") as handle:
|
||||
json.dump(
|
||||
[
|
||||
{"content": chunk["content"], "metadata": chunk["metadata"]}
|
||||
for chunk in chunks_with_embeddings
|
||||
],
|
||||
handle,
|
||||
ensure_ascii=False,
|
||||
)
|
||||
print(f"Indexed {len(chunks_with_embeddings)} chunks from {book_name}")
|
||||
|
||||
print("Indexing complete!")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -4,6 +4,7 @@
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Sistema RAG Espiritismo</title>
|
||||
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'%3E%3Ctext y='.9em' font-size='90'%3E📚%3C/text%3E%3C/svg%3E">
|
||||
<link rel="stylesheet" href="/static/css/style.css">
|
||||
</head>
|
||||
<body>
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
[]
|
||||
+47
-26
@@ -1,8 +1,6 @@
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
qdrant:
|
||||
image: qdrant/qdrant:latest
|
||||
image: docker.io/qdrant/qdrant:latest
|
||||
container_name: espiritismo-qdrant
|
||||
ports:
|
||||
- "6333:6333"
|
||||
@@ -11,7 +9,7 @@ services:
|
||||
restart: unless-stopped
|
||||
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
image: docker.io/library/redis:7-alpine
|
||||
container_name: espiritismo-redis
|
||||
ports:
|
||||
- "6379:6379"
|
||||
@@ -20,52 +18,74 @@ services:
|
||||
restart: unless-stopped
|
||||
|
||||
web:
|
||||
build: ./backend
|
||||
build:
|
||||
context: .
|
||||
dockerfile: backend/Dockerfile
|
||||
container_name: espiritismo-web
|
||||
ports:
|
||||
- "8000:8000"
|
||||
volumes:
|
||||
- ./PDF:/app/uploads:ro
|
||||
- ./backend/schedules.json:/app/schedules.json
|
||||
- ./PDF:/app/books:ro
|
||||
- uploads_data:/app/uploads
|
||||
- ./data/schedules.json:/app/schedules.json
|
||||
- ./backend/models:/app/models
|
||||
environment:
|
||||
- QDRANT_URL=http://qdrant:6333
|
||||
- CELERY_BROKER_URL=redis://redis:6379/0
|
||||
- CELERY_RESULT_BACKEND=redis://redis:6379/0
|
||||
- TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN}
|
||||
- TELEGRAM_CHAT_ID=${TELEGRAM_CHAT_ID}
|
||||
QDRANT_URL: http://qdrant:6333
|
||||
CELERY_BROKER_URL: redis://redis:6379/0
|
||||
CELERY_RESULT_BACKEND: redis://redis:6379/0
|
||||
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN:-}
|
||||
TELEGRAM_CHAT_ID: ${TELEGRAM_CHAT_ID:-}
|
||||
GPT4ALL_MODEL: /app/models/llama-3.2-3b-instruct-q4_0.gguf
|
||||
BOOKS_DIR: /app/books
|
||||
UPLOAD_DIR: /app/uploads
|
||||
SCHEDULES_PATH: /app/schedules.json
|
||||
TZ: America/Sao_Paulo
|
||||
depends_on:
|
||||
- qdrant
|
||||
- redis
|
||||
restart: unless-stopped
|
||||
|
||||
celery:
|
||||
build: ./backend
|
||||
build:
|
||||
context: .
|
||||
dockerfile: backend/Dockerfile
|
||||
container_name: espiritismo-celery
|
||||
command: celery -A app.tasks.celery_tasks worker --loglevel=info
|
||||
volumes:
|
||||
- ./PDF:/app/uploads:ro
|
||||
- ./backend/schedules.json:/app/schedules.json
|
||||
- ./PDF:/app/books:ro
|
||||
- uploads_data:/app/uploads
|
||||
- ./data/schedules.json:/app/schedules.json
|
||||
- ./backend/models:/app/models
|
||||
environment:
|
||||
- QDRANT_URL=http://qdrant:6333
|
||||
- CELERY_BROKER_URL=redis://redis:6379/0
|
||||
- CELERY_RESULT_BACKEND=redis://redis:6379/0
|
||||
- TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN}
|
||||
- TELEGRAM_CHAT_ID=${TELEGRAM_CHAT_ID}
|
||||
QDRANT_URL: http://qdrant:6333
|
||||
CELERY_BROKER_URL: redis://redis:6379/0
|
||||
CELERY_RESULT_BACKEND: redis://redis:6379/0
|
||||
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN:-}
|
||||
TELEGRAM_CHAT_ID: ${TELEGRAM_CHAT_ID:-}
|
||||
GPT4ALL_MODEL: /app/models/llama-3.2-3b-instruct-q4_0.gguf
|
||||
BOOKS_DIR: /app/books
|
||||
UPLOAD_DIR: /app/uploads
|
||||
SCHEDULES_PATH: /app/schedules.json
|
||||
TZ: America/Sao_Paulo
|
||||
depends_on:
|
||||
- qdrant
|
||||
- redis
|
||||
restart: unless-stopped
|
||||
|
||||
celery-beat:
|
||||
build: ./backend
|
||||
build:
|
||||
context: .
|
||||
dockerfile: backend/Dockerfile
|
||||
command: celery -A app.tasks.celery_tasks beat --loglevel=info
|
||||
volumes:
|
||||
- ./backend/schedules.json:/app/schedules.json
|
||||
- ./data/schedules.json:/app/schedules.json
|
||||
environment:
|
||||
- CELERY_BROKER_URL=redis://redis:6379/0
|
||||
- CELERY_RESULT_BACKEND=redis://redis:6379/0
|
||||
- TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN}
|
||||
- TELEGRAM_CHAT_ID=${TELEGRAM_CHAT_ID}
|
||||
CELERY_BROKER_URL: redis://redis:6379/0
|
||||
CELERY_RESULT_BACKEND: redis://redis:6379/0
|
||||
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN:-}
|
||||
TELEGRAM_CHAT_ID: ${TELEGRAM_CHAT_ID:-}
|
||||
SCHEDULES_PATH: /app/schedules.json
|
||||
TZ: America/Sao_Paulo
|
||||
depends_on:
|
||||
- redis
|
||||
restart: unless-stopped
|
||||
@@ -73,3 +93,4 @@ services:
|
||||
volumes:
|
||||
qdrant_data:
|
||||
redis_data:
|
||||
uploads_data:
|
||||
@@ -85,6 +85,7 @@ header p {
|
||||
|
||||
.message-content {
|
||||
line-height: 1.6;
|
||||
white-space: pre-wrap;
|
||||
}
|
||||
|
||||
.message-source {
|
||||
@@ -170,3 +171,36 @@ th {
|
||||
border-color: #667eea;
|
||||
background: #f9f9f9;
|
||||
}
|
||||
|
||||
.book-item {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
align-items: center;
|
||||
padding: 8px 0;
|
||||
border-bottom: 1px solid #eee;
|
||||
}
|
||||
|
||||
.schedule-item {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
padding: 8px 0;
|
||||
border-bottom: 1px solid #eee;
|
||||
}
|
||||
|
||||
.schedule-info {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 4px;
|
||||
}
|
||||
|
||||
button.schedule-delete {
|
||||
width: auto;
|
||||
padding: 8px 12px;
|
||||
}
|
||||
|
||||
#upload-status {
|
||||
margin-top: 10px;
|
||||
min-height: 1.2em;
|
||||
}
|
||||
@@ -17,9 +17,11 @@ document.addEventListener('DOMContentLoaded', () => {
|
||||
|
||||
async function sendChatMessage() {
|
||||
const input = document.getElementById('chat-input');
|
||||
const button = document.getElementById('send-btn');
|
||||
const message = input.value.trim();
|
||||
|
||||
if (!message) return;
|
||||
if (!message || button.disabled) return;
|
||||
button.disabled = true;
|
||||
|
||||
// Add user message to chat
|
||||
addMessage(message, 'user');
|
||||
@@ -35,39 +37,39 @@ async function sendChatMessage() {
|
||||
body: JSON.stringify({ message, top_k: 3 }),
|
||||
});
|
||||
|
||||
const data = await response.json().catch(() => ({}));
|
||||
if (!response.ok) {
|
||||
throw new Error('Erro ao enviar mensagem');
|
||||
const detail = typeof data.detail === 'string' ? data.detail : 'Erro ao enviar mensagem';
|
||||
throw new Error(detail);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
// Remove loading
|
||||
hideLoading();
|
||||
|
||||
// Add bot response
|
||||
addMessage(data.response, 'bot');
|
||||
|
||||
// Add sources
|
||||
if (data.sources && data.sources.length > 0) {
|
||||
const sourceText = data.sources.map(s => `📖 ${s.metadata.title} (chunk ${s.metadata.chunk_id})`).join('\n');
|
||||
addMessage(`Fontes: ${sourceText}`, 'bot', true);
|
||||
const sourceText = data.sources.map((source) => {
|
||||
const meta = source.metadata || source;
|
||||
return `📖 ${meta.title || 'Desconhecido'} (trecho ${meta.chunk_id ?? '?'})`;
|
||||
}).join('\n');
|
||||
addMessage(`Fontes:\n${sourceText}`, 'bot');
|
||||
}
|
||||
} catch (error) {
|
||||
hideLoading();
|
||||
addMessage(`❌ Erro: ${error.message}`, 'bot');
|
||||
addMessage(`Erro: ${error.message}`, 'bot');
|
||||
} finally {
|
||||
button.disabled = false;
|
||||
input.focus();
|
||||
}
|
||||
}
|
||||
|
||||
function addMessage(content, type, isSource = false) {
|
||||
function addMessage(content, type) {
|
||||
const container = document.getElementById('chat-messages');
|
||||
const messageDiv = document.createElement('div');
|
||||
messageDiv.className = `message ${type}`;
|
||||
|
||||
if (isSource) {
|
||||
messageDiv.style.fontSize = '0.85rem';
|
||||
}
|
||||
|
||||
messageDiv.innerHTML = `<div class="message-content">${content}</div>`;
|
||||
const body = document.createElement('div');
|
||||
body.className = 'message-content';
|
||||
body.textContent = content;
|
||||
messageDiv.appendChild(body);
|
||||
container.appendChild(messageDiv);
|
||||
container.scrollTop = container.scrollHeight;
|
||||
}
|
||||
@@ -103,10 +105,13 @@ async function loadBooks() {
|
||||
books.forEach(book => {
|
||||
const bookDiv = document.createElement('div');
|
||||
bookDiv.className = 'book-item';
|
||||
bookDiv.innerHTML = `
|
||||
<div class="book-icon">📖</div>
|
||||
<div class="book-name">${book.title}</div>
|
||||
`;
|
||||
const icon = document.createElement('div');
|
||||
icon.className = 'book-icon';
|
||||
icon.textContent = '📖';
|
||||
const name = document.createElement('div');
|
||||
name.className = 'book-name';
|
||||
name.textContent = book.title;
|
||||
bookDiv.append(icon, name);
|
||||
booksList.appendChild(bookDiv);
|
||||
});
|
||||
|
||||
@@ -189,17 +194,22 @@ function renderSchedules(schedules) {
|
||||
schedules.forEach(schedule => {
|
||||
const div = document.createElement('div');
|
||||
div.className = 'schedule-item';
|
||||
div.innerHTML = `
|
||||
<div class="schedule-info">
|
||||
<span class="schedule-time">🕐 ${formatDateTime(schedule.time)}</span>
|
||||
<span class="schedule-book">${schedule.book || 'Todos os livros'}</span>
|
||||
</div>
|
||||
<button class="schedule-delete" data-id="${schedule.id}">🗑️</button>
|
||||
`;
|
||||
const info = document.createElement('div');
|
||||
info.className = 'schedule-info';
|
||||
const time = document.createElement('span');
|
||||
time.className = 'schedule-time';
|
||||
time.textContent = formatDateTime(schedule.time);
|
||||
const book = document.createElement('span');
|
||||
book.className = 'schedule-book';
|
||||
book.textContent = schedule.book || 'Todos os livros';
|
||||
info.append(time, book);
|
||||
const remove = document.createElement('button');
|
||||
remove.className = 'schedule-delete';
|
||||
remove.type = 'button';
|
||||
remove.textContent = 'Excluir';
|
||||
remove.addEventListener('click', () => deleteSchedule(schedule.id));
|
||||
div.append(info, remove);
|
||||
container.appendChild(div);
|
||||
|
||||
// Add delete handler
|
||||
div.querySelector('.schedule-delete').addEventListener('click', () => deleteSchedule(schedule.id));
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -1,28 +0,0 @@
|
||||
# SDD ledger — plan: docs/superpowers/plans/2025-11-01-espiritismo-rag-system.md
|
||||
|
||||
## Pre-flight Review (from references/preflight.md)
|
||||
|
||||
| Check | Status | Notes |
|
||||
|-------|--------|-------|
|
||||
| Plan exists and reachable? | ✅ | `docs/superpowers/plans/2025-11-01-espiritismo-rag-system.md` |
|
||||
| Spec exists and reachable? | ✅ | `docs/superpowers/specs/2025-11-01-espiritismo-rag-system-design.md` |
|
||||
| Global Constraints present? | ✅ | CPU-only, 8GB RAM, ~5GB storage, Docker, etc. |
|
||||
| Tasks independent? | ⚠️ | Many tasks touch common code (embedding, PDF processing) - use batching for small same-shape work |
|
||||
| Worktree isolated? | ✅ | Using existing workspace at `lab/espiritismo` |
|
||||
|
||||
**Rulings:**
|
||||
- Ruling: Use batching for small same-shape work (Tasks 2, 4, 6, 7) — they touch common utilities and can be dispatched together to avoid context pollution. Costs: additional coordination but keeps context clean. Correctness: reduces context churn and improves parallelization.
|
||||
|
||||
## Task Progress
|
||||
|
||||
| Task | Status | Commits | Review |
|
||||
|------|--------|---------|--------|
|
||||
| 1 | ✅ Complete | - | - |
|
||||
| 2 | ✅ Complete | - | - |
|
||||
| 3 | - | - | - |
|
||||
| 4 | - | - | - |
|
||||
| 5 | - | - | - |
|
||||
| 6 | - | - | - |
|
||||
| 7 | - | - | - |
|
||||
| 8 | - | - | - |
|
||||
| 9 | - | - | - |
|
||||
Reference in new issue
Block a user