discours.io/core
5
0
Fork 0
Code Pull Requests Actions Projects Activity
3,559 Commits 10 Branches 0 Tags
3c40bbde2bfed547d4c54a8001550bbe42f3567a
Commit Graph

83 Commits

Author SHA1 Message Date
Untone
3c40bbde2b 0.9.29] - 2025-10-08 
### 🎯 Search Quality Upgrade: ColBERT + Native MUVERA + FAISS

- **🚀 +175% Recall**: Интегрирован ColBERT через pylate с НАТИВНЫМ MUVERA multi-vector retrieval
- **🎯 TRUE MaxSim**: Настоящий token-level MaxSim scoring, а не упрощенный max pooling
- **🗜️ Native Multi-Vector FDE**: Каждый токен encode_fde отдельно → список FDE векторов
- **🚀 FAISS Acceleration**: Двухэтапный поиск O(log N) для масштабирования >10K документов
- **🎯 Dual Architecture**: Поддержка BiEncoder (быстрый) и ColBERT (качественный) через `SEARCH_MODEL_TYPE`
- **⚡ Faster Indexing**: ColBERT индексация ~12s vs BiEncoder ~26s на бенчмарке
- **📊 Better Results**: Recall@10 улучшен с 0.16 до 0.44 (+175%)

### 🛠️ Technical Changes

- **requirements.txt**: Добавлены `pylate>=1.0.0` и `faiss-cpu>=1.7.4`
- **services/search.py**:
  - Добавлен `MuveraPylateWrapper` с **native MUVERA multi-vector** retrieval
  - 🎯 **TRUE MaxSim**: token-level scoring через списки FDE векторов
  - 🚀 **FAISS prefilter**: двухэтапный поиск (грубый → точный)
  - Обновлен `SearchService` для динамического выбора модели
  - Каждый токен → отдельный FDE вектор (не max pooling!)
- **settings.py**:
  - `SEARCH_MODEL_TYPE` - выбор модели (default: "colbert")
  - `SEARCH_USE_FAISS` - включить FAISS (default: true)
  - `SEARCH_FAISS_CANDIDATES` - количество кандидатов (default: 1000)

### 📚 Documentation

- **docs/search-system.md**: Полностью обновлена документация
  - Сравнение BiEncoder vs ColBERT с бенчмарками
  - 🚀 **Секция про FAISS**: когда включать, архитектура, производительность
  - Руководство по выбору модели для разных сценариев
  - 🎯 **Детальное описание native MUVERA multi-vector**: каждый токен → FDE
  - TRUE MaxSim scoring алгоритм с примерами кода
  - Двухэтапный поиск: FAISS prefilter → MaxSim rerank
  - 🤖 Предупреждение о проблеме дистилляционных моделей (pylate#142)

### ⚙️ Configuration

```bash
# Включить ColBERT (рекомендуется для production)
SEARCH_MODEL_TYPE=colbert

# 🚀 FAISS acceleration (обязательно для >10K документов)
SEARCH_USE_FAISS=true              # default: true
SEARCH_FAISS_CANDIDATES=1000       # default: 1000

# Fallback к BiEncoder (быстрее, но -62% recall)
SEARCH_MODEL_TYPE=biencoder
```

### 🎯 Impact

- ✅ **Качество поиска**: +175% recall на бенчмарке NanoFiQA2018
- ✅ **TRUE ColBERT**: Native multi-vector без упрощений (max pooling)
- ✅ **MUVERA правильно**: Используется по назначению для multi-vector retrieval
- ✅ **Масштабируемость**: FAISS prefilter → O(log N) вместо O(N)
- ✅ **Готовность к росту**: Архитектура выдержит >50K документов
- ✅ **Индексация**: Быстрее на ~54% (12s vs 26s)
- ⚠️ **Latency**: С FAISS остается приемлемой даже на больших индексах
- ✅ **Backward Compatible**: BiEncoder + отключение FAISS через env

### 🔗 References

- GitHub PR: https://github.com/sionic-ai/muvera-py/pull/1
- pylate issue: https://github.com/lightonai/pylate/issues/142
- Model: `answerdotai/answerai-colbert-small-v1`
 2025-10-09 01:15:19 +03:00
Untone
4489d25913 ## [0.9.18] - 2025-01-09
Some checks failed
Deploy on push / deploy (push) Failing after 1m34s
Details 
### 🔍 Search System Redis Storage
- **💾 Redis-based vector index storage**: Переключились обратно на Redis для хранения векторного индекса
  - Заменили файловое хранение в `/dump` на Redis ключи для надежности
  - Исправлена проблема с правами доступа на `/dump` папку на сервере
  - Векторный индекс теперь сохраняется по ключам `search_index:{name}:data` и `search_index:{name}:metadata`
- **🛠️ Improved reliability**: Убрали зависимость от файловой системы для критичных данных
- **⚡ Better performance**: Redis обеспечивает более быстрый доступ к индексу
- **🔧 Technical changes**:
  - Заменили `save_index_to_file()` на `save_index_to_redis()`
  - Заменили `load_index_from_file()` на `load_index_from_redis()`
  - Обновили автосохранение для использования Redis вместо файлов
  - Удалили неиспользуемые импорты (`gzip`, `pathlib`, `cast`)
 2025-09-01 15:09:36 +03:00
Untone
db3dafa569 embedding-search
Some checks failed
Deploy on push / deploy (push) Failing after 22m28s
Details
2025-08-31 19:20:43 +03:00
Untone
4f63da037d alembic-removed 2025-08-28 19:42:03 +03:00
Untone
00a866876c search-wrapper
Some checks failed
Deploy on push / deploy (push) Failing after 4m31s
Details
2025-08-23 14:08:34 +03:00
Untone
b4f683a7cc fmt
Some checks failed
Deploy on push / deploy (push) Failing after 36s
Details
2025-08-23 10:47:52 +03:00
Untone
1b25738714 publish-shout-fix
Some checks failed
Deploy on push / deploy (push) Failing after 32s
Details
2025-08-21 12:16:30 +03:00
Untone
2a3464005f versions-fix 2025-08-20 20:54:58 +03:00
Untone
663942c41e feat: migrate to uv package manager 
- Add pyproject.toml with project configuration
- Update requirements.txt and requirements.dev.txt with versions
- Add .uv configuration file
- Update .gitignore for uv
- Update README with uv instructions
- Configure hatchling build system
- Add mypy configuration
- Test uv sync and pytest integration
 2025-08-12 13:12:39 +03:00
Untone
e7230ba63c tests-passed 2025-07-31 18:55:59 +03:00
Untone
b60a314ddd tested-auth-refactoring
Some checks failed
Deploy on push / deploy (push) Failing after 5s
Details
2025-07-25 01:04:15 +03:00
Untone
b5aa7032eb docs+featured/unfeatured-upgrade
All checks were successful
Deploy on push / deploy (push) Successful in 6s
Details
2025-06-19 11:28:48 +03:00
Untone
89f6c32b78 passlib-fixed-ver
Some checks failed
Deploy on push / type-check (push) Failing after 7s
Details
Deploy on push / deploy (push) Has been skipped
Details
2025-06-02 23:28:53 +03:00
Untone
3327976586 Improve topic sorting: add popular sorting by publications and authors count 2025-06-02 02:56:11 +03:00
Untone
bad4928219 depfix 2025-05-26 09:32:55 +03:00
Untone
91258721c6 Merge branch 'staging' of https://dev.dscrs.site/discours.io/core into feature/auth-internal 2025-05-21 23:00:03 +03:00
Untone
dc5ad46df9 wip 2025-05-19 11:25:41 +03:00
Stepan Vladovskiy
f19248184a debug: without ersions for starlette and ariadne
All checks were successful
Deploy on push / deploy (push) Successful in 1m5s
Details
2025-05-18 22:48:34 +00:00
Stepan Vladovskiy
e38a1c1338 with vers for starlette, ariadne, granian
Some checks failed
Deploy on push / deploy (push) Failing after 50s
Details
2025-05-18 22:36:47 +00:00
Stepan Vladovskiy
e382cc1ea5 Merge branch 'dev' into feat/sv-searching-txtai
All checks were successful
Deploy on push / deploy (push) Successful in 6s
Details 
:
 2025-04-15 19:20:48 -03:00
Untone
b9f6033e66 generate seo text when draft created 2025-04-15 20:09:22 +03:00
Stepan Vladovskiy
9a02ca74ad merged with dev
All checks were successful
Deploy on push / deploy (push) Successful in 1m24s
Details
2025-03-31 13:38:32 -03:00
Untone
247fc98760 cachedep-fix+orjson+fmt
All checks were successful
Deploy on push / deploy (push) Successful in 1m16s
Details
2025-03-20 11:55:21 +03:00
Untone
a1781b3800 depfix
All checks were successful
Deploy on push / deploy (push) Successful in 1m4s
Details
2025-03-20 11:36:12 +03:00
Untone
ae48a18536 comment-delete-handling-patch
All checks were successful
Deploy on push / deploy (push) Successful in 1m15s
Details
2025-03-20 11:01:39 +03:00
Stepan Vladovskiy
f249752db5 feat: moved txtai and search procedure in different instance
All checks were successful
Deploy on push / deploy (push) Successful in 2m18s
Details
2025-03-12 12:06:09 -03:00
Stepan Vladovskiy
d55448398d feat(search.py): change to txtai server, with ai model. And fix granian workers 2025-03-05 20:08:21 +00:00
Untone
a84d8a0c7e 0.4.9-c
All checks were successful
Deploy on push / deploy (push) Successful in 7s
Details
2025-02-10 18:04:08 +03:00
Untone
db76ba3733 0.2.14
Some checks failed
deploy / deploy (push) Failing after 2m1s
Details
2023-11-22 19:38:39 +03:00
Untone
bbd8f61408 redis update 2023-10-13 13:13:45 +03:00
Tony Rewin
8a3aa1dae6 fix-reqs 2023-10-06 03:55:43 +03:00
Tony Rewin
b64d9d5014 poetry-rty 2023-10-06 03:22:37 +03:00
Tony Rewin
b2e196d261 forked-ariadne 2023-10-06 02:01:18 +03:00
Tony Rewin
0e8e8f4d04 git+ssh 2023-10-06 01:49:34 +03:00
Tony Rewin
8de2eb385b async-fix 2023-10-06 01:45:32 +03:00
Tony Rewin
120208a621 rollback-requests 2023-10-06 00:10:46 +03:00
Tony Rewin
8524d0f843 edge 2023-10-06 00:05:15 +03:00
Tony Rewin
d26d444975 deps-workaround2 2023-10-06 00:02:25 +03:00
Tony Rewin
e0bd938a6e deps-workaround 2023-10-05 23:57:04 +03:00
Tony Rewin
aed91c6375 deps... 2023-10-05 23:55:23 +03:00
Tony Rewin
c57f3857a6 import-fix4 2023-10-05 23:47:51 +03:00
Tony Rewin
c665c0056c import-fix4 2023-10-05 23:45:21 +03:00
Tony Rewin
3c936e7860 deps... 2023-10-05 23:22:11 +03:00
Tony Rewin
5fedd007c7 git-dep3 2023-10-05 23:18:06 +03:00
Tony Rewin
3d659caa6e git-dep2 2023-10-05 23:05:09 +03:00
Tony Rewin
9d2cd9f21f git-dep 2023-10-05 23:04:09 +03:00
Tony Rewin
45d187786b fix-imports 2023-10-05 22:59:50 +03:00
Tony Rewin
9537814718 deps-fixes 2023-10-05 22:38:35 +03:00
Tony Rewin
b8e6f7bb5a requests-removed+fixes 2023-10-05 22:18:05 +03:00
Tony Rewin
fbc85f6c2d aioredis-removed 2023-10-05 22:00:24 +03:00