Help, how to optimize the knowledge base retrieval speed without compromising query result accuracy

Help Me Build Seeking help
1

:bullseye: What I Need Help With

[The Dify version used is 1.9.1. When retrieving knowledge base in Chatflow, no matter how I optimize, the retrieval time remains above 8 seconds. The optimizations I’ve tried include:

  • Simplifying documents;
  • Breaking down documents into smaller ones based on major headings, adding metadata to the knowledge base, and using metadata filtering during retrieval to narrow the query scope;
  • Adding WORKER_CONCURRENCY=10 to .env and restarting, but the concurrency setting did not take effect; modifying the command parameter of the worker service in docker-compose.yaml to add --concurrency=10 and --queues=knowledge_base,document, but the concurrency still did not change (finally succeeded in setting it to 10 via .env’s CELERY_WORKER_AMOUNT, but the single retrieval task time did not decrease);
  • Adjusting Weaviate’s environment variables in docker-compose.yaml (LIMIT_RESOURCES_MEMORY=2G/4G, HNSW_EF=50/100, HNSW_EF_CONSTRUCTION=200/300);
  • Restarting the Weaviate service multiple times and forcibly rebuilding all knowledge base indexes via command line;
  • Verifying Weaviate resource usage (CPU/memory both under 90%), but retrieval speed showed no improvement;
  • Adjusting Top-K to 2 (not down to 1) in the Chatflow retrieval node and disabling Rerank re-ranking;
  • Disabling hybrid retrieval and switching to pure vector retrieval.]