Implementation of FastText- and Hybrid Search-Based Retrieval-Augmented Generation for PDF Document Processing: A Case Study of a Booking Engine Chatbot System
Keywords:
Retrieval-Augmented Generation, FastText, Hybrid Search, Maximal Marginal Relevance, RAGAS, Chatbot, Natural Language ProcessingAbstract
This study proposes and evaluates a Retrieval-Augmented Generation (RAG) system integrating ChatGPT with FastText word embeddings for processing Indonesian-language PDF documents. The limitations of Large Language Models (LLMs) in handling domain-specific queries motivated the development of an RAG architecture with an advanced retrieval strategy. The CRISP-DM methodology was applied to compare two models: (1) a Baseline Model using pure FastText cosine similarity search, and (2) a Proposed Model combining Hybrid Search (FastText + BM25 with dynamic weighting), Maximal Marginal Relevance (MMR), and Cross-Encoder TF-IDF re-ranking. Evaluation was conducted using the RAGAS framework with five metrics—Context Relevance, Context Precision, Context Recall, Faithfulness, and Answer Relevance—across 20 test queries on the PT Omni Hottilier Booking Engine user guide. Results show that the Proposed Model outperformed the Baseline on 16 of 20 queries (80%), with the Overall Score increasing from 0.7853 to 0.8572 (+9.16%). The most significant improvements were observed in Context Relevance (+19.85%), Context Recall (+16.76%), and Faithfulness (+14.76%). The developed architecture is proven effective as an adaptive solution for transforming technical documents into intelligent chatbot knowledge bases, particularly for the morphologically complex Indonesian language.
References
H. Alboqami, "Factors Affecting Consumers Adoption of AI-Based Chatbots: The Role of Anthropomorphism," American Journal of Industrial and Business Management, vol. 13, no. 04, pp. 195–214, 2023. https://doi.org/10.4236/ajibm.2023.134014
K. Wang, J. Ramos, and R. Lawrence, "ChatEd: A Chatbot Leveraging ChatGPT for an Enhanced Learning Experience in Higher Education," arXiv:2401.00052, 2023.
Y. Gao et al., "Retrieval-Augmented Generation for Large Language Models: A Survey," arXiv:2312.10997, 2023.
P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, "Enriching Word Vectors with Subword Information," Transactions of the Association for Computational Linguistics, vol. 5, pp. 135–146, 2017.
E. M. Dharma et al., "The Accuracy Comparison Among Word2Vec, GloVe, and FastText Towards CNN Text Classification," Journal of Theoretical and Applied Information Technology, vol. 31, no. 2, 2022.
A. Bansal, "Optimizing RAG with Hybrid Search and Contextual Chunking," Journal of Engineering and Applied Sciences Technology, 2023. https://doi.org/10.47363/JEAST/2023(5)E114
O. Zaland, M. Abulaish, and Mohd. Fazil, "A Comprehensive Empirical Evaluation of Existing Word Embedding Approaches," arXiv:2303.07196, 2023.
A. Amalia et al., "An Efficient Text Classification Using fastText for Bahasa Indonesia Documents Classification," in Proc. DATABIA 2020, pp. 69–75.
Ö. Hakdağlı, "Hybrid Question-Answering System: A FAISS and BM25 Approach for Extracting Information from Technical Document," Orclever Proceedings, vol. 5, no. 1, pp. 226–237, 2024.
H.-L. Hsu and J. Tzeng, "DAT: Dynamic Alpha Tuning for Hybrid Retrieval in Retrieval-Augmented Generation," arXiv:2503.23013, 2025.
J. Carbonell and J. Goldstein, "The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries," in Proc. SIGIR, 1998.
S. G. Aranzamendez et al., "An Enhanced Content-based Filtering Using Maximal Marginal Relevance," International Journal of Computing Sciences Research, vol. 8, pp. 3070–3087, 2024.
S. Es et al., "RAGAS: Automated Evaluation of Retrieval Augmented Generation," arXiv:2309.15217, 2023.
A. A. Khan et al., "Developing Retrieval Augmented Generation (RAG) based LLM Systems from PDFs: An Experience Report," arXiv:2410.15944, 2024.
L. S. Hartono, E. I. Setiawan, and V. Singh, "Retrieval Augmented Generation-Based Chatbot for Prospective and Current University Students," IJESTY, vol. 5, no. 3, pp. 268–277, 2025.
G. D. Albert and A. Voutama, "Pengembangan Chatbot Berbasis PDF Menggunakan Local RAG dan Ollama," Jurnal Informatika dan Teknik Elektro Terapan, vol. 13, no. 2, pp. 937–944, 2025.
F. Schlatt et al., "Set-Encoder: Permutation-Invariant Inter-Passage Attention for Listwise Passage Re-Ranking with Cross-Encoders," 2024.
P. J. Sager et al., "Deep Retrieval at CheckThat! 2025: Identifying Scientific Papers via Hybrid Retrieval and Re-Ranking," arXiv:2505.23250, 2025.
G. Byun et al., "Secure Multifaceted-RAG for Enterprise: Hybrid Knowledge Retrieval with Security Filtering," arXiv:2504.13425, 2025.
W. Zhang and J. Zhang, "Hallucination Mitigation for Retrieval-Augmented Large Language Models: A Review," Mathematics, vol. 13, no. 5, 2025.
