Optimization of RNN and Tree-Based Models with Imbalance Handling for Fraud Detection in Digital Banking Transactions

Abstract

This study focuses on addressing the growing challenge of fraud detection in digital banking transactions, which has intensified alongside the rapid expansion of digital financial services. Fraud detection is particularly complex due to the highly imbalanced nature of transaction data, large data volumes, and intricate transaction patterns that make fraudulent activities difficult to identify accurately. Although previous research has applied a wide range of methods, from conventional machine learning techniques to advanced deep learning models, many approaches still face limitations in balancing high detection accuracy with computational efficiency. The main objective of this research is to compare the performance of Recurrent Neural Network (RNN)–based models, including Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), and Bidirectional LSTM (BiLSTM), with tree-based ensemble models such as XGBoost and LightGBM in detecting fraudulent banking transactions. To enhance model effectiveness, the study implements a comprehensive data preprocessing framework that includes data cleaning, feature engineering, and techniques for handling class imbalance, particularly the use of Synthetic Minority Over-sampling Technique (SMOTE). Furthermore, model performance is optimized through systematic hyperparameter tuning using Optuna, Hyperopt, and Keras Tuner. Evaluation is conducted using metrics suitable for imbalanced datasets, such as precision, recall, F1-score, and AUC-ROC. The expected outcome is the identification of a robust and efficient fraud detection model that improves detection accuracy and sensitivity, while offering valuable insights for both academic research and practical banking applications.