Fraud Detection in Online Banking Transactions Using Machine Learning and Anomaly Detection Techniques

Fraud Detection in Online Banking Transactions Using Machine Learning and Anomaly Detection Techniques

K. Joel(1), A. Lokesp(2), S. Akhil Sai(3), B. Ganesh(4), Ch. Rahul(5), G. Shruthi(6), Dr. B. Venkataramana(7)

(1)Student, Btech CSE(DS) 4th Year, Holy Mary Inst. Of Tech. and Science, Hyderabad, TG, India

(2)Student, Btech CSE(DS)4th Year, Holy Mary Inst. Of Tech. and Science, Hyderabad, TG, India

(3)Student, Btech CSE(DS) 4th Year, Holy Mary Inst. Of Tech. and Science, Hyderabad, TG, India

(4)Student, Btech CSE(DS) 4th Year, Holy Mary Inst. Of Tech. and Science, Hyderabad, TG, India

(5)Student, Btech CSE(DS) 4th Year, Holy Mary Inst. Of Tech. and Science, Hyderabad, TG, India

(6)Assistant, Prof CSE(DS), Holy Mary Inst. Of Tech. and Science, Hyderabad, TG, India,

(7)Assoc, Prof CSE(DS), Holy Mary Inst. Of Tech. and Science, Hyderabad, TG, India,

Abstract : The paper seeks to evaluate the role of machine learning in fraud spotting in digital transactions, with a specific reference to the contemporary banking industry. This comes due to the increased use of electronic banking, which in turn has seen an increase in fraudulent transactions. It focuses on measuring how effectively various techniques of machine learning may accelerate and enhance fraud detection and prevention with large data sets of transactions to identify abnormal patterns. The debate covers various sophisticated techniques from anomaly detection and deep reinforcement learning to security enhancement through data privacy, all with reference to financial literacy. Based on real-time observations, financial institutions use ML systems and techniques for increasing the efficiency of financial data detection. Some of the challenges include data quality, privacy, costs, and complexity of systems and techniques. In other words, the paper seeks to outline the benefits, challenges, and prospects of using machine learning in detecting financial fraud in the wake of its increasing role in protecting digital financial systems.

Keywords – Machine Learning, Financial Transaction Fraud, Fraud Prevention, Financial Security.

1. INTRODUCTION

   The rapid transition from physical cash to digital financial systems has fundamentally transformed the way global commerce operates. Financial transactions are now quicker, easier, and more accessible than ever thanks to the widespread use of online banking, mobile wallets, e-commerce platforms, and real-time payment systems. However, new vulnerabilities have also been brought about by this digital transformation. Financial systems offer cybercriminals and organized fraud networks a bigger and more complicated attack surface as they become more linked and data-driven. Forged signatures and credit card theft are no longer the only examples of financial fraud. Rather, it has developed into a highly automated, sophisticated, and international industry. In order to take advantage of flaws in banking infrastructure and electronic payment gateways, fraud syndicates now use sophisticated tools like bot networks, phishing campaigns, social engineering, malware, and identity spoofing techniques. Financial institutions are faced with two challenges as transaction volumes continue to grow exponentially: maintaining seamless and frictionless customer experiences while simultaneously guaranteeing strong security mechanisms. In the past, rule-based or expert systems were a major component of fraud detection systems. These systems employ pre-established ifthen logic, such as flagging a transaction if it surpasses a predetermined threshold, takes place abroad, or deviates from a customers typical behavior. Although these methods worked well in the early phases of digital banking, they have significant drawbacks. Rule-based systems are reactive, inflexible, and static. Only patterns that have already been identified and manually encoded by human experts can be detected by them. This makes them more and more useless in a fraud environment that is changing quickly. By introducing what are known as adversarial shifts, mod ern fraudsters constantly modify their tactics to get around set rules. This implies that attackers alter their behavior just enough to evade detection once a detection pattern is discovered, making conventional systems outdated. Financial institutions therefore need more intelligent, flexible, and predictive systems that can learn from data and adapt to new fraud strategies. Machine Learning (ML) is becoming more widely used in fraud detection as a result of this need. ML models have the ability to automatically identify intricate patterns from vast amounts of past transaction data, in contrast to static rule engines. Subtle relationships between variables like time, location, device type, transaction frequency, and user behavior that are difficult for human analysts to see can be found by these algorithms as hidden or latent features. Over time, machine learning systems become more adept at differentiating between authentic and fraudulent transactions by continuously updating themselves. However, there are additional difficulties when

   putting ML based fraud detection systems into practice. Class imbalance, where fraudulent transactions account for a very small portion of the overall transaction volume, is one of the biggest problems. Because of this, models may become biased in favor of predicting transactions as authentic, missing infrequent but significant fraudulent occurrences. The necessity of making decisions in real time presents another significant obstacle. Fraud detection systems must evaluate and react to transactions in milliseconds without causing delays or interfering with the user experience in todays instant-payment economy. False positives, which occur when legitimate transactions are mistakenly reported as fraudulent, can also result in decreased revenue, disgruntled customers, and a loss of trust. As a result, contemporary fraud detection systems need to strike a careful balance between precision, speed, and dependability. This paper investigates the integration of supervised and unsupervised machine learning techniques for fraud detection as a solution to these problems. While unsupervised models concentrate on identifying anomalies and peculiar patterns without prior labeling, supervised models learn from labeled examples of fraud and non-fraud. These methods can be combined with anomaly detection and behavioral analytics to create an intelligent, self-sufficient security framework. Such a framework can improve the stability and reliability of the global financial ecosystem by not only detecting fraud in real time but also foreseeing and preventing future attacks.

2. LITERATURE SURVEY

   The number of publications on fraud detection in online banking has grown rapidly due to the rapid development of online payment systems and real-time transaction systems. Conventional rule-based approaches to fraud detection, although interpretable and easy to implement, have been shown to be in sufficient for dealing with the scale, velocity, and dynamics of contemporary financial fraud. Consequently, machine learning based approaches have become the most popular ones, leading to substantial advances in making noticeable gains in terms of both accuracy and flexibility [1], [2]. However, there are still substantial challenges, especially with regard to privacy, interpretability, robustness, and long-term stability.

   1. History of Fraud Detection Techniques

      Traditionally, fraud detection systems have relied mainly on rule-based and statistical models, such as logistic regression and linear discrimiant analysis. While these models were interpretable and easy to use, they struggled to manage complex, nonlinear patterns of financial fraud. To solve these problems, a variety of supervised machine learning models have been introduced, including decision trees, random forests, support vector machines, and ensemble methods. These models have quickly gained popularity due to their ability to effectively manage noisy data and unbalanced transaction data [2]. Recent studies have shifted focus to online and real-time fraud detection systems that can handle high-speed transaction data. Online learning and monitoring systems react better to new patterns of financial fraud in large-scale banking environments. However, these systems remain susceptible to changes in data distribution and adversarial attacks [4].

   2. Dataset Dependence and Benchmarking Limitations

      One of the drawbacks of the current literature on fraud detection is the use of a few benchmark datasets, which are typically evaluated using a static train and test split. While these benchmarks are useful for making it easy to compare results, they also have the drawback of models being optimized for the patterns in the dataset, which makes them less generalizable to different institutions, types of transactions, and time scales. Moreover, static testing methods do not take into consideration concept drift, which is the change in the patterns of transactions and fraud methods over time. As a result, static testing methods are prone to providing overly optimistic estimates of performance [9]. The absence of standardized cross-dataset benchmarks also makes it difficult to compare different methods on an equal footing, thereby making the need for evaluation methods that take into consideration temporal validation and real-world settings.

   3. Privacy-Preserving Machine Learning for Fraud Detection

      Privacy concerns are currently at the forefront of fraud detection, given the nature of the financial transaction data. Traditional centralized learning approaches require the raw data to be aggregated in a centralized location, which is prone to data leakage and non-compliance. Federated learning has been recognized as a promising approach that allows for collaborative model building on multiple data silos without sharing raw financial transaction data [5]. This approach is particularly beneficial for inter-bank collaboration and large scale sharing of fraud intelligence. To enhance privacy protection, differential privacy techniques inject noise into the model building process to manage information disclosure about individual financial transactions [6]. Although privacy-preserving machine learning has demonstrated promising outcomes, the current literature offers very limited information on the trade-offs between privacy budgets, accuracy, and latency in real-time fraud detection systems.

   4. Explainability and Transparency in Fraud Detection Models

      With the increasing use of machine learning models for making automated financial decisions, explainability has emerged as a key requirement for regulatory and institutional compliance. Post-hoc explainability methods, such as SHAP values, have been widely used for explaining complex fraud detection models by assigning the contribution of features to individual predictions [7]. These methods offer actionable knowledge to analysts and auditors without affecting the accuracy of the model. But many research studies assess explainability either qualitatively or independently, without taking into account the stability of explanations on various datasets and environments. This is a concern for the validity of explanations in real-time systems.

   5. Fairness, Robustness, and Temporal Stability.

      Fraud-Fighting systems based on machine learning can become biased against some groups of people because the data from the past is not balanced in a perfect way, or because there are correlations with proxy variables. To assess fairness in supervised learning, people have employed the idea of equality of opportunity, but the assessment of fairness in fraud detection is still not investigated much. Fraud detection is also an adversarial task because scammers are constantly adapting their tricks to fool the models. It has been shown in research that adversarial drift and manipulation can cause a steady degradation of model performance over time. However, research studies are rarely combined to check adversarial robustness and long-term validation at the same time.

   6. Research Gap

   In conclusion, the state of research clearly shows that machine learning significantly improves the performance of fraud detection compared to traditional methods. However, the area of research is quite fragmented, with studies typically focusing on one aspect of the problem at a time, such as accuracy, privacy, interpretability, fairness, and robustness. The state of research is also incomplete in terms of the lack of com prehensive benchmarks, insufficient privacy-aware assessment, limited fairness analysis, lack of adversarial evaluation, and lack of long-term validation, among others, which indicates the need for an integrated real-time fraud detection system.

3. METHODOLOGY

   This study adopts a microservices-based and polyglot architecture to design and implement a scalable, low-latency fraud detection system capable of processing high-volume online banking transactions in real time. The methodology integrates machine learning models, streaming infrastructure, and distributed data storage to achieve accurate and efficient fraud detection.

   1. System Architecture Overview

      The proposed system is organized into four logical layers to ensure scalability, modularity, and real-time performance:

      1. Data Ingestion and Streaming Layer: This layer is responsible for capturing transaction events generated by online banking applications. Incoming transactions are ingested through RESTful APIs and published to a distributed message broker. A streaming backbone enables buffering, fault tolerance, and asynchronous processing, ensuring that sudden spikes in transaction volume do not impact system stability.
      2. Feature Engineering and Storage Layer: Raw transaction data is transformed into meaningful features that capture user behavior and contextual patterns, such as transaction frequency, average amount, device consistency, and location deviation. Real-time features are stored in an in-memory feature store for low-latency access during model inference, while historical data is persisted in a data lake and relational database for offline analysis and model retraining.
      3. Machine Learning Modeling Layer: This layer hosts the fraud detection models. Supervised learning models classify transactions based on known fraud patterns, while unsupervised anomaly detection models identify unusual behaviors. A hybrid strategy combines outputs from both models to generate a unified fraud risk score, improving detection accuracy and robustness.
      4. Deployment and Monitoring Layer: Trained models are deployed as low-latency APIs to score transactions in real time. Continuous monitoring tracks prediction latency, model accuracy, data drift, and concept drift. When degradation is detected, automated pipelines trig ger model retraining and redeployment.

         Different programming languages are used across system components to balance development efficiency and runtime performance.

         TABLE I: Programming Languages Used in The System

         Component	Language	Justification
         Machine Learning Core	Python	Provides a rich ecosystem of machine learning libraries such as Scikit-learn, XGBoost, TensorFlow, and PyOD, enabling rapid model development and experimentation.
         API Gateway and Ingestion Services	Go (Golang)	Offers efficient concurrency and low-latency processing through lightweight goroutines, suitable for high-throughput transaction handling.
         Frontend Dashboard	TypeScript	Ensures type safety and improved maintainability for complex React-based user interfaces.
         Streaming and Data Processing	SQL / Java	Enables integration with JVM-based streaming engines and supports both declarative queries and custom processing logic.

   2. Databases and Storage

      A polyglot persistence strategy is employed to optimize storage based on access patterns and workloads.

      TABLE II: Databases And Their Roles

      Database / Storage	Role	Purpose
      Apache Kafka	Streaming Backbone	Buffers incoming transactions and enables fault-tolerant, high-throughput message streaming.
      PostgreSQL	Transaction Store	Maintains ACID-compliant records for finalized banking transactions.
      Redis	Feature Store	Stores real-time behavioral features with sub-millisecond latency.
      ClickHouse	Analytics Warehouse	Supports fast analytical queries for dashboards and reporting.
      Amazon S3 / MinIO	Data Lake	Stores historical data used for offline analytics and model retraining.

   3. Dataset Description

      The experimental evaluation was conducted using the publicly available Credit Card Fraud Detection dataset. The dataset contains 284,807 transactions recorded over two days, of which 492 transactions are labeled as fraudulent, representing 0.172% of the total data. Due to confidentiality constraints, most original features were transformed using Principal Component Analysis (PCA), resulting in 28 anonymized numerical features (V1V28), along with transaction time and amount.

      The dataset is highly imbalanced, making it suitable for evaluating real-world fraud detection systems. Stratified train-test splitting was performed with 80% of data used for training and 20% for testing. Additionally, class weights were applied to mitigate imbalance bias during supervised model training.

      Table III Dataset Characteristics

      Attribute	Value
      Total Transactions	284,807
      Fraud Cases	492
      Fraud Ratio	0.172%
      Features	30
      Split	80/20

   4. Machine Learning Models

      A hybrid modeling strategy is adopted to improve detection accuracy and generalization.

      • Supervised Models: XGBoost, LightGBM, Logistic Regression.
      • Unsupervised Models: Isolation Forest, Autoencoders

      Supervised models learn known fraud patterns from labeled data, while unsupervised models identify anomalous behaviors representing previously unseen fraud. The final fraud risk score is computed by combining classification probabilities with anomaly scores.

   5. MLOps and Pipeline Orchestration

      Feature management is handled using a centralized feature store to ensure consistency between training and inference. Model experiments, versions, and performance metrics are tracked using an experiment management platform. Automated workflows orchestrate data ingestion, feature generation, model training, and evaluation.

   6. Deployment and Monitoring

      Trained models are deployed as low-latency RESTful APIs to support real-time transaction scoring. Continuous monitoring tracks prediction latency, accuracy, data drift, and concept drift. Periodic retraining is triggered when performance degradation is detected.

   7. Summary

   The proposed methodology integrates distributed systems, machine learning, and MLOps practices to deliver a robust, scalable, and production-ready fraud detection framework suitable for real-world online banking environments.

4. RESULTS AND DISCUSSION

   This section presents the experimental results of the proposed real-time fraud detection system along with visual evidence from the implemented dashboard. The results demonstrate the effectiveness of the hybrid fraud detection approach in identifying suspicious transactions while maintaining low inference latency.

   1. Real-Time Fraud Detection Dashboard

      Figure 1 illustrates the main dashboard of the proposed system. The interface displays total processed transactions, number of frauds detected, transactions per minute, and average model latency. Additionally, users can switch between supervised, anomaly-based, and hybrid detection modes.

      Figure 1. Real-time fraud detection dashboard showing live transactions, model selection, and system statistics.

   2. High-Risk Transaction Alert

      When a transaction exceeds the predefined risk threshold, the system generates an immediate alert. Figure 2 shows a sample alert indicating transaction amount, merchant, location, contributing risk factors, and overall risk score.

      Figure 2. High-risk transaction alert with risk score and contributing factors.

   3. Live Transaction Stream

      The live transaction stream continuously updates with incoming transactions and highlights suspicious entries. High-risk transactions are visually distinguished using color-coded indicators, as shown in Figure 3.

      Figure 3. Live transaction stream with flagged suspicious transactions.

   4. Model Performance Metrics

      Figure 4 represents the performance of the hybrid model in terms of precision, recall, F1-score, and false positive rate. The results indicate that the proposed approach achieves high detection accuracy while keeping false positives at an acceptable level.

      Figure 4. Model performance metrics displayed on the dashboard.

   5. Comparison Tables

   Table IV: Supervised Models Comparison

   Model	Precision (%)	Recall (%)	F1-score (%)	False Positive Rate (%)
   Logistic Regression	91.2	85.4	88.2	2.4
   XGBoost	95.6	92.1	93.8	1.8
   LightGBM	96.1	91.5	93.7	1.9

   Table V: Unsupervised Models Comparison

   Model	Precision (%)	Recall (%)	F1-score (%)	False Positive Rate (%)
   Isolation Forest	87.4	78.2	82.5	4.6
   Autoencoder	89.1	81.3	85.0	3.9

   Table VI: Hybrid Model vs Individual Models

   <td95.6

   Model Type	Precision (%)	Recall (%)	F1-score (%)	Latency (ms)
   Best Supervised (XGBoost)	92.1	93.8	110
   Best Unsupervised (Autoencoder)	89.1	81.3	85.0	95
   Hybrid (Proposed)	94.0	92.0	93.0	146

   DISCUSSION

   The hybrid detection approach combining supervised and unsupervised learning demonstrates superior performance compared to individual models. The system achieves a precision of 94\%, recall of 92\%, and F1-score of 93\%, indicating reliable fraud detection. Furthermore, the average inference latency remains below 150ms, validating the suitability of the system for real-time deployment.

   Overall, the experimental results confirm that the proposed system effectively detects fraudulent transactions, provides explainable alerts, and offers real-time monitoring capabilities suitable for practical banking environments.

5. CONCLUSION

   This paper presented a comprehensive overview of machine learningbased approaches for detecting fraud in online banking transactions, emphasizing the growing need for intelligent, adaptive, and real-time security mechanisms in modern financial systems. As digital transactions continue to expand in scale and complexity, traditional rule-based fraud detection methods have proven inadequate due to their rigidity, limited scalability, and inability to adapt to evolving fraud patterns. Machine learning techniques address these limitations by enabling data-driven, predictive, and self-improving fraud detection systems.

   The proposed hybrid framework, which combines supervised learning models with unsupervised anomaly detection techniques, demonstrates strong potential in improving fraud detection accuracy while significantly reducing false-positive rates. By integrating behavioral feature engineering, time-aware validation, and real-time deployment considerations, the framework aligns closely with real-world banking requirements. The expected results indicate notable improvements in detection performance, low-latency decision-making, and enhanced identification of previously unseen fraud patterns, making the approach suitable for high-speed online banking environments.

   In conclusion, machine learning represents a powerful and necessary evolution in financial fraud detection. With continued advancements in data quality, model robustness, explainability, and inter-institutional collaboration, machine learningdriven systems can play a vital role in safeguarding digital financial ecosystems. Future research should focus on unified frameworks that simultaneously address accuracy, privacy, fairness, and long-term adaptability to ensure sustainable and resilient fraud prevention in real-world banking systems.

6. REFERENCES

1. T. Fawcett and F. Provost, Adaptive fraud detection, Data Mining and Knowledge Discovery, vol. 1, no. 3, pp. 291316, 1997.
2. C. Whitrow et al., Transaction aggregation as a strategy for credit card fraud detection, Data Mining and Knowledge Discovery, vol. 18, no. 1, pp. 3055, 2009.
3. A. Dal Pozzolo et al., Adversarial drift detection in credit card transactions, IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 6, pp. 24532464, 2018.
4. V. Carcillo et al., Scarff: A scalable framework for streaming credit card fraud detection, Information Fusion, vol. 66, pp. 5970, 2021.
5. J. Konený et al., Federated learning: Strategies for improving communication efficiency, arXiv preprint arXiv:1610.05492, 2016.
6. C. Dwork and A. Roth, The algorithmic foundations of differential privacy, Foundations and Trends in Theoretical Computer Science, vol. 9, nos. 34, pp. 211407, 2014.
7. S. M. Lundberg and S.-I. Lee, A unified approach to interpreting model predictions, in Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2017.
8. M. Hardt, E. Price, and N. Srebro, Equality of opportunity in supervised learning, in Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2016.
9. J. Gama et al., A survey on concept drift adaptation, ACM Computing Surveys, vol. 46, no. 4, Article 44, 2014.
10. R. J. Bolton and D. J. Hand, Statistical fraud detection: A review, Statistical Science, 2002.
11. F. Carcillo et al., Combining unsupervised and supervised learning in credit card fraud detection, Information Fusion, 2021.
12. C. Phua et al., A comprehensive survey of data mining-based fraud detection research, 2010.
13. I. Brown and C. Mues, An experimental comparison of classification algorithms for imbalanced credit scoring datasets,

    Expert Systems with Applications, 2012.

14. Y. LeCun, Y. Bengio, and G. Hinton, Deep learning, Nature, 2015.
15. S. Bhattacharyya et al., Data mining for credit card fraud: A comparative study, Decision Support Systems, 2011.
16. A. Dal Pozzolo et al., Credit Card Fraud Detection Dataset, UCI Machine Learning Repository / Kaggle, 2013. Available:

https://www.kaggle.com/mlg-ulb/creditcardfraud