Algorithms for detecting anomalies

[Mutt]

Machine learning (ML) for anomaly detection in banking systems, such as those used in 3D-Secure, relies on specialized algorithms, including Isolation Forest and autoencoders , among other approaches. These algorithms are effective in detecting deviations from normal behavior that may indicate fraudulent transactions. Below is a detailed explanation of how they work, how they work, and how they are applied in risk analysis, with a focus on the banking context.

1. Isolation Forest​

Isolation Forest is an unsupervised learning algorithm specifically designed for anomaly detection. It is based on the idea that anomalies (outliers) are easier to isolate than normal data because they differ in characteristics.

How Isolation Forest Works​

• Principle: The algorithm builds a set of random Isolation Trees. Anomalies are isolated faster because they are further away from the main data.
• Process:
  1. Data Splitting: Given a data set (e.g. transactions with features: amount, time, IP address), the algorithm randomly selects a feature and a value to split the data into two parts.
  2. Tree construction: The process is repeated recursively, creating a tree until every data point is isolated or the maximum depth is reached.
  3. Anomaly scoring: Points that are isolated in fewer splits (short path in the tree) are considered anomalies because they are different from the majority.
  4. Anomaly score: For each transaction, the average path length across all trees is calculated. A short path (low value) indicates an anomaly.
• Example: If a customer usually spends $10 in London, and a new transaction for $500 is from another region, it will be isolated faster (less splits), since it deviates greatly from normal behavior.

Benefits in a banking context​

• Speed: Very fast as it does not require complex calculations (O(n log n) to build trees).
• Big Data Efficiency: Suitable for processing millions of transactions in real time.
• Dealing with imbalanced data: Anomalies are rare, and Isolation Forest does not require a large number of labeled fraudulent transactions.
• Noise Robustness: Handles high-dimensional data well (e.g. sum, geolocation, device type).

Disadvantages​

• Setting Sensitivity: Incorrect selection of tree number or depth can reduce accuracy.
• Limited interpretability: It is difficult to explain why a particular transaction is marked as an anomaly.
• Problems with clustered anomalies: If fraudulent transactions form a cluster (e.g. mass attacks), the algorithm may miss them.

Application in banks​

• Used to detect unusual transactions, such as purchases from a new device or in a suspicious region.
• Integrates into FDS systems (e.g. FICO Falcon), where it analyzes features such as deviation of the amount from the average or change of geolocation.

2. Autoencoders​

Autoencoders are neural networks used for unsupervised learning that compress data into a compact representation and attempt to reconstruct it. Anomalies are detected when the reconstructed data is very different from the original.

How Autoencoders Work​

• Structure:
  • They consist of two parts: an encoder (compresses data into a low-dimensional representation) and a decoder (recovers data from the compressed representation).
  • For example, a transaction with 50 attributes (amount, time, IP) is compressed to 10 attributes and then restored back.
• Process:
  1. Training: The model is trained on normal transactions, minimizing the reconstruction error (the difference between input and output data).
  2. Anomaly Detection Transactions that the model does not recover well (high recovery error) are considered anomalies because their characteristics differ from normal ones.
  3. Score: The recovery error (e.g. MSE - mean square error) is used as the anomaly score.
• Example: If a customer typically buys groceries for $10, the autoencoder will learn to reconstruct such transactions well. Buying a yacht for $1 million will cause a large reconstruction error, signaling an anomaly.

Benefits in a banking context​

• Handling complex data: Autoencoders work well with high-dimensional and non-linear data (e.g. combinations of geolocation, time, device type).
• Adaptation to new anomalies: Does not require explicit fraud markers, which is useful for detecting new types of attacks.
• Flexibility: Can be integrated with deep neural networks to improve accuracy.

Disadvantages​

• High computational complexity: Require significant resources to train and operate, especially on big data.
• Tuning hyperparameters: It is necessary to carefully tune the network architecture (number of layers, neurons) and the error threshold.
• Risk of overfitting: If a model fits normal data too well, it may miss out on anomalies.

Application in banks​

• Used for real-time transaction analysis, especially in 3D-Secure 2.0, where dozens of features (IP, device, behavior) are collected.
• Suitable for detecting complex anomalies, such as when a fraudster imitates normal behavior, but small details (typing speed, unusual browser) reveal an anomaly.

3. Other algorithms for detecting anomalies​

In addition to Isolation Forest and autoencoders, banks also use other approaches:

Random Forest (supervised learning)​

• How it works: Creates multiple decision trees, each of which votes on whether a transaction is fraudulent or legitimate. Used when there is labeled data.
• Use: To assess risk based on historical fraud data. For example, Random Forest can detect that transactions from certain IP addresses are often associated with fraud.
• Advantages: High accuracy, interpretability (you can understand which features are important).
• Disadvantages: Requires high-quality labeled data, which is expensive and labor-intensive.

DBSCAN (Density-Based Spatial Clustering of Applications with Noise)​

• How it works: Clusters data based on its density. Points that are not in dense clusters (normal behavior) are considered anomalies.
• Application: Suitable for identifying rare transactions that do not fit into common patterns (e.g. purchase from a suspicious store).
• Advantages: Does not require specifying the number of clusters, effective for multidimensional data.
• Disadvantages: Sensitive to the choice of parameters (radius, minimum number of points).

One-Class SVM (Support Vector Machine)​

• How it works: Trains only on normal data, creating a hyperplane that separates normal points from potential anomalies.
• Application: Used to detect new types of fraud when labeled data is scarce.
• Advantages: Works well with small data sets.
• Disadvantages: Does not scale well to millions of transactions due to high computational complexity.

LSTM (Long Short-Term Memory)​

• How it works: A type of recurrent neural network that analyzes sequences of transactions over time. Detects anomalies based on deviations in the sequence (e.g. a series of rapid transactions).
• Application: Suitable for analyzing time patterns, such as detecting card testing (repeated small payments).
• Advantages: Takes into account the time context, which is important for sophisticated attacks.
• Disadvantages: Requires large computing resources and complex setup.

4. Example of a script​

Let's say a bank analyzes a customer's transaction:

• Data: Purchase for $1,000 in an online electronics store, new IP address, time - 2:00 am, device - unknown Android.
• Isolation Forest: Quickly isolates the transaction because it differs from the customer's typical purchases (usually $10 in supermarkets). A short path in the tree gives a high anomaly score.
• Autoencoder: Attempts to recover transaction data. High recovery error (due to unusual amount, time, and device) indicates an anomaly.
• Random Forest: If there is labeled data, the model can mark that similar transactions from this IP are associated with fraud and give a high risk.
• Result: The bank requests OTP or biometrics via 3D-Secure or rejects the transaction.

5. Practical aspects in banks​

• Combined approach: Banks rarely use just one algorithm. For example, Isolation Forest can quickly filter out obvious anomalies, while autoencoders can identify complex patterns.
• Model Updating: Models are retrained daily or weekly to adapt to new types of fraud.
• Integration with 3D-Secure: Algorithms work in real time, analyzing up to 100 features (IP, geolocation, behavior) to decide on "frictionless" or "challenge" flow.
• Infrastructure: Cloud platforms (AWS, Google Cloud) or systems like Apache Spark are used to process big data.

6. Limitations and Challenges​

• False positives: Algorithms can flag legitimate transactions (such as a gift purchase) as anomalies, reducing the customer experience.
• Scammer adaptation: Attackers can mimic normal behavior by making the task more difficult (for example, by using a stolen device).
• Performance tradeoff: Isolation Forest is faster but less accurate for complex anomalies; autoencoders are more accurate but more resource intensive.

If you want to dive deeper into a specific algorithm (like the math behind Isolation Forest or the architecture of autoencoders) or learn how banks counter fraud adaptation, let me know and I'll give you a more detailed answer!