Payment service in the bank

[Man]

Hiello. I would like to tell you about the architecture of the "Payment Service" - a system for orchestrating and processing payments. The architecture was developed as part of a project to modernize the bank's existing solution. The work began at the end of 2022 and was successfully completed a year later. The system is now in commercial operation.

A few words about the bank to understand its scale. The bank works primarily with individuals, offering them classic banking products, as well as investment services. The client base is over 15 million clients.

Subject of automation​

Among the services that banks provide to clients, a significant role is given to the acceptance and processing of payments and transfers - the transfer of clients' money to the accounts of both individuals and legal entities, as well as to service providers (for example, mobile operators and housing and communal services).

When a client makes a payment, a financial transaction occurs, as a result of which the payer's account balance decreases, and the recipient's balance increases. This is recorded by the banks' master systems, which keep track of the balances of funds in clients' accounts.

Master systems manage the so-called accounting books (or "general ledgers"), where the movement of funds in bank accounts is recorded, for simplicity - "income" and "expense". Thus, a completed financial transaction of a client (ultimately) changes the state of the general ledger.

There is a business process behind payments and transfers, the purpose of which is to process the client's payment order through all internal checks and procedures, and then record the financial result in the general ledger. The payment service is responsible for automating this process.

At the same time, special requirements are imposed on the accuracy and coordination of the process execution - the money must be received by the recipient on time and in the right amount, there must be no confusion. The bank is also interested in promptly detecting and stopping illegal and dubious transactions.

It is important to note that in this case, the automation work did not affect the processing of card transactions in retail, i.e. acquiring. Acquiring is implemented by separate tools and a set of technologies.

Also, the payment service is not responsible for storing and recording information on accounts and balances, being only an orchestrator. Automated banking systems - ABS act as master systems.

Payment process​

The payment process may differ from bank to bank, from payment type to payment type, and contain a different number of steps. For example, in interbank transfers, the stages of crediting and debiting funds are arranged differently and may contain additional checks.

However, the soul of the process can be presented as follows:

• Incoming payment orders from clients undergo verification and preliminary processing
• The money is debited from the account or temporarily blocked on the sender's account depending on the type of payment.
• Funds are credited to the recipient's account
• The payment or transfer is finalized and accounting entries are generated in the general ledger

Some properties of the process​

It is important to note some properties of our process that influenced architectural decisions:

• The process is not interactive. Direct interaction between the client and the payment service during the processing is not provided for. The client forms an order in the service channel, after which he receives the result of the operation - a change in the balance on his account and a notification from the bank
• The business process does not contain cycles and returns to previously completed stages. It develops in one direction, but can have branches and conditional transitions
• The process is within one transaction and can be considered successfully completed only when it is completed in full. It is important to emphasize that "in full" does not always mean that funds are credited to the recipient (for example, when transferring to foreign banks)
• The process is carried out quickly – a few seconds during the operating day (when the bank is open)

Motivation​

At the time the decision was made to upgrade, a monolithic application on the IBM stack in the WebSphere Application Server (WAS) environment running the AIX operating system was responsible for automating payment processes.

The mobile and internet banking components were connected to it using SOAP, passing an XML document with the client’s payment order for processing.

Oracle was used as the DBMS. The data model consisted of a set of various assignment details, statuses and other business data and technical information required during the processing.

The main reasons for the deep reworking of the system were:

• Insufficient performance. While the system handled online payment processing, it had difficulties with batch operations (e.g. mass salary transfers). Batch processing is done within strict time limits, which was not always possible to ensure
• Outdated technology stack. In the current situation with the availability of IBM products in the Russian Federation, this factor is of particular importance
• Low scalability. The ability to increase the system's performance was almost exhausted. In addition, the bank expected the introduction of new payment categories, and therefore a significant number of improvements on an outdated solution. The associated low T2M indicator is included in the general scalability problem, only not in terms of load, but in terms of functionality

Architectural principles​

The requirements for the architecture of the target solution were formed in such a way as, on the one hand, to avoid obtaining a product that becomes obsolete immediately after launching into industrial operation, and on the other hand, to avoid architectural redundancy.

The architecture is based on the following principles:

• Modularity. We abandoned the "monolithic" approach in the target solution. On the other hand, we also did not pursue the goal of "switching to microservices" in their strict canonical representation
• Loose coupling. Modules should know as little as possible about their environment and other modules, preferably nothing. In other words, each module is a standalone application with the ability to function in isolation from its neighbors.
• Independent scaling. The ability to increase the speed of execution of exactly those parts of the process that need to be accelerated right now. In other words, the ability to increase the performance of the most popular modules independently of the rest
• Independent modifications. Routine modifications are localized only in those modules that are directly responsible for the corresponding sections of the process.
• Horizontal scaling. Support for the ability to increase the solution's performance by adding additional modules (copies) and including them in parallel processing
• Failure isolation. A complete failure of a module and all its instances should not lead to a failure of the entire system or to the stoppage of all payments. For example, a failure of a pre-processing module should not affect payments that have already passed this stage, and so on.
• No SPOF and bottlenecks in the form of a common database or any single orchestrating link
• Minimal technological diversity. The diversity of technologies within the overall solution should be kept to a minimum. The technologies themselves should not only be mature, but also well known to the bank's development and operations services.

It is important to note that the payment service as such is not autonomous. It depends on the bank's master systems (ABS), where the general ledger is directly located, accounts and balances are maintained. Therefore, the qualities of the new system will be somewhat limited from above by the capabilities of the ABS. This limitation was adopted because the problems of the payment service affected the efficiency of the process much more than the behavior of the ABS. And the ABS themselves have a significantly greater margin of safety.

Restrictions​

A number of important restrictions were also adopted during the design:

• Prohibition of changes to the business process, including the business logic of some operations. Payment processes were transferred to the new solution "as is" even when there was an opportunity for optimization
• Payments of different categories should move to the new system sequentially, not simultaneously. It was necessary to take into account the temporary coexistence and joint work of the two systems - the old and the new

Target architecture​

Infrastructure​

All components of the new system are hosted on x86 servers. The main modules were developed on the Java and Spring Boot stack, and deployed in the OpenShift cluster (the bank had one at the time of deployment). The cluster's virtual machines are distributed across the infrastructure in such a way as to minimize the impact of hardware failures on the system's performance.

API​

The new payment service has its own API, like its predecessor. The API is designed to enter new orders and allows you to receive information on previously sent payments, as well as manipulate them manually if necessary. This API is implemented by two independent modules.

The "Payment Acceptance Module" accepts an order from a third-party system, such as a DBO (mobile or Internet bank), and performs primary control. Then the system receives a quick response within one synchronous session. If the checks are successful, an asynchronous processing process begins, otherwise an error is returned.

The "Consolidation Module" handles information requests, allows you to extract information on demand payments and historical data. This module also sends notifications and events for some important stages of the process.

The chosen approach allows us to separate the extremely intensive flow of information requests from the flow of new orders, which is more sensitive to the level of availability, but less intensive.

DBMS​

The architecture of the new service uses the PostgreSQL DBMS. This DB does not participate in the payment processing process in any way, being only a showcase for the consolidation module. If the DB is unavailable, this module also does not work, but the acceptance of new payments and the processing of old ones does not stop.

Direct use of this DB in modules is prohibited. The system architecture does not provide for centralized storage of any data required for processing in a common DB (Kafka is not included).

Integration​

Kafka is used as a data exchange environment between modules and other components. Kafka fits well with the asynchronous nature of the process and helps to achieve weak coupling, support the parallel payment processing mode by several instances of modules.

"Topic" (Kaka Topic) is the main configuration unit in the integration architecture of the new system. Topic allows semantic distribution of data flows between modules by separate logical "channels" within the boundaries of one Kafka cluster.

Orchestration​

The architecture of the new payment service was decided to rely on the "choreography" pattern without a central orchestrator. The pattern meets the principles of independent scaling, failure separation, and the absence of SPOF. In addition, choreography better meets the properties of the process.

Modules​

The system modules collectively implement all business processes for processing payments and transfers, and individually – specific stages or subprocesses. Each module consists of one or more functions, where each function implements a subprocess of the general payment process.

The functions rely on Kafka Streams for streaming payment processing.

Module architecture​

The modules exchange some object between themselves – a document in JSON format. The object is a container of a given format and contains:

• Information about the channel of receipt (mobile bank and others)
• Customer payment order details such as account numbers, customer ID, amount
• Technical information
• Business status

Processing of a document received from the receiving module is performed sequentially, in asynchronous mode. Each module is capable of processing documents only with a certain status or with a limited set of statuses, from which a transition is possible in this module. Thus, the architecture can develop additively - by developing and including in the process a new module that will be familiar with the new status.

The following strategy was used to determine the overall composition and functional load of the modules:

• The process steps that are always and under any circumstances executed one after another and together (“linked”) remain in one module
• If a process branches into alternative chains, the split of modules occurs at the branching point. For example, if after the verification step a payment can take two alternative routes depending on the type of payment, then what follows after this step will be split.
• Modules must perform a specific, explicit, job and change the payment status. For example, if a module accepts a payment with the status "checks passed", it is assumed that the output status will be different

The general logic of each module can be described as follows:

1. Receiving a document from an incoming Kafka topic
2. Execution of the module function, exchange with external systems
3. Change of business status
4. Enriching a document with new data
5. Sending an updated document to an outgoing topic

Module subscriptions​

Each module subscribes to at least one input topic. Reading a new document from an input topic is a trigger that causes the module's functions to run. The final payment status is a superposition of the functions of all modules affected during its execution.

The input topic of one module is also the output topic for one or more neighboring modules.

Additionally, all modules without exception work with a special NOTIFY topic and some with optional service topics for synchronization and distributed locks.

Composite modules​

If the module is composite and contains several functions, then the required one is selected by the selector based on optional flags in the entry header. All flags are global for the system, so their number is limited.

Composite modules are used in cases where, for example, there is a trivial alternative to the main function of the module (dummy), which does not make practical sense to separate into a separate module, and the choice cannot be made "without a hint". For example, when you need to skip a document without changing anything in it, but still switch the status. Optional flags in the header act as a "hint", prompting the module to change its main behavior.

Routing​

The behavior of modules resembles a finite state machine model. The data transfer route from module to module is determined by a matrix transition function (the "routing table") defined at the system-wide level. The table links business statuses and Kafka topics, allowing modules to determine which output topic to send a document to after processing.

Each topic has a subscriber module for which this topic is an input (more precisely, all functionally identical instances of the module in a horizontally scalable environment). Also, each business status can be processed by one and only one specific module. This (implicitly) establishes a correspondence between the status and the module.

Accordingly, a significant part of the improvements comes down to only changing the routing table. Except for the situation with adding a new status, where a new module may also be required.

At the current stage of service implementation, the routing table is contained in the configuration common to all modules.

Consolidation​

The system architecture does not provide for a centralized database for storing information on all payments. Therefore, in order to ensure the operation of the information API and to provide information on any payment at the moment, an approach based on the NOTIFY topic and the consolidation module is used.

Each module (except the consolidation module itself) must send a copy of the document to NOTIFY. The consolidation module reads this topic, reduces the document flow from all modules, highlighting the latest current status for each payment, and updates its display. The processing history is also saved.

The contents of the showcases are used to show the client the results of payments and transfers in the mobile application and other channels, for the needs of the operational block employees and technical support, as well as when manual payment processing is necessary.

To be continued.

Source