Automation of testing of paid services on iOS

[Tomcat]

For those interested in the topic of automation on iOS, I have two news - good and bad. Good: in an iOS application for paid services, only one integration point is used - in-app purchases (in-app purchases). The bad: Apple doesn't provide any tools to automate purchase testing.

In this article, I invite you to join me in searching for a universal automation method beyond the good and evil of Apple. The article will be useful to anyone who integrates third-party services into their applications that are a “black box”: advertising, streaming, location management, etc. Typically, such integrations are very difficult to test, since there is no possibility of flexible configuration of a third-party service for testing the application.

My name is Victor Koronevich, I am a Senior Test Automation Engineer at Badoo. I have been involved in mobile automation for more than ten years. Together with my colleague Vladimir Solodov, we gave this report at the Heisenbug conference. He also helped me in preparing this text.

In the previous article, we described what methods Badoo uses to test integrations with payment providers, of which we have more than 70. In this material, we will talk in more detail about how we managed to achieve stable and inexpensive automation of testing paid services in an iOS application.

Let's start with a general description of our research:

1. Problem Definition
2. Assignment of tasks
3. Solution #1. Apple Sandbox
4. Solution #2. Method of mocking functions and using a fake object
5. Solution assessment: main risks
6. Result
7. Conclusion

Problem Definition​

Automation should be done when a natural need arises. When did this moment come to us?

The Badoo app has many free features, but paid ones give the user more options. You can get them in two ways: for credits - Badoo’s internal currency - or by purchasing a premium subscription. For a certain number of credits, you can raise your profile to the first place in search results, give a gift to another user, and so on. A premium subscription is valid for a certain period of time and gives several options at once: turn on the invisible mode, see people who have shown sympathy for you, cancel the result of your vote, and others.

These features have been introduced to Badoo gradually. And a couple of years ago, we tested paid services in iOS applications only manually. But as features and new screens appeared, manual testing took more and more time. Requests for changes to the application's operation came from different quarters: from client-side developers, server-side developers, and even the Apple provider itself. For one tester, one testing iteration began to take about eight hours. It became impossible for a developer to receive quick feedback on his branch within 30 minutes, which ultimately could have a negative impact on the competitiveness of the product.

We wanted to receive test results as quickly as possible. And we were faced with a problem: how to inexpensively organize regression testing of paid services in our iOS applications in order to get fast and stable results?

Assignment of tasks​

So, taking into account the specifics of our final product delivery process and the size of the team, we want:

• test any purchases within the client application (one-time payments and subscriptions);
• repeat testing iterations 10–20 times a day;
• get test results for ~150 test scenarios in less than half an hour;
• get rid of noise;
• be able to run tests on a specific branch of the developer’s code regardless of the results of other runs.

Now that we have formulated the problem, it is time to begin our journey into the wonderful world of engineers and their solutions.

Solution #1. Apple Sandbox​

First of all, we started looking for information about organizing automatic testing of paid services in Apple documentation. And they found nothing. Automation support looks very poor. If something appears, then setting up automation with the proposed tools causes difficulties (let's remember at least UIAutomation , as well as the time when the first xcrun simctl utility for iOS Simulator appeared) and you have to look for engineering solutions, including in the open-source segment.

In Apple documentation for testing paid services you can only find Apple Sandbox. It was not clear how to connect this sandbox to automation, but we decided to seriously investigate this solution. What gave us confidence was that Android’s sandbox was stable and by that time we had already successfully written tests on Android. Maybe Apple's sandbox will be just as good?

But when we implemented autotests using this sandbox, we had a blast. Let's quickly go over the main issues.

1. Pool of test users​

The main limitation for automation was the peculiarities of keeping test users in the pool, which should ensure the independence of running autotests.

To run just one subscription purchase autotest, we need:

1. take a new user to authorize in the sandbox;
2. change the current linked Apple ID on the simulator;
3. Log in to the Badoo application as a Badoo user;
4. go to the subscription purchase screen and select a product;
5. confirm the purchase and log in via Apple ID;
6. make sure that the purchase was successful;
7. send Badoo user for cleaning;
8. clear the sandbox user of subscriptions.

If you try to immediately use the same user in the next test, it will be impossible to buy a second subscription. You need to wait until the first subscription expires, or unsubscribe in the settings. As we said in the first article, in the sandbox there is a certain subscription period. If you buy a monthly subscription, you will have to wait five minutes for it to close automatically. The unsubscribe process itself is also slow.

Accordingly, for a new run of the same test, we will need to either wait until the subscription ends, or take another, “clean” user. If we want to run two tests simultaneously independently of each other, then we need to have at least two sandbox users in the pool. Thus, to run 100 autotests in parallel in 100 threads, we need 100 different users.

Now let's imagine that we are running autotests on two agents, each of which can run them in 100 threads. In this case, we need at least 200 users!

2. “Bad” notifications​

Well, okay, what the hell is not joking! We organized a pool of users and started watching how the tests ran. They fell along the way, but most of them for new reasons unknown to us. We started to look into it and realized that when authorizing, confirming a purchase, and working as a user in the sandbox, the App Store sends alerts: for example, it asks you to reenter your name and password, confirm authorization by clicking on the “OK” button, and displays information about an internal error with the “OK” button. . Sometimes they appear, sometimes they don't. And if they appear, they are always in a different order.

How is it possible that a suspicious error is simply ignored in the autotest? And if a real error arrives, what should you do? This area automatically became a “blind spot” for us, and we had to write special handlers for all possible alerts that could arrive from the App Store.

All this made the tests very slow:

• alerts could arrive at different steps of the test scenario, destroying the main idea of the test - Predictable Test Scenario; we had to add an error handler that would wait for a possible series of known ignored alerts to appear;
• sometimes new variations of alerts arrived or other errors occurred, so we were forced to restart failed tests; this increased the running time of all tests.

3. Was there a test?​

So, users in the pool are blocked, then cleared within n minutes. We run tests with 120 threads, and there are already quite a few users in the pool, but this is not enough. We made our user management system, made an alert handler - and then THIS happened. The sandbox became unavailable for a couple of days for any test user.

Nobody expected this. And this was the last straw in the cup of our patience, which finally killed the love for the Apple sandbox and forced us to take the path beyond good and evil. We realized that we didn't need this kind of automation and that we didn't want to suffer any more with this dangerous decision.

Solution #2. Method of mocking functions and using a fake object​

So, we've had enough problems with automation in the Apple sandbox. But don't think that everything is completely bad in the mobile world. On Android, the sandbox is much more stable - you can run autotests there.

Let's try to find another solution for iOS. But how to search? Where to look? Let's look into the history of software testing and development: what happened before the crazy world of Apple? what do people who have written a bunch of books and earned authority in the world of automation and software development say?

I immediately remembered the work “xUnit Test Patterns: Refactoring Test Code”, written by Gerard Meszaros (reviewed by Martin Fowler), - in my opinion, one of the best books for any tester who knows at least one high-level programming language and wants to engage in automation. A couple of chapters in this book dedicated to testing the SUT in isolation from other application components, which are a “black box” for us, can help us.

1. Introduction to mocks and fakes​

It should be noted that in the world of automated testing there is no generally accepted boundary between the concepts Test Doubles, Test Stub, Test Spy, Mock Object, Fake Object, Dummy Object. You should always take into account the author's terminology. We need only two concepts from the big world of Test Doubles: mock functions and fake objects. What is this? And why do we need this? Let us give a brief definition of these concepts so that we do not have any disagreements.

Let's say we have an application and a component embedded in it, which is a “black box” for us. Inside the application, we can call functions by accessing a given component and receive the results of those functions. Depending on the result obtained, our application reacts in a specific way. Sometimes the result of executing a function can be an entire entity with a bunch of fields reflecting real user data.

Let's call substituting a function for any other that will return the desired result a function mock, or simply a mock. These functions may have the same signature, but they are two different functions.

And the replacement of an entity obtained as a result of executing a function with a fake entity (containing the necessary data in the fields, and sometimes even corrupted data) will be called the injection of a fake object. You can read more about this in the book I mentioned above, or in any other compendium on software testing and development.

To finish with this, let's highlight some features of using mock functions and fake objects:

1. In order to mock functions, you need to access the source code and know how the application works with the component from the inside at the developer level.
2. In order to implement a fake object, you need to know the structure of the real object.
3. Using a mock function allows you to flexibly customize how the application works with the component.
4. Using a fake object allows you to assign any properties to an entity.

The mock and fake object method is ideal for isolating the operation of a component within an application. Let's see how we can apply this method to solve our problem, where the App Store will be the component. Due to the peculiarities of using this method, we first need to turn to studying the nature of how our application works with the component, and then to the technical implementation in order to make specific mocks and a fake object.

2. How the actual purchase occurs​

Before we begin to describe the interaction of all parts of the system, let's highlight the main actors:

• application user - any actor who performs actions with the application, it can be a person, or it can be a script that carries out the necessary instructions;
• application (in our case we use the Badoo iOS application installed in the iOS simulator);
• server - an actor that processes requests from the application and sends back responses or asynchronous notifications without a client request (in this case we mean a single abstract Badoo server to simplify the structure);
• The App Store is an actor that is a “black box” for us: we don’t know how it works inside, but we know its public interface for processing in-app purchases (StoreKit framework), and also knows how to check data on the Apple server.

Let's see how the purchase happens. The whole process can be seen in the diagram:

Figure 1. Payment scheme in the App Store

Let us describe step by step the main actions of the actors.

1. The starting point is the state of all actors before opening the screen with a list of products.

What is this screen and how did we get to it?

Let's say a user found an interesting person, opened his profile, wrote one message and wanted to send a gift. Sending a gift is a paid service. The user can scroll the profile to the section for sending gifts or immediately select a gift from the chat.

If the user has chosen a gift and does not have money in his account, he will see a list of different packages of credits (Payments Wizard) for purchase. The starting point in our example is the gift list. In the diagram, we can consider any screen to be such a point before displaying a list of products for purchasing credits or subscriptions.

2. Opening a list of products.

We're at a starting point, like a gift list. The user selects one of the gifts in the application. The application makes a request to our server to get a list of possible Product IDs of credit packages (100, 550, 2000, 5000). The server returns this list to the application.

Next, the application sends the resulting list of Product IDs to the App Store actor (the StoreKit iOS system framework that goes to the Apple server) for verification. It returns a list of verified products - and as a result, the application shows the user a final list of credit packages with icons and prices.

3. Product selection and receipt generation.

The user selects a paid product. The App Store requires proof of purchase and authorization via Apple ID. After successful user authorization, control is transferred to the application. The application waits for the receipt to be generated inside its own package. At this time, the user sees the sun, which blocks the screen. The fact that a receipt has been generated can be understood using the appStoreReceiptURL method of the Bundle class. After the receipt is generated by the App Store, the application selects the receipt from its package and sends a request with the receipt and user data to the Badoo server.

4. Checking the receipt on the Badoo server.

Once the Badoo server receives the receipt and user data, it sends it back to the Apple server side to perform the first verification cycle. This is one of Apple's recommendations. Then, in this first check cycle, the server receives information about the current state of the subscription.

5. Sending a push notification from the server.

The Badoo server processes the received information again after verification by Apple and sends a response to the application along with a push notification.

6. Push notification in the application.

If this was a purchase of credits, then the user’s balance in the application will immediately change and he will see the sent gift in the chat. If this was a subscription purchase, then the user must wait for the final push notification that the subscription has been activated.

3. Define dependencies and test outline​

For further discussion, we will introduce two more concepts - external dependence and testing loop.

External dependence​

By external dependencies we will understand any interaction with a component, which is a “black box” for us. In this case, such a component is the App Store in the form of a system iOS framework (StoreKit), with which our iOS application works, and an Apple server, where verification requests go.

Managing these dependencies in real conditions is impossible; the application is forced to respond to output signals from the “black box” (see Fig. 2).

We have three external dependencies:

1. Reviewing StoreKit products.
2. Receiving and replacing a purchase receipt.
3. Verification is waiting on the Badoo servers.

Figure 2. External dependencies

Test circuit​

The testing loop is the sections of the path that we will go through and check during the testing process.

Figure 3. Testing loop

The goal of our work on eliminating dependencies is to build a testing loop that would be as close as possible to the real path and would allow us to eliminate all external dependencies and transfer control to our side.

Let's consider each dependence sequentially.

4. Dependency Isolation: Technical Implementation​

In our company, to implement payments, we adopted the PPP concept, which is based on the Payment Provider interface. This is the main interface for interacting with the App Store actor (StoreKit) inside our application, which has two main methods:

1. prepare - a method that is responsible for checking products;
2. makePayment is a method that handles in-app purchases.

All payments on iOS have been refactored in accordance with this concept, resulting in a simple and convenient Mock Payment Provider class. This is the main interface for interacting with a convenient copy of the StoreKit behavior inside our application. What does "convenient copy" mean? This provider has mocks of the prepare and makePayment methods, which do what we want. Let's look at the example of pieces of code to see how we managed to integrate mocks.

Addiction #1. Reviewing StoreKit Products​

To check the list of products, the prepare function is used, which returns a list of checked products. We can use a mock in which we disable the verification and return the incoming list of products as fully verified. This way the dependency will be eliminated.

Figure 4. Scheme for eliminating the first dependency

At the very top of the architecture in our application is the Payment Provider. It reflects the interface of a possible provider in the application. The mock implementation code can be found in the Mock Payment Provider class.

public class MockPaymentProvider: PaymentProvider {
   public static var receipt: String?
   public static var storeKitTransactionID: String?

   public func prepare(products: [BMProduct]) -> [BMProduct] {
      return products
   }
   ...
}

Listing 1. Client verification mockup

In the Mock Payment Provider we can see the implementation of the prepare method. The magic of the mock turns out to be very simple: the method skips checking the products on the StoreKit side and simply returns the incoming list of products. A real implementation of prepare looks like this:

public func prepare(products: [BMProduct]) -> [BMProduct] {
   let validatedProducts = self.productsSource.validate(products: products)
   return validatedProducts
}

Listing 2. Real Store Payment Provider

Dependency No. 2. Receiving and replacing a purchase receipt​

With the second dependency, the situation is a little more complicated: we need to first remove authorization so as not to hold a pool of user accounts, and then somehow get the check itself. We can simply delete the authorization form:

Figure 5. Removing the authorization form when making a payment

With a check, everything is not so simple. Many questions arise:

1. How can I get a receipt for the product I need in advance?
2. If we do receive a check, then when and how to place it inside the application?

Here the “User” actor has a new role - QA. When we run a test, we can not only click on interface buttons, but also call test framework API methods (methods that simulate user actions) and REST API services (methods that can do magic from the Badoo internal service). At Badoo we use a very powerful QA API tool (you can see all its capabilities at the link:

). He is the one who helps us in testing and gives us a receipt for the desired product on the Badoo server side. The Badoo server is the best place to generate receipts: there is encryption and decryption of the receipt, so the server knows everything about this data structure.

Once we have received a fake check, we can issue it through a backdoor on the application side. Next, the application will send the fake receipt along with user data to our server.

Figure 6. Scheme for receiving a check

How did this become technically possible?

1. To issue a fake check in the application, we were able to use a backdoor that saved the fake check in the receipt field of the MockPaymentProvider:

#if BUILD_FOR_AUTOMATION

@objc
extension BadooAppDelegate {
    
   @objc
   func setMockPurchaseReceipt(_ receipt: String?) {
      PaymentProvidersFactory.useMockPaymentProviderForITunesPayments = true
      MockPaymentProvider.receipt = receipt
   }
   ...
}

#endif

Listing 3. Backdoor of issuing a fake receipt

2. The application was able to take our receipt thanks to the MockPaymentProvider, in which we used the makePayment mock and the saved receipt in MockPaymentProvider.receipt:

public class MockPaymentProvider: PaymentProvider {
   ...
   public func makePayment(_ transaction: BPDPaymentTransactionContext) {
      ...
      if let receiptData = MockPaymentProvider.receipt?.data(using: .utf8) {
         let request = BPDPurchaseReceiptRequest(...)
         self.networkService.send(request, completion: { [weak self] (_) in
            guard let sSelf = self else { return }
            if let receipt = request.responsePayload() {
               sSelf.delegate?.paymentProvider(sSelf, didReceiveReceipt: receipt)
            }
         })
      } else {
         self.delegate?.paymentProvider(self, didFailTransaction: transaction)
      }
   }
}

Listing 4. Calling a mock purchase processing with a fake receipt[z

3. Receiving a fake receipt

To receive a fake receipt, we used a method on the server (see Listing 5). It takes the default array with data for generating receipt data and adds to it the data that is needed for a specific product.

$new_receipt_model = array_replace_recursive(
    //create an array with the default receipt scheme
    $this->getDefaultModel(),
    //define additional parameters based on the saved data
    //necessary if we are processing a previously used payment
    $this->enrichModelUsingSubscription($nr),
    //define additional parameters depending on the desired state
    $this->enrichModelUsingInput($input)
);
//create a signature
$new_receipt = $this->signReceipt(
    json_encode($new_receipt_model, true),
    $new_receipt_model
)

Listing 5. Server part of receipt generation

To replicate the structure of a real receipt, a custom receipt sent by the application must be encrypted using a certificate. We use our production certificate instead of the Apple certificate.

function signReceipt($receipt, $response) {
    //add headers and base64 encode check
    $receipt = 'Subject: ' . base64_encode(json_encode($response)) . PHP_EOL.   PHP_EOL. $receipt;
    file_put_contents($receipt_file, $receipt);
    ...
    //sign the check with our certificate
    $sign_result = openssl_pkcs7_sign($receipt_file, $signed_receipt_file, 'file://'.$path_cert, 'file://'.$path_key, [], PKCS7_BINARY);
    ...
    //add headers
    $signed_content_with_headers = file_get_contents($signed_receipt_file);
    list($headers, $signed_content) = explode(PHP_EOL . PHP_EOL, $signed_content_with_headers);
    
    //return the check    
    return str_replace(["\r\n", "\r", "\n"], '', $signed_content);

Listing 6. Method for signing a check with certificate

4. As a result, in the test we get:

And(/I am generating a new payment receipt for the purchase of "((\d+) credits|subscriptions for (\d+) months?/) do |service_type|
   # service type processing
   service_details = parse_options(service_type)
   # QA API call (internal Badoo service)
   receipt = QaApi::Billing.order_get_app_store_receipt(service_details)
   # call backdoor
   Backdoors.set_fake_receipt(receipt)
end

Listing 7. Gherkin test step for the Cucumber framework

Dependency No. 3. Checking a receipt on the Badoo server​

To remove the third dependency, you need to get rid of check verification on the server. It is important to remember here that verification is done in two stages. The first step is to verify the authenticity of the check based on signatures and certificates. On the second, the check is sent to the App Store. If validation is successful at this stage, we will receive a decrypted check that can be processed.

Figure 7. Removing server verification

First, the server performs initial verification of the receipt in the verifyReceiptByCert method of the parent class. Here the signature is verified using the App Store certificate. In the case of a fake check, this verification will fail because it is signed by our certificate, and we will call the verifyReceiptByLocalCert method to verify with a local certificate. In this method, we will try to decrypt a receipt using a local certificate, and if successful, we will place the decryption result in the internal field local_receipt of the child class (addLocallyVerifiedReceipt method).

class EngineTest extends Engine 
   function verifyReceiptByCert($receipt) 
{
      $result = parent::verifyReceiptByCert($receipt);

      if ($result === -1 || empty($result)) {
         $result = $this->verifyReceiptByLocalCert($receipt);

      }

      return $result;

   }

   function verifyReceiptByLocalCert($receipt) {
      $receipt_file = tempnam(sys_get_temp_dir(), 'rcp');
      file_put_contents($receipt_file, base64_decode($receipt));
      $result = openssl_pkcs7_verify($receipt_file, PKCS7_BINARY, '/dev/null', [$DIR]);
      if ($result) {
         $this->addLocallyVerifiedReceipt($receipt, base64_decode($response));
      }
      unlink($receipt_file);
      return $result;
   }

class Engine 
   function verifyReceiptByCert($receipt) {
      $receipt_file = tempnam(sys_get_temp_dir(), 'rcp');
      file_put_contents($receipt_file, base64_decode($receipt));
      $result = openssl_pkcs7_verify($receipt_file, PKCS7_BINARY, '/dev/null', [$DIR]);
      unlink($receipt_file);
      return $result;
   }

Listing 8. Primary verification

During secondary verification (verifyReceipt), we get the value of the local_receipt field of the getLocallyVerifiedReceipt child class. If it is not empty, then we use its value as a verification result.

If the field is empty, then we call secondary verification from the parent class ( parent ::verifyReceipt). There we make a request to the App Store for verification on its side. The result of verification in both cases is a decrypted receipt.

class EngineTest extends Engine
   function verifyReceipt($receipt_encoded, $shared_secret, $env) {
      $response = $this->getLocallyVerifiedReceipt($receipt_encoded);
      if (!empty($response)) {
          return json_decode($response, true);
      }
      return parent::verifyReceipt($receipt_encoded, $shared_secret, $env);
   }

class Engine
   function verifyReceipt($receipt_encoded, $shared_secret, $env) {
      $response = $this->_sendRequest($receipt_encoded, $shared_secret, $env);
      return $response;
   }

Listing 9. Secondary verification

5. Test run video: purchasing credits and subscriptions​

Test No. 1. Purchasing a subscription​

When	I log in to the application as a new user with a photo
I	I am generating a new payment receipt for one month subscription
I	I go to my profile
That	I make sure the subscription is disabled
When	I open the grocery list
I	I buy a one month subscription package
That	I am checking the successful purchase notification
I	I make sure that the subscription has been activated

Test chase video:

Test No. 2. Buying credits and sending a gift​

When	I log in to the application as a new user with a photo
I	I add ten credits to my profile
I	I generate a new payment receipt for 550 credits
I	I am creating a new user Leela
I	Leela voted "Yes" for me
I	I go to People Nearby and open Leela's profile
I	I vote Yes for Leela
That	I'm checking the match page
When	I choose to send a regular gift
That	I check the payment screen with a list of packages
When	I choose to buy 550 credits
That	I am checking the successful purchase notification
I	I make sure Leela received the gift in chat

Test chase video:

Solution assessment: main risks​

Removing external dependencies comes with certain risks.

1. Incorrect configuration.
Since the verification does not happen on our side, we may configure our products incorrectly on the Apple side. To protect against the error, we wrote a separate server unit test that checks that all the products that we run on the Apple side match the products that we have in our config.

2. Borderline cases.
For example, when a payment goes through in full, the user receives a notification that it has been completed, but our application cannot find the check that should be deposited as a result of this payment. The risk is that we ourselves submit the check using a backdoor, and of course we cannot track such a case. To somehow compensate for this risk, we conduct end-to-end checks using a sandbox or real payment after release.

3. Unscrupulous counterfeit or fraud.
After reading this article, you might think that since Badoo uses fake receipts, then we can plant something fake and use the service for free. To prevent such a risk from materializing, we sign everything with our own certificate and limit the use of mocks and fake checks to functional tests that are run only in our development environment.

4. Changing the check format.
This is the most serious risk. Changing the check format is possible when Apple changes something without telling us. We had such a case: when we switched to iOS 11, the format of the check completely changed. We generated a fake receipt on our server and used it in the test. Everything was fine with us: all the fields were in place, everything was great, everything was being processed. But when we moved to the real system, nothing worked. The fields that were significant in the check simply ceased to exist.

How to compensate for this risk? Firstly, we do not exclude end-to-end testing of the sandbox before release and real payment after release. Now we are in an active phase with a project to check notifications, when we are trying to classify all the checks that we receive from production according to the principle of whether we understand what it is or not. If the answer is no, then we begin to process everything manually, to see what has changed, what is wrong, what needs to be changed in our system.

Risk	Reason	How to compensate
incorrect configuration	deleting a check	unit test on the server
edge cases (check not delivered)	using a backdoor	E2E checks (sandbox and real payment)
dishonest fake, fraud	generation of notifications and the server check	own certificate
change the form of the check	generation of notifications and the server check	checking real notifications and receipts on the product (new project), E2E checks (sandbox and real payment)

Result​

Let's look at the main advantages that we were able to obtain as a result of using the mock and fake object method.

Inexpensive, fast and stable automation of paid services on iOS​

Together with the iOS manual testing team (special thanks to Colin Chan), we were able to write more than 150 automated tests for payments. This is a fairly large amount of coverage for one application area.

Thanks to parallelization, we can get results in just 15–20 minutes on any branch of the iOS client developer or billing server developer. Before automation, manual testing of this area by one person took eight hours.

We can also test the vast majority of test cases thanks to setting up the Mock Payment Provider through mocks in the way we need. Using mocks, we learned how to disable product verification and simulate cases when the verification is partially performed. Thus, cases were opened to us that we could not test in principle before.

Functional regression when developing new features​

Automation worked very well in cases where the developer, while working on a new feature, affected old functionality. We had an example where a developer made a complex feature with caching and ran our autotests. Some of them fell with an error. He saw it and fixed it. Then I restarted the autotests again - and again something crashed. As a result, he made a series of iterations until the moment when everything began to work normally on the application side.

Functional regression when refactoring payments​

Perhaps the most successful and effective automation that is possible happens in the area of code refactoring. In this case, only the internal implementation changes—there is no need to change the autotest code. The UI does not change in any way, and autotests can be run efficiently.

Testing experimental features from Apple: grace period​

A system like this is completely replaceable when you're testing new integrations that haven't yet been implemented in the sandbox. This is how it was with our grace period. Sandbox does not have this functionality. Grace period on Apple is not yet available to everyone. This is an experimental project that Badoo is implementing together with Apple. In order to make a check with a grace period, we needed to add the following piece of JSON code to it:

pending_renewal_info:[
{
      expiration_intent: 2
      grace_period_expires_date: 2019-04-25 15:50:57 Etc/GMT
      auto_renew_product_id: badoo.productId
      original_transaction_id: 560000361869085
      is_in_billing_retry_period: 1
      grace_period_expires_date_pst: 2019-04-25 08:50:57 America/Los_Angeles
      product_id: badoo.productId
      grace_period_expires_date_ms: 1556207457000
      auto_renew_status: 1
}]

Listing 10. Grace period for subscription

We did this very easily in just a few seconds. In our system, we were able to test our reaction to a new feature. Now we are testing this functionality in production.

Testing product quality in a composition of methods​

As a result of our research, we were able to describe a method that allows us to eliminate noise from external dependencies. This helped client developers find bugs in the early stages during feature development.

But don’t think that we were able to test everything using this method. To test everything, it is better to use a composition of methods: testing with a real card in production, testing in a sandbox, the method of mocks and fake objects, unit and integration testing. Please remember to balance the testing pyramid and do not try to solve all problems with one method. This can lead to sad automation in the sandbox, to sad manual testing of all cases with a real map, and many other serious errors exactly in the place where their occurrence is most painful.

Conclusion​

As a result of our research, we received an inexpensive, fast and stable method for testing not only paid services on iOS, but also any components built into the application as a “black box”. Now at Badoo we are implementing this method for testing on Android paid providers (Global Charge, Boku, Centili) that have unstable sandboxes or any other restrictions. We also use the mocking method to test advertising, streaming and geolocation.

It is worth saying that the process of introducing the new method itself was not quick. We had to negotiate with four teams: iOS QA, iOS Dev, Billing QA, Billing Dev. Not everyone wanted to switch to the new method, fearing the risks. Sometimes it was dogmatic adherence: we tested in the sandbox for many years, and the main force that was able to destroy the dogma was the desire of the billing and iOS platform testers to change the situation and get rid of the torment. Later, the developers realized such advantages of this method as accurate diagnostics (we were able to find not sandbox bugs, but bugs of our client or server), flexibility in setting up the component (we were able to easily test negative cases at the integration level) and, of course, a response within 30 minutes on the branch with the code being developed.

To everyone who read to the end, thank you very much. Many thanks to everyone who helped and participated in this project. Special thanks go to these people:

• Pyotr Kolpashchikov is an iOS developer who helped make mocks on the client side and developed the PPP concept;
• Vladimir Solodov - Billing QA, who helped with the QA API for generating fake checks and a verification mock from the billing server;
• Maxim Filatov and Vasily Stepanov - Billing Dev Team, who helped with the billing server code;
• iOS Dev Team - developers who were able to refactor our payments into a new concept, making the use of mocks possible;
• iOS QA Team is an amazing testing team that wrote a ton of automated tests;
• Billing QA Team - testers who helped investigate problems.