We monitor the emergence of new subdomains in real time

[Lord777]

An article about how, by understanding the logic of the program and manipulating requests, you can get confidential information about site users and give yourself administrator rights.

When testing a web resource for penetration, one of the strategies is perimeter hacking. Large companies with a developed web structure may have network nodes that are not actively used or supported - simply put, everyone has forgotten about.

On the one hand, such nodes may be more vulnerable to attacks due to outdated software or simply because they were not intended to work with external users (in production). But at the same time, they can be connected to the local network and using the network pivoting methods, you can continue collecting information and attacking nodes in production through the local network through them.

Inside the article, we will analyze this topic in detail. Enjoy reading!

We search for subdomains and find hidden data.

Finding subdomains is one way to explore the perimeter. There are already three articles devoted to this issue and three tools that use a variety of techniques for identifying subdomains:

• The fastest search for subdomains and monitoring the appearance of new subdomains (works on Windows too!)
• Search subdomains (subdomains) without brute force
• Finding subdomains and building graphs of network structure with Amass

Some of the tools discussed in these articles use Certificate Transparency log analysis. In addition to finding long-standing subdomains, Certificate Transparency can be used to detect subdomains as they appear in near real-time! This is what this article is about - we will look at how to learn about subdomains as soon as they appear, what tools are there for this, and in what cases this technique will work.

Let's start by getting to know Certificate Transparency. The next section contains information from Wikipedia.

What is Certificate Transparency
Certificate Transparency (CT) is an Internet security standard and open source platform for monitoring and auditing digital certificates. The standard creates a public log system that aims to ultimately record all certificates issued by publicly trusted CAs, thereby effectively identifying certificates that have been issued in error or maliciously.

Certificate transparency is described in Experimental RFC 6962.

From 2021, certificate transparency is mandatory for all TLS certificates, but not for other types of certificates.

One problem with digital certificate management is that browser vendors take a long time to detect, report, and revoke fake certificates. Certificate transparency helps with this problem by making it impossible to issue a certificate for a domain without the knowledge of the domain owner.

Certificate Transparency does not require side-channel communication for certificate validation, as some competing technologies, such as Online Certificate Status Protocol (OCSP) and convergence, do. Certificate transparency also works without the need to trust a third party.

Certificate Transparency Logs
Certificate transparency depends on the certificate transparency logs being checked. The magazine adds new certificates to the ever-growing Merkle hash tree. To be considered to be working properly, the journal must:

• Ensure that every certificate or pre-certificate submitted has a valid signature chain leading to a trusted root CA certificate.
• Refuse to publish certificates without this valid signature chain.
• Save the entire verification chain from the newly accepted certificate to the root certificate.
• Provide this chain for audit upon request.

The journal can accept certificates that are not yet fully valid and certificates that have expired.

Certificate Transparency Monitors
Monitors act as clients for log servers. Monitors check the logs to make sure they are working correctly. Inconsistency is used to prove that the log is behaving incorrectly, and the signatures in the log data structure (Merkle tree) prevent the log from denying this incorrect behavior.

Certificate Transparency Auditors
Auditors also act as clients of the log servers. Certificate transparency auditors use partial log information to compare the log with other partial information they have.

Certificate Transparency in plain language
1. All issued TLS certificates must be recorded in the Certificate Transparency log. If you do not log the certificate, it will not be considered valid even if all other conditions are met.

2. The journal of issued TLS certificates is public for everyone.

It follows from this that as soon as I issue the certificate for the project2.hackware.ru subdomain,

information about this certificate immediately falls into a publicly accessible journal

and information about the project2.hackware.ru subdomain becomes publicly available.
As a result, as soon as I received a certificate for project2.hackware.ru, then even if I did not manage to configure anything at project2.hackware.ru, anyone who wants to know about this subdomain is already aware of this subdomain.

This is the whole essence of the technique considered in this article.

Limitations of this technique
Obviously, for subdomains operating over the HTTP protocol (that is, for which a TLS certificate is not issued at all), this method of searching for subdomains is inapplicable. It is applicable only for subdomains using the HTTPS protocol.

Also, if separate certificates are not registered for subdomains, and the work is done with a certificate with a wildcard character (for example, * .hackware.com), then this technique will also fail.

Why do companies issue certificates for subdomains instead of using a wildcard?
By the way, the question may arise, why issue separate certificates for each subdomain if you can issue one certificate with a wildcard character (for example, * .hackware.com) that will cover all subdomains at once?

You can see in the properties of the certificate of the HackWare.ru website that it is the certificate with a wildcard character that is used here.

Why, for example, are the system administrators of yandex.ru not so “smart” and “didn’t think of it” to issue one * .yandex.ru certificate and use it for everything?

The fact is that in large organizations it is difficult to guarantee the absence of leaks, and if the private key from the developer responsible for the secondary direction of the third-rate service for some domain eda.kino.dostavka.yandex.ru leaks, then this is one thing, and if the private key for * .yandex.ru, including subdomains passport.yandex.ru, mail.yandex.ru and others, then this is much more serious. A leak of the private key for the * .yandex.ru certificate will mean the ability to perform man-in-the-middle attacks for the main domain and all subdomains, while the web browser will show the connection as secure (green lock)!

For this reason, organizations always try to issue separate certificates for subdomains. Information about these subdomains is recorded in a public log. Which means for us that we can find out about them exactly at the moment when the certificate is issued (the subdomain is created).

Where to View Certificate Transparency Logs
The official website of Certificate Transparency https://certificate.transparency.dev/ has a list of monitoring companies as well as a list of those providing access to the journals.

A couple of addresses where you can enter a domain name and get a complete list of certificates ever issued for it:

• https://crt.sh/
• https://transparencyreport.google.com/https/certificates

Tool for monitoring new subdomains (issuing new TLS certificates)
The list of issued certificates for a DOMAIN can be obtained by parsing a link of the form

https://crt.sh/?q=DOMAIN

. If you prefer to work with the JSON format, then use the link like https://crt.sh/?q=DOMAIN&output=json.

The algorithm is very simple:

1. We get a list of subdomains from the link https://crt.sh/?q=DOMA and save them to a file
2. From time to time, we get a list of subdomains again at the link https://crt.sh/?q=DOMAIN and compare them with the existing ones
3. If new subdomains are found, then display them on the screen
4. If there are several domains, then we repeat this procedure for each of them.

This is a simple task and to automate it, you can write your own Bash script in several lines. But I was lucky to have a ready-made Sublert tool. It simply does not work in its original form, I had to spend quite a lot of time on it, but still I managed to figure it out.

Installing Sublert on Kali Linux

sudo apt install git python3 python3-termcolor python3-tld python3-requests python3-dnspython python3-psycopg2 python3-tld
git clone https://github.com/yassineaboukir/sublert && cd sublert
If an error occurs
Traceback (most recent call last):
  File "/home/mial/bin/sublert/sublert.py", line 16, in <module>
    from tld.utils import update_tld_names
ImportError: cannot import name 'update_tld_names' from 'tld.utils' (/usr/lib/python3/dist-packages/tld/utils.py)

Then run the following commands:

sudo apt install python3-pip
sudo pip3 install tld==0.9.6

Installing Sublert in BlackArch

sudo pacman -S sublert

Sublert patch

In its original form, Sublert sends new subdomains through Slack when it finds new subdomains. I don’t know what it is and I don’t like the idea of using third-party services, so now we will change the code of the program so that it outputs the results obtained directly to standard output. If you are satisfied with Slack, then refer to the official instructions: https://medium.com/@yassineaboukir/...un-and-profit-release-of-sublert-634cfc5d7708

Open config.py file - for Kali Linux users this file is in the same folder where they downloaded Sublert, for BlackArch users this file is located at /usr/share/sublert/config.py :

sudo gedit /usr/share/sublert/sublert.py

In this file find the snippet

url = "https://" + url.replace('+ ', '')

rev_url.append(get_fld(url))

data = "{}:new: {}".format(at_channel(), url)

slack(data)

and replace it all with

print (url)

Next problem: Sublert uses two ways to get data from the log:

1) connecting to the database

2) if the first method fails, then an address like https://crt.sh/?q=DOMAIN&output=json is simply parsed

So, when connecting to the database, the program DOES NOT SEE new subdomains. I don't know what the problem is, but without the next change, the program will work, but even with the appearance of subdomains, it will not show you anything.

So in the sublert.py file find and completely remove the following snippet:

try:

#connecting to crt.sh postgres database to retrieve subdomains.

    unique_domains = set()

    domain = domain.replace('%25.', '')

    conn = psycopg2.connect("dbname={0} user={1} host={2}".format(DB_NAME, DB_USER, DB_HOST))

    conn.autocommit = True

    cursor = conn.cursor()

    cursor.execute("SELECT ci.NAME_VALUE NAME_VALUE FROM certificate_identity ci WHERE ci.NAME_TYPE = 'dNSName' AND reverse(lower(ci.NAME_VALUE)) LIKE reverse(lower('%{}'));".format(domain))

    for result in cursor.fetchall():

        matches = re.findall(r"\'(.+?)\'", str(result))

        for subdomain in matches:

            try:

                if get_fld("https://" + subdomain) == domain:

                    unique_domains.add(subdomain.lower())

            except: pass

    return sorted(unique_domains)

except:

In Kali Linux, to start, specify the path to the sublert.py file :

python3 sublert.py -u DOMAIN

Run in BlackArch like this:

sudo sublert -u DOMAIN

To add new domains to monitoring use the -u option, for example:

python3 sublert.py -u yandex.ru
sudo sublert -u yandex.ru

You will be asked if you want to view the list of found subdomains (enter "Y" for "Yes" or "N" for "No"):

If you do not want to be prompted to list subdomains each time, then use the -q false option, for example:

sudo sublert -u wise.com -q false
sudo sublert -u hackware.ru -q false
sudo sublert -u paysend.com -q false

To display a list of all domains to monitor the issuance of new certificates, run the command

sublert -a

To check the issuance of new certificates for subdomains, run the command:

sudo sublert

If no new subdomains are found, it will show:
[!] Done.

If new subdomains are found, the following line will be displayed (although nothing is actually sent to Slack):

[!] Exporting the result to Slack. Please don't interrupt!

The found subdomains will also be shown.

As you can see, the project2reloaded.hackware.ru subdomain was found, for which I created a subdomain and received a certificate just before running the script.

You can manually run the script, or set up automatic launch using Cron and sending notifications to you by email or in another way.

On the card page of the Sublert program https://kali.tools/?p=6651 you will find a complete list of options. Of these, note the following:

-u TARGET, --url TARGET
                       Domain for monitoring. For example: yahoo.com (default: None)

-q true | false, --question true | false
                       Prompts for user input (default: True)

-d DOMAIN, --delete DOMAIN
                       Domain to remove from the monitoring list. For example: yahoo.com (default: None)

-t THREADS, --threads THREADS
                       The number of threads used concurrently. Default: 10

-a, --list
                       Show all monitored domains.

-m, --reset
                       Reset everything to its initial state.

That's all!