The developers of the collaborative development platform SourceHut published a report on an incident, as a result of which the service was disrupted for 7 days due to a long-term DDoS attack, for which the project’s infrastructure was not ready. Basic services were restored on the third day, but some services were unavailable from January 10 to January 17. At the initial stage of the attack, the developers did not have time to react and try to counter the problem on the side of their servers, so all traffic to the SourceHut servers was completely blocked on the side of the upstream provider.
To ensure the operation of SourceHut, servers were used in three data centers. The first was used for production configuration, the second was used for backup, and the third was used for experiments on migrating infrastructure to a more scalable and fault-tolerant implementation of the service (the next generation version of SourceHut was being developed).
It took about 9 hours to resolve the issue of providing access to their servers in the main data center after blocking, but the developers did not have time to do anything, since in the morning the attack intensified and began to cover the entire subnet, after which the provider again rerouted the traffic to the null interface. The developers were forced to urgently begin work on deploying the SourceHut infrastructure in another data center from backup copies (a temporary subnet for access to the main servers was obtained only after 2 days).
To protect against DDoS attacks carried out at the network level, it was considered optimal to place an intermediate server in the network of the cloud provider OVH, which provides DDoS protection. All requests were sent to this server and then forwarded to the working infrastructure. During the migration, it was not possible to avoid errors that took up additional time, for example, restoration was performed incorrectly using the rsync utility, errors were made in network configuration, and problems with traffic redirection had to be resolved (before DDoS protection worked in OVH, to working servers and DDoS attack traffic was broadcast, to which the DDoS protection system responded and recognized the server receiving requests as the source of the attack).
The developers also contacted Cloudflare and some other DDoS protection services, but the cost of protection was prohibitive. Later, Cloudflare employees were able to negotiate with management to provide protection to the SourceHut project for free as sponsorship, but the SourceHut developers refused the offer, since by that time they had already made significant progress in solving the problem on their own.
It was planned to gradually implement the new SourceHut infrastructure and transfer the project to servers in another data center over at least one year, but in the current circumstances the migration had to be carried out urgently within 7 days. Currently, all SourceHut services have been transferred to another data center and the platform has been fully restored.
The SourceHut platform has a distinctive interface, not similar to GitHub and GitLab, but simple, very fast and works without JavaScript. SourceHut provides features such as working with public and private Git and Mercurial repositories, flexible access control system, wiki, receiving error messages, built-in continuous integration infrastructure, chat, email-based discussions, tree view of mailing archives, review of changes via the Web , adding annotations to the code (attaching links and documentation). When the appropriate settings are enabled, users without local accounts can participate in development (authentication via OAuth or participation via email). The code is written in Python and Go, and is distributed under the GPLv3 license.
To ensure the operation of SourceHut, servers were used in three data centers. The first was used for production configuration, the second was used for backup, and the third was used for experiments on migrating infrastructure to a more scalable and fault-tolerant implementation of the service (the next generation version of SourceHut was being developed).
It took about 9 hours to resolve the issue of providing access to their servers in the main data center after blocking, but the developers did not have time to do anything, since in the morning the attack intensified and began to cover the entire subnet, after which the provider again rerouted the traffic to the null interface. The developers were forced to urgently begin work on deploying the SourceHut infrastructure in another data center from backup copies (a temporary subnet for access to the main servers was obtained only after 2 days).
To protect against DDoS attacks carried out at the network level, it was considered optimal to place an intermediate server in the network of the cloud provider OVH, which provides DDoS protection. All requests were sent to this server and then forwarded to the working infrastructure. During the migration, it was not possible to avoid errors that took up additional time, for example, restoration was performed incorrectly using the rsync utility, errors were made in network configuration, and problems with traffic redirection had to be resolved (before DDoS protection worked in OVH, to working servers and DDoS attack traffic was broadcast, to which the DDoS protection system responded and recognized the server receiving requests as the source of the attack).
The developers also contacted Cloudflare and some other DDoS protection services, but the cost of protection was prohibitive. Later, Cloudflare employees were able to negotiate with management to provide protection to the SourceHut project for free as sponsorship, but the SourceHut developers refused the offer, since by that time they had already made significant progress in solving the problem on their own.
It was planned to gradually implement the new SourceHut infrastructure and transfer the project to servers in another data center over at least one year, but in the current circumstances the migration had to be carried out urgently within 7 days. Currently, all SourceHut services have been transferred to another data center and the platform has been fully restored.
The SourceHut platform has a distinctive interface, not similar to GitHub and GitLab, but simple, very fast and works without JavaScript. SourceHut provides features such as working with public and private Git and Mercurial repositories, flexible access control system, wiki, receiving error messages, built-in continuous integration infrastructure, chat, email-based discussions, tree view of mailing archives, review of changes via the Web , adding annotations to the code (attaching links and documentation). When the appropriate settings are enabled, users without local accounts can participate in development (authentication via OAuth or participation via email). The code is written in Python and Go, and is distributed under the GPLv3 license.
