I need to check 200 million domains for availability and accessibility to one or another CMS.
I use PHP 7.1 and do a lot of checking.
Iron and settings
- Server: Multicore CPU, 64GB RAM, SSD disks, 500 Mbits dedicated bandwidth (OVH server).
- In the resolv.conf Google DNS file: 8.8.8.8 / 8.8.4.4
- ulimit -n is set to 655350
- I use nload for the analysis of loading of the channel
Testing
I checked the first 1 million domains from the database using a different number of parallel running processes. I ran into the problem that increasing the number of processes, the number of domains that do not respond within 30 seconds, has greatly increased. Here are the results.
1. 1000 processes
Test: 1,000,000 domains, 1000 parallel processes, average load of a channel is 85 Mbits, the total scan time is 1 hour. Result: 65% of domains were successfully resolved , 35% were not resolved due to a timeout.
2. 300 processes
Test: 1,000,000 domains, 300 parallel processes, average channel load 70 Mbits, total scan time 2 hours. Result: 85% of domains have been successfully resolved , 15% have not been resolved due to a timeout.
findings
As we can see, increasing the number of processes by 3 times, we do not get an increase in channel loading by 3 times. The number of domains that were not available / unresolved is greatly increasing. At the same time, the check speed was increased 2 times.
Question
Where is the bottleneck of such a test? How can I use all the channel bandwidth of 500 Mbit? Should I use my own DNS server and if so, how to configure it correctly?
I will be glad to any ideas and advice!