Google's Gary Illyes published a blog post explaining how Googlebot works as one client of a centralized crawling platform, ...
It's a solved problem, but I actually prefer a simple web UI.