Fetch pages, extract links, queue what you have not seen, repeat. The loop is trivial; everything interesting is in the constraints around it.
Politeness is the binding one. You may not hit a single host in parallel or faster than its crawl-delay, so your queue cannot be a simple FIFO — it has to be partitioned by host, with each partition rate-limited independently. Throughput comes from breadth across hosts, never from depth on one.
Deduplication decides whether you finish. A billion URLs cannot each be checked against a database, so you need a probabilistic filter in front — a Bloom filter accepts false positives (skipping a page you have not seen) in exchange for constant memory, and that trade is almost always right here.
Freshness is the recurring cost. Pages change at wildly different rates, and recrawling everything uniformly wastes most of your capacity. Crawl rate should follow observed change rate.
You start with a single worker pulling from one undifferentiated queue.