For the exam
i cut corners by implementing my politeness in a weird way. I keep a running list of domains that ive crawled and check that i havent crawled that domain in the last x amount of ms, if i already have then i go to the next one, that way i can crawl as fast as i want without needing to wait, assuming i have a large enough list of urls and different domains. I know the proper way would be to look what the crawl limit it, but i just set my number large enough to not disturb anyone.
Theoretically i could multithread/multiproccess my implementation using this method, but i dont want to deal with race conditions.