Before you start…
Check what your target has to offer. Many websites will have a Public API available to prevent getting hit by thousands of different scrapers.
Not only will it save you time, but also functionalities that API's now provide will let you get more clear data with low-maintenance.
Expect the unexpected
As scraping & crawling are getting more and more popular, website owners tend to tighten their web security to prevent sites from going down due to the amount of incoming requests.
Make sure that you investigate how your target handles security as this will be one of the biggest fallbacks if something happens wrong after your scraper / crawler is already in business.
Work with robots.txt
It's important to know what your target allows to crawl, since it can potentially show you where you will & will not meet additional security roadblocks.
It can also save you tons of time by showing where the exact information can be found as these files are used for SEO.
Each website should have this file available, which will be mostly found in format like yourwebtarget.com/robots.txt such as https://limeproxies.com/robots.txt
Look for traps
The easiest way to detect a scraper or a crawler browsing through your website is by showing a link which cannot be seen by any other user on page load.
It can only be checked by looking through the HTML code of the website.
Make sure you inspect your target using built-in tools from Chrome / Firefox, you can simply hit F12 to open Developers Tools. In most cases these links are going to be hidden with an additional CSS code.
Have your connection look human-like
Each website tracks what requests they get, some taking extreme security measures and tracking the whole fingerprint of the request.
When sending requests from scrapers or crawlers, make sure that you include a User-agent and, if needed, send all of the required Cookies.
In other cases, you may need to follow a certain path for the requests to go through since asking for some links directly may give a clear indication that the request is not genuine.
Be responsible with request amounts
It's important to understand that when you send a request to the target, you add to its current load.
Sending too many rapid request will not only slow down your process, but could also result in the website being unavailable for a longer period of time.
Being smart about the amount of your requests will not only help you to achieve quality results faster, but will also decrease the chance of the web owner investigating incoming requests and tightening web security which could result in your scraper needing many additional changes.