So whilst AI companies are known to behave badly, there could be side effects:
1. Cloudflare may have a financial incentive if they are taking a cut from the payments
2. Possible risk that this may end up stopping every day users who have setup crawlers for their own needs.
Cloudflare are launching a 'pay per crawl' service https://www.cloudflare.com/paypercrawl-signup/
So whilst AI companies are known to behave badly, there could be side effects:
1. Cloudflare may have a financial incentive if they are taking a cut from the payments 2. Possible risk that this may end up stopping every day users who have setup crawlers for their own needs.