still have the ability to disable the crawler via robots.txt flags
Never rely on robots.txt. Configure it, but do not rely on it. A bot will do what a bot can do. Their bots are trivial to spot so close the connection, bonus if done silently closing the socket and inhibiting first RST from web daemon and second RST from the OS.
This implies the solution is every actor in the world willfully complying which doesn’t sound very realistic if you ask me.
Similar to saying world hunger would be solved in everyone who could donated $100 to those in need. Sure it may be true but foot stomping about why isn’t it organically happening is missing the point.
Yeah it reminds me of the typical copyright notice found in a book, even decades ago:
> All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the publisher.
For AI, wasn't the book first "reproduced or transmitted" into a training system?
Returning some text files from the server fs is _wayyy_ lighter than letting the bot go through the search, middleware, and db while crawling your stuff.
I'm not convinced that it's possible to poison AI. Real data all correlates, fake data is all unique in its own way. I suspect feeding AI fake data is a waste of time.
Angel banning is better, 100% agreed on giving it false data that the scraper builders can't easily detect. Give it seemingly correct and useful but totally wrong answers
I would assume it's exactly what they described, banning something by pretending to still helpfully assist them while silently undermining them instead.
I'm a bit confused why people are freaking out about these term changes - Meta scrapes my website already on a fairly regular basis (even with robots.txt). Is the point of this change to justify more intense scraping?
Color me shocked... one of the most profitable companies in the world, that doesn't contribute anything positive to society, is unethical. Just another day in America that ends in Y.
You can but you'll probably want to do it with a browser extension or an undetectable webdriver.
I suggest doing it, actually. Could be a fun project. I suggest not making the script make any extra requests on your main account just in case - anything already on the page is free for scraping.
Blocking by UA is pointless now, meta-externalagent rotates and a chunk comes off non-Meta ASNs. We matched on ASN plus TLS JA4 and that actually stuck.
still have the ability to disable the crawler via robots.txt flags
Never rely on robots.txt. Configure it, but do not rely on it. A bot will do what a bot can do. Their bots are trivial to spot so close the connection, bonus if done silently closing the socket and inhibiting first RST from web daemon and second RST from the OS.
I have been writing bots and scrapers for almost 20 years.
Not once have I had someone I work for in this capacity even ask about robots.txt
You are part of the problem, sounds like.
This implies the solution is every actor in the world willfully complying which doesn’t sound very realistic if you ask me.
Similar to saying world hunger would be solved in everyone who could donated $100 to those in need. Sure it may be true but foot stomping about why isn’t it organically happening is missing the point.
Yeah, calling this "disabling" it is inaccurate, it's just "asking nicely".
Yeah it reminds me of the typical copyright notice found in a book, even decades ago:
> All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the publisher.
For AI, wasn't the book first "reproduced or transmitted" into a training system?
How would we reformulate it today?
Legal notices are designed to be as scary as possible so they usually say you can't do things that you actually can do.
See also: Nintendo
Wouldn't feeding them false data be even better?
That could consume as much, if not more, resources as returning real data
Returning some text files from the server fs is _wayyy_ lighter than letting the bot go through the search, middleware, and db while crawling your stuff.
Or just sending it a tiny zip bomb, or even just some sh*t right out of /dev/random
I'm not convinced that it's possible to poison AI. Real data all correlates, fake data is all unique in its own way. I suspect feeding AI fake data is a waste of time.
Angel banning is better, 100% agreed on giving it false data that the scraper builders can't easily detect. Give it seemingly correct and useful but totally wrong answers
Would you mind explaining what "angel banning" is? I've never heard this term before and even Google doesn't seem to know what it is.
I would assume it's exactly what they described, banning something by pretending to still helpfully assist them while silently undermining them instead.
We've already been using AI to target military strikes without human intervention.
Make of that what you will.
I'm a bit confused why people are freaking out about these term changes - Meta scrapes my website already on a fairly regular basis (even with robots.txt). Is the point of this change to justify more intense scraping?
just rename the company to Meta Law or Zuckerberg & Associates already
Meta law actually sounds pretty good to me.
Color me shocked... one of the most profitable companies in the world, that doesn't contribute anything positive to society, is unethical. Just another day in America that ends in Y.
And they spend huge resource so that we can't scrap Facebook - our own feed.
> so that we can't scrap Facebook
Whilst I assume you meant to say “scrape”, I can’t help but appreciate the poetry of that typo.
You can but you'll probably want to do it with a browser extension or an undetectable webdriver.
I suggest doing it, actually. Could be a fun project. I suggest not making the script make any extra requests on your main account just in case - anything already on the page is free for scraping.
DON'T TOUCH META's apps. Zuck is PUBLIC ENEMY #1
I for sure don't love any of the oligarchs, but how is Zuck worse than e.g. Musk?
Similar resources and anti-social intent, but better organized and disciplined.
Does it really matter what they have "permission" for when they'll do it anyway?
Humanity experts warn oligarchs, "not good"
News at never.
Blocking by UA is pointless now, meta-externalagent rotates and a chunk comes off non-Meta ASNs. We matched on ASN plus TLS JA4 and that actually stuck.