If you haven't already, try adding some "trap streets" to your data. Map makers occasionally include streets that don't exist, so if a competitors map includes it too, it's clear that the competitor copied it:

https://en.wikipedia.org/wiki/Trap_street

I did that with an online marketing dictionary I wrote years ago, some of the definitions included strange usage examples that contained the names of several of my friends. When a competitor scraped us, instead of shutting them down, the boss negotiated a data licensing arrangement with the scraper instead, so we ended up getting a revenue stream & backlinks out of the incident.

If that fails, and talking to them directly doesn't work, then the DMCA is often effective. I've made DMCA requests against websites that distributed cracks of my software & they often disappeared in a couple of days.

Similar to trap streets are "phantom settlements", aka "paper towns", which are fake towns rather than streets.

Now, this idea is not limited to maps: Google used trap search results to catch Microsoft using Internet Explorer to scrap Google search results: https://googleblog.blogspot.com/2011/02/microsofts-bing-uses...

I was told all these phenomena (on maps) are referred to as "cartographers' follies", so the other names are interesting to hear.

In the "real world", it was commonplace for compilers of mailing lists to include several "phantom names" in their lists. If those names received a mail, the list-holder would send an invoice to the person who sent it. Simple, elegant, very difficult to bypass way to protect your knowledge-based business.

This is the same as "poison pills" right? A directory website would include some fake entries so that if they pop up on other sites they could have only come from copying the original directory.

A scraper could get around that by x-referencing the data with another source, if possible.

Though that creates quite a bit of extra overhead, and you only really have to catch them once.

Copyright protects only creative works, not any information as I understand.

It doesn't in the US, but collections of information are protected in the EU and some other places [1]

1: https://en.wikipedia.org/wiki/Sui_generis_database_right

It's a little weird. Trap streets often aren't protected by copyright as the existence or non-existence of something is a fact and facts aren't copyrightable.

And maps are nothing more than a collection of facts. As are recipes. You can take the recipes in a cookbook, write them down, and disseminate them how you choose and no one could say boo. Which is why Coca-Cola guards its recipe carefully. Once it is known, they can't do anything about it. You can't copyright "a lot of sugar, carbonated water, caramel coloring, mix".

But you also can't just photocopy a map and sell it as your own. Presentation falls under copyright. The look of the map is a distinct piece of art that has copyright protections. The combination of colors, fonts, line thickness, placement of labels, etc, are all things you can't reproduce.

But the information that Bob Street crosses Bill and Jill streets at these points, that's just facts.

Include a base64 encoded image as a street name.

This is pretty much what the DMCA and other copyright mechanisms were made for. Send your request to Google and have them delisted.

https://support.google.com/legal/answer/3110420?hl=en

I had my entire marketing site copied, word-for-word. I still have no idea why (other than perhaps to damage google ranking due to duplicate content?).

I reported it to google, and they removed it from search results in a few days. I also reported it to the site's host, who took it down completely.

Almost the same here - they copied my whole website, improved the user interface, and added a little content. Luckily I still outrank them :)

Yes, I would combine this with a couple of fake entries, and when those show up on their site, you file DMCA claims with google, and with their hosting provider. An american company has no choice but to shut their site down under threat of DMCA claims. If the site gets back up, add another fake entry, and do it again when they scrape you, until they give up.

He can also go after them with this. He can write them to take it down. If they don’t, go to the host and have them take the site down.

"Excuse me, could you put down my TV?"

Asking nice can work, but I find it depends on what is in my hand at the time.

Does anything stop the scraper from using DCMA on the original site?

They'd lose their source of data

Anything receiving 5k visits a month, is likely crawled at least once a day unless it's told not to. In this case, Google could check crawls and determine the original source of content between the two competitors.

If the source that is copying is also being crawled once per day isn't possible that it would be crawled first and appear to Google that it is the original source?

presumably site A got popular before site B started scraping it; else there wouldn't be much point in scraping A. There is ofc the scenario was A was a generally unknown resource, and B realized it had the potential to be valuable if you just SEO'd it right.. but there's still probably evidence somewhere that A cropped up first (how did B stumble into it?)

If they do, you respond with a counter-notice. nce you respond with a counter-notice, your provider needs to restore the content unless the copyright owner initiates legal action.

https://help.github.com/articles/guide-to-submitting-a-dmca-...

I would think that the scraper wouldn't want the source of their material to disappear, for one.

I worked at a company that had this problem and while I was not involved with the solution, I did sit near the guy that was and had extensive talk with him on how to resolve the issue (Or solve the issue well enough).

* Require accounts with good robot account creation detection. Actually, not hard since there is a lot of canned coded for this.

* Subscribe to Tor and Proxy IP collection services and then block all of the Tor and Proxy addresses. Tor IP list is free from the Tor Project and Proxy IP's can be found through various services.

* Put in crawler detection code. Most scrappers are really simple and are really easy to detect. I think Apache even has a module to detect simple scrapers and autoblock them.

* Feed bad data to scrapers once they are detected. This was the thing that made scrapers go away. Once they know you can detect them and feed bad data, they know they are screwed and will give up. I did get a kick out of looking at the scrapers website and seeing "Penis won the 5K Penis run". Of course, you should really feed the scrapers bad data that looks believable.

* DMCA their website with their provider. The company I worked for did have several scraper sites deactivated.

* Ask Google to delist them. The company I worked for did have several scraper sites removed from Google.

* There are a number of services that will help you with all of this but your website is only 5K unique users a day and not really worth the cost.

Banning Tor and proxies is pretty unfair to legitimate users who use those services.

While i agree, the op isnt doing a charity either, and op needs to keep their interest at a higher priority to be able to keep providing the service.

If you are able to detect them, serve them garbage data instead of blocking them. Mix in some false entries etc. If you completely block them they will work hard to work around the block, but if you give them some bad data randomly it will take them longer to notice.

Also, you can use the fake data as evidence they scraped you.

And also to try to obtain a pattern of what request headers are common... Try to create a special hash from the request headers. Then find a way to infiltrate the data with it so you can identify them better

That is what I did initially. My API was sending garbage data Saying this website is shutting down, please visit [mysite] for more info :D. But later on, they figured, I was sending garbage data and people were visiting my site because of that. I even put in a tracked link to see how many visitors I'd get from there site and I was getting around 500 people every day.

It was like a cat and mice game for a few weeks.

It's not nearly as valuable as cryptographically provable evidence of the pages having identical content as yours, with yours being of an early publication date.

Write a few things. Register them with the copyright office. Then post them and wait for them to copy it. Now you can sue them for the max and strong-arm them to take it all down or face obvious legal fees and punishments over 100k.

Can you sue for monetary damages if you yourself were not making any money off the content? If anything the other site could be called a mirror, and saving our op money on the hosting cost by diverting traffic.. (although maybe not from scraping?)

I'm not sure about how this works in the US but in my country (Germany) "commercial use" does not actually require money to change hands or operating at a net profit. And last I checked copyright laws (if they apply to the content) apply to everything you create, whether you monetize it or not.

That HN comment of yours? If it's sufficiently non-trivial, it's protected by copyright. Even though neither you nor HN directly make money of it. If someone scrapes HN and republishes that comment, you could go after them and demand they delete it.

You can sue for the loss of value of your website, even if it doesn't make an income it remains an asset with monetary value.

(Note:I am not a lawyer.) If you're a US citizen, any content that you've created yourself is -automatically- copyrighted. (That's also true in all countries where Berne Convention standards apply.) See this link for further basics: https://smallbusiness.findlaw.com/intellectual-property/what...

If they're scraping content you created, they have already broken the copyright law. (They may not realize that. Make sure they do.)

Since 1989, in Berne countries you don't even have to post copyright notice for them to be in violation. So your first step would be to notify them that they've already broken the law, and invite them to cut it out.

Your second step would be to contact a lawyer.If the other site is not in a Berne country, that might complicate things. On the other hand, they may simply not be aware that they've broken the law.

Should be recognised that not all data is eligible for copyright. "Facts" is a broad category of exemption with strange case law. It may be relevant here if the data being scraped is available elsewhere.

Note also that in some places, databases are explicity exempt from this "facts" exemption if you made considerable effort in compiling the database. (IANAL)

Yes indeed. See:

https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....

Does the same thing apply to newsletters? Let's say I get a bunch of free newsletters to my inbox and share them on my website (I mean literally copy the body so people don't have to sign up for the newsletter themselves), does it count as copyright infringement?

Also not a lawyer buy If you're in europe I believe there's also something called database rights which gives some protection to people that gather data from different sources event if they are not content creator.

Correct, the database directive: https://en.wikipedia.org/wiki/Database_Directive

Aka Directive 96/9/EC of the European Parliament

OP mentioned that they just buy the data from someone else and therefore OP is not a database rights holder.

What if I created my content by slightly altering other content? Like writing a news article from a Reuters press release, how much difference does there need to be?

Seems like a pretty hard line to draw.

If it's based on the other content, then it is a derived work and copyright infringement.

Keep in mind that copyright is for creative works. Facts can not be copyrighted. In a news article, it is the expression of the facts that covers the copyright, not the facts themselves. So you can definitely write an article stating the same facts as a Reuters press release (and be even based on that press release). You just can't base your prose on their prose. You have to completely rewrite it as a human -- it has to be an artefact, not an automated process.

However, there is another class of copyright for collections. A collection of facts can be copyrighted. You can't copy the same collection of the facts, nor a substantial portion of the collection. You can take individual facts out of the collection and include them in a creative work, but you can't just grab facts one at a time and create a new collection -- that would be a derived work.

Like all laws, copyright law is subject to interpretation by humans (lawyers and judges). While you might think that you are satisfying the law, they may disagree. To avoid that circumstance it is best to stay in an area where it is obvious to everyone that you are within the law. If you don't mind being sued, then you can try to push the boundaries. It is your choice.

You sound like you're asking this question as if it's a brand new question that nobody has ever thought of before, instead of something core to the entire idea of copyright: https://en.wikipedia.org/wiki/Derivative_work

Yes, it is a hard line to draw. It is advisable that if your business is dependent on being on the right side of copyright issues that you simply stay well away from the line.

It's actually extremely easy to detect. Even if you change verbiage and move around paragraphs. Most sham content providers are much more sloppy than moving paragraphs around, and will just use a thesaurus to modify verbiage. This typically makes it extremely easy to detect. Because, even though a word might be a synonym, it might not work grammatically in it's context.

> Seems like a pretty hard line to draw.

All laws result in tricky distinctions, and that's what case law is for.

This is a good article I've followed in the past. idk how helpful this is, you might have already gone through it.

https://github.com/JonasCz/How-To-Prevent-Scraping

That was definitely a very good resource. I've gone through that and tried to implement as many things as I could.

That's a great resource (had to share it on HN LOL)

Learn to accept it. Only way to stop scrapers is to make it a financial burden for them to scrape your website. There are to many ip address, proxy, vpn, and botnets for you to try and block them all. You can write code to try and tell legit traffic from scrapers traffic but then they will just figure out how to bypass that through trial and error. You can try to take legal action, but that will cost you money up front and might take longer then you think. You can try to get there service terminated, but most host won't do it without a court order. You can make you html/css hell to read randomize autogen all your class and id tags. Put all you elements in random order so the people writing the scraping code don't have a default template they can update in minutes. Use javascript to actually send the data which then you use to identify people and have a call home function so they cannot hide behind proxies and vpn and such.

These are all just things that will slow them down, but won't stop them. Make it cost them more money then they make using your data will be the only way to truly stop them.

> If I change certain things, they counter it immediately and make it work.

If the content is markup based, are your countermeasures about changing the IDs, classes, or overall tag structure of the markup you serve? I was wondering if you could have several variations of the above, and serve your content via a random one each time that would be visually indistinguishable to a human viewer. The person maintaining the scraper would have to have seen and adapted to all of them to get all your new content reliably. Not an impossible hurdle, but they might try easier targets if too many barriers are in the way.

Since the seemingly colluded disappearance of any freely available finance data APIs (excepting IEX), both Yahoo and Google Finance employ this method of randomizing class names on pages with finance data (quotes, fiscal data etc.). Inspect element on those pages for a good example of this tactic. I feel like this could make it much more difficult for your content to be stolen.

Easily bypassed: just retry on failure until scraper gets syntax it likes.

The other end retrying until it gets what it wants will dramatically change its usage pattern in ways that may be easy to detect unless they have an enormous store of IPs to connect from.

There are enough suggestions in here to provide a bunch of useful options, and while the site itself may not be making money, the experience dealing with this may be very useful on a resume or for building a client base with similar issues.

Possible approach: look for abnormal usage patterns to ID opponent systems. Randomize format and possibly other steps to assist that. Build that randomization in marginally effective ways that are easy to improve later. Build a way to feed bad/poison/"test" data to specific source IPs. At a time chosen to maximize impact, start feeding poison data to the suspect IPs using the marginally effective randomization, while feeding regular data to most visitors but with much improved randomization. Basically make your opponent's site visibly unreliable.

If you feel particularly vicious and know something about the opponent's infrastructure, make the poison vicious e.g. feed SQL injections. Be aware that this may have costs - you'd likely be fine on a legal basis ("I'm not responsible for their crappy sanitizing of inputs they shouldn't have had anyway") but you might still incur costs (lawyer if sued).

Edit: also, anyone going to serious measures to continue scraping after you act against it may also be inclined to ddos your site if you actually fully block them.

The content is served using API that is open for everyone to use. And that is making it difficult for me to protect it. I've tried changing the structure of API response several times, but they counter it within few hours. Tried adding unique headers to requests etc (That worked for quite some time) but they figured it out ultimately. I come up with some solution, they figure it out in a day or two. And that went on for few weeks

Can you figure out how the person is scraping your site? If so, start putting in content that only shows up for that scraper, such as...

"this content was unabashedly stolen from <your site URL>"

...or maybe...

"my favorite movie is Mac & Me, check it out! https://youtu.be/vNjACYfQlbI"

"this content was unabashedly stolen from <your site URL> *by <IP address of scraper> on <date time>".

Unless the content must be 100% static, you can embed the caller's IP address in it.

Add sensible rate-limits (modelled after an average user's expected usage) and present a captcha after that. Make it per-IP.

Sure, the crawler can change IPs, but I doubt they have unlimited addresses and eventually they will run out.

Another option (if you can reliably detect the scraper) is to poison their data by sending them bad (but valid-looking) data.

This will be more effort to implement than it will be to route around: between free VPNs and TOR, everyone has infinite IPs.

Proxies are actually shockingly cheap

If you're able to detect when your content is being scraped, you can send them the same text but with "confusable" [1] characters (i.e. utf8 characters that look similar to ascii characters).

[1] https://unicode.org/cldr/utility/confusables.jsp?a=test&r=No...

Find a way to contact the owner of the site and bait them with something like an acquisition offer, you might be able to use the email in the whois data if there is one. The point is to get the owner to click a link you provide, behind this link you want to log IP/Headers, maybe do some JS fingerprinting/etc... It is unlikely they will open this link over VPN/Tor. If you feel like it check your server logs to see if they have accessed your site from that IP and when. Then send a C&D containing all this information and tell them that you have legally contacted their ISP for their subscriber information and not only will they be sued for violating DMCA for possessing your data but that they will also get hit with the CFAA for circumventing protections against access and that they will enjoy a five year sentence in a federal pound me in the ass prison.

I don’t disagree with the content of your post and I know this is a US cultural trope at this point, but trivialising rape benefits precisely no-one.

"Federal pound me in the ass prison" is a quote from the movie Office Space.

If you already knew this, then I guess the applicable movie quote this time is "Lighten up, Francis" from Stripes.

Please don't invoke the CFAA for scraping.

Why not? Scraping seems like exactly what the law is aiming to prevent, in spirit and in letter.

The scraper is 100% authorized to access the data. It's the rehosting that is the problem.

That's well thought out!

Everyone else here has a lot of good suggestions, but the biggest thing I would recommend is NOT actively blocking them. As you already know, if you detect them and make things harder...they adjust.

Start with successfully detecting them and go from there. Log them. Track origins. Compile patterns.

Then, when you decide exactly what you want your move to be use all that data to be as effective as possible.

ideas:

1. Create content that can't be scraped. I'm not sure exactly what your "content" is in this case, but images can be watermarked, text can be given lots of references to your own brand and service, ect.

2. Submit legal requests to google to remove the content. Enough violations can get their domain blacklisted. I've done this successfully in the past for competitors using my trademark without permission to get it removed from Google. Ads. https://support.google.com/legal/answer/3110420

3. Talk to the hosting provider, if applicable. If someone is repeatedly breaking copyright law using their platform, they may have some incentive to stop providing hosting.

You can also apply steganography to your data, i.e., embed a copyright message that can't be detected or easily removed.

https://en.wikipedia.org/wiki/Steganography

Instead of blocking them by IP, send false information to their IP. Make it subtle so they don't immediately notice, and end up hosting tons of unreliable information.

Maybe even automate it: one lots of requests start coming from the same source, a sign that somebody is scraping your site, start messing with the data you send them in later requests.

This all assuming there's no legal objection to providing false data on this particular topic. There might be, depending on what it's about.

Google ReCaptcha is a pretty low friction way for (most) users to get your content.

You can do this while serving your real content to Google.

ReCaptcha is very high friction for many users, particularly those not logged into Google services

For example I have never reached the end of their 'identify the road signs' challenge and if I encounter a site that uses them I'll just close the tab. Even Google Search.

Much better to use rotating series of questions related to your domain of data.

The spammers & scrapers seem to have found their way around ReCaptcha. I made a simple spam filter for my website comment form that logs every submission, and a lot of the spam being sent (maybe 20%?) has a valid Google Recaptcha. I'm thinking of removing Recaptcha because simple keyword based filters have been more effective than ReCaptcha has been.

ReCaptcha and the original two-word based one are just Google using your visitors as 'mechanical turks' - by solving ReCaptcha et al, they are actually helping Google apply identification tags to images in streetview, and previously the Google books collection.

An afternoons coding would get you an independent self-hosted Captcha system that would be just as effective (if not more so, as it's proprietary) and probably less annoying to your visitors.

> and probably less annoying to your visitors.

As someone who has used both, absolutely 100% no in every way.

Google's captcha is so superior to what you would write, please don't tell people not to.

How so ? It's really not that hard. I've done it, and it zeroed spam coming through a web form almost 100%.

Both (custom and recaptcha) are essentially security by obscurity, in the sense that a skilled attacker will eventually bypass it. The problem is not that difficult. The advantage Google has is being able to move the goalposts, potentially on a daily basis, while you presumably have better things to do. On the other hand, they are a juicier target, so will draw more effort. Ymmv.

Google tends to strike a good balance between convenience and irritation.

So would disabling submissions. The real question is how many legitimate submissions weren't made successfully because of a bad captcha.

I've build a ReCaptcha breaker in an evening (which beats Google 100% of the time). The only reason why most spammers don't evade ReCaptcha is because they make enough money through other means (and most spammers can't program).

There are also a lot of services where you can simply send your challenge token and some guy will solve it for a fraction of a penny.

As soon as other money-making opportunities will dry up, spammers will evade ReCaptcha and spam will increase regardless of your security mechanisms.

this won't work in this case. The site is being specifically targetted. A custom captcha will be trivial to bypass. The reason it worked for you would be your site was just bulk scraped and no patterns could be found in your captcha to match it to a solving solution so the spammer moved on to the next.

When your site is specifically targetted, captchas will not work

They're probably using https://2captcha.com/

They pay 50c per 2 hours to the workers, and API access costs 50c-$1 per 2000 solved captchas.

I think most of the people doing this are in India, where US 50c for a couple hours translates well enough. (Completely naive about the truth of the situation though.)

One of the regular spam messages mentions the software they're using ("You read this, so you know it works!"), so it's pretty clear that ReCaptcha has already been defeated. I'd mention the software, but I'd rather not give them publicity.

Others have been able to beat reCaptcha too, though:

https://www.youtube.com/watch?v=fsF7enQY8uI

didn't watch, but the Russian xrumer forum spammer has been at the forefront of captcha solving for the last decade or more.

Something I haven't seen mentioned yet, but did you try to simple get in contact with them to discuss the issue?

Since your website is free, it might be worth combining forces to serve your users better. I know it's hard to swallow, but in the hand what matters is that what you do is useful to people isn't it? And if your users are moving away it probably is because `some` of the things they do is right?

Are you using server side rendering? There are ways to make it harder. But ultimately you don't want to sacrifice the user experience. Why don't you require registration provided by a third party like facebook or google - it's one click sign on and you can make your access requirements extremely low.

"If you're reading this, you should probably read its article on its original source (your link). This entry is scrapped from site (your link)."

Have you tried filing a spam report with Google?

https://support.google.com/webmasters/answer/93713

When you say "my content", are you referring to content that you created, like a blog or online resource or are you referring to content your users created? Also is the information they're copying factual in nature (like a map or results of a calculation)?

If it's factual in nature: It's not protected by copyright.

If it's content your users created: You are not the copyright holder and cannot file a DMCA takedown notice. Your best bet is really going to be making yourself unscrapable.

If it's content you created: File a DMCA takedown notice.

> If it's factual in nature: It's not protected by copyright

Compilations of facts can however be protected in the EU under the Database Directive.

Also, Terms of Use etc.

> If it's content your users created: You are not the copyright holder and cannot file a DMCA takedown notice.

But you can still have copyright on some of your data. If the other company simply copies everything, you can take legal action.

This is my website if anyone was wondering: https://freephonenum.com/

My competitor was scrapping all my phone numbers and SMS received from my site and displaying it on their site. It took me several months to figure that out. I was seeing abnormal bandwidth usage for JSON (my API endpoint) but never realized that someone was constantly pinging my APIs.

[edit] Now, I've removed the API and everything is served as HTML (not easier for someone to scrap my API)

How can a "service" be scraped, unless the service is just making content available?

Or are you referring to the copy that promotes the service?

Scraping is very easy if your content is rendered on the server. If it's rendered on the client it's a little more complex, I guess your competitor would have to use headless browsers.

Technologically, you're potentially up against services like Crawlera which will be pretty hard to beat.

The best solution is probably filing copyright claims through DMCA which would remove them from Google rankings.

Usually scrapping means it will go though all links you provide. If you add non visible links on your site that link to rubbish content he will automatically add them as wel.

We use DistilNetworks to do bot and scraping limiting. I wouldn't exactly say I recommend it, I'm skeptical about it's usefulness, but it might be worth trying to see if it solves the problem. At least then they are battling a third party with some expertise in it and additional data, rather than you having to try to come up with countermeasures from whole cloth.

What about a bunch of dummy links that aren't visible in the UI (funky css selectors) that are mixed in with other links but provide junk content and take forever to load.

If you can randomise how those selectors are applied enough it might make it less fun to scrape.

"Ask HN: How to deal with GDPR / cookie notices in the context of a crawler?"

https://news.ycombinator.com/item?id=17795750

Might give an idea.

Curious how this turns out..Let us know what you tried and if it worked.

Find a way to subtly detect them (user agent? Sneaky headers?) and serve them janky content. Serve them the same copy with copious swears. Have a bit of fun while you go through legal channels.

Can you implement a captcha somewhere? Or figure a way to obfuscate your html if that’s what he’s scraping. Something that can scramble the css class names each build.

It really depends on how complex/simple the page is. If its small site with little context you can usually traverse from parent body ignoring classes/ids. However, for more complex sites changing class/id values on each build can certainly help make this a lot more challenging.

Ideally changing the children counts to prevent things like $(body).children().children()[3].html().

If you can detect them try to serve them wrong data

Yea I was thinking a "hellban" type tactic might wreak more havoc. Just subtle enough to de-rank them.

Loads of great suggestions. The DMCA request is the tool you want. That will cause a downrank if you keep doing it. Problem solved.

I thought Cloudflare's "under attack" mode can prevent scraping bots. Correct me if I'm wrong.

It can be bypassed and they don't make it very difficult to bypass it.

How advanced is their scraping? Can you just serve them crap based on some simple combination of identifiers?

Encode a timestamp. If they are running at the same time of day feed them a bag of dicks.

You can add typos to a certain set of items and follow them on their service for a match

How about offering them a paid-for API with your content?

I was thinking the same thing. I believe with the latest changes in the algorithm this should be even harder.

It requires some work but they can be beat

Just use imperva....