I’m a little prone to conspiousy theories, so I might be jumping the gun here, and I may end up looking totally silly, but I think Yahoo deleted my Color Fader from of their index after I reported some spam in their search results.
A little background: For some reason most of the pages on this site don’t rank well in Yahoo. For the month of May, only 1.87% of my visitors came from Yahoo. That’s compared to 46.37% that came from Google. The Color Fader was the only page that really pulled in visitors from Yahoo. It used to rank #1 for the search phrases “Color Fader” and “Online Color Fader”. Those phrases only brought in around 20 visitors a day, but that’s better than nothing.
Anyway, seeing as these are the only phrases that bring me anything from Yahoo, I check them out every once in a while to see where I am. Around 3 weeks ago I noticed a new site was #1 for them (not unusual – the results tend to get shaken up every so often). However, this new site had the exact same title as my site, so I was a little curious. I opened it up and discovered the page was just auto-generated gibberish with lots of ads.
I was annoyed, but figured I’d just report the site to Yahoo’s Customer Care. In my note I said something like “I was searching for my site when I discovered one that had the exact same title. This site turned out to be auto-generated spam with lots of ads.” I’m not sure why I mentioned I was searching for my own site, I guess I was just typing off the top of my head.
After I submitted my report, I got an email confirmation saying it’d be processed within 48 hours. I checked back twice within that time period. The first time both sites were there, although mine had recaptured the #1 spot. The second time I checked back, both sites were gone.
At first I thought they’d just done some reindexing and I’d fallen a bit. However, after some pretty thorough searching, I found that my app, along with that spam page, was gone from their index. I quickly filled out a resubmit form explaining what I thought had happened – that the reviewer had accidentally deleted my site. I read my message over a couple of times to make sure it was short and polite and then sent it off. A day or so later I got back this reply:
Hello Patrick,
Thank you for writing to Yahoo! Search.
If you are not seeing your pages in the Yahoo! Search results, they may
not have been indexed or they may not be in the top results for the
searches that you are conducting. Here are some Yahoo! Help resources
for submitting your page to Yahoo! Search:http://help.yahoo.com/l/us/yahoo/search/indexing/indexing-06.html
and for improving your site’s ranking:
http://help.yahoo.com/l/us/yahoo/search/ranking/ranking-02.html
in the Yahoo! Search results. There are no current indications that your
pages have been blocked from the Yahoo! index. Please note that all
pages are at all times subject to Yahoo!’s Content Quality Guidelines
located at:http://help.yahoo.com/l/us/yahoo/search/basics/basics-18.html
Thank you again for contacting Yahoo! Search.
Regards,
Carmen
So according to that email I wasn’t blocked, which was good. However, something still didn’t seem right to me. It seemed like too much of a coincidence that my app and the spam page would disappear from the index at the same time. So I decided to check out Yahoo’s Site Explorer to see what it had on my Color Fader. This is what I found (note: searching with the “www” brings up additional links):
So for some odd reason only the style sheet for the Color Fader is in the index, and not the Color Fader page itself. Even though the links it has recorded for the Color Fader URL point to the app itself and not its style sheet. That seems really bizarre to me. My gut feeling tells me that the page was removed and didn’t drop from the index due to algorithmic reasons. I’m guessing it was either removed by accident, the person processing the spam report just glanced at the URLs and decided to remove them both, or the guy processing the request just didn’t like the fact that I was searching for my own site and decided to knock me off too. The last idea is a little cynical, but my mind tends to wonder all over the place when stuff like this happens. I suppose it could also be a quality issue, but I’d be surprised if that was the case since I don’t think I did anything particularly egregious in my design.
I probably shouldn’t care too much, the Color Fader is kind of girly app, but it annoys me when I try to do something good and then ended up getting punished. As thing are right now, I wouldn’t recommend reporting spam pages to Yahoo, since they might remove your content along with the spam.
It’s been over two weeks since I sent them my last email. I may try to resubmit the site again later today, however, if I say blocked, I stay blocked. It’s annoying, especially since I was beginning to like Yahoo, but I’m not going to lose any sleep over it.
That is why although it is circumventable it is helpful to deny access to resources based upon user-agent. There is big business in content scraping for SEO purposes, which does tend to be a pain in the ass. Search engines will generally lower the rank of a page in the results, or remove it depending on the nature of the content, if it detects it as duplicate content.
seomoz.org had an interesting post on duplicate content a few days ago. They basically talked about how lyrics sites get away with it. Since they’re all posting the same thing up. I actually kind of wonder how much duplicate content is needed before you’re penalized.
However, I don’t think that was the case with me. The spam page was mostly gibberish. They took my title and some of the words, but other than that, the content on its page was totally different.
Which user agents do you block? I may set up something like that.
I block an astounding number of abusive user-agents from “wget” to “java”, Internet Explorer 5, Firefox 0 (I forgot what it’s actually called), et cetera. If you would like I can send them to you.
Sure, I’ll take the list.
sounds like they accidentally deleted yours along with that spam one.
and then just gave you an automated response to your email 🙁
Yeah, that’s probably the most likely scenario. I was a little worried about it when my site recaptured the #1 position the day after I filed my spam report since I had said in my report that it was the current #1 position that was the spam site.
The situation kind of sucks, but oh well.