Attention Bots and Ad Blockers

Let us first say thank-you for all of the users that have frequented this site since it began in about 2005.  The ETF market has changed significantly through that time, and our user base has grown as well.  If you want the short version, you might skip to “The Way Forward” below.  Otherwise we’ll include some background.

Our goal with this site was to make most information available for “free”, supported by advertising.  That is probably because we’re cheap and don’t like subscriptions, but we also don’t like abusive advertising so we have stressed to our ad network to only show acceptable ads.

We have shared our concern about the increasing use of ad-blocking software in the past, particularly among our non-subscribers..  We have watched our stats as the percentage of users blocking ads has grown from the mid-teens percent to the current mid-thirty percent level.  That trend is only sustainable if those choosing not to view ads are willing to subscribe to our Premium Access service or provide some other economic benefit.

If that weren’t bad enough, we recently ran a study on all pages served and found some disturbing statistics.  First, let us categorize our users into three groups which we’ll call Good Bots, Bad Bots, and Web Users.  The last category, the Web Users, are our target audience.   You and we are generally in that category.  When we speak of 35% of our users not viewing ads,  we’re speaking of 35% of the targeted Web Users.

The term Good Bot is applied liberally here.  It includes the bots, crawlers, and spiders from the likes of Google, Bing, Yahoo, Baidu (China), Yandex (Russia), and a slew of lesser known players. The term Bad Bot is generally reserved for an automated system pretending to be a regular browser.  A few of these may be corporate based, but many are from traders that have automated the harvesting of data from our site.   A few even access the  site with a valid user ID.

What we found is that most of the pages we create are at the request of the Bad Bots, utilizing precious server time. We also found that the “Good Bots” actually request more pages than our targeted Web Users do.  Finally, we found that the bills are being paid by roughly 15% of the pages being viewed.

The Way Forward

Most of the “Good Bots” are truly good bots and provide a benefit by making the site known to our target audience.  Some, however, provide little benefit to our particular site, and a few of these have already been restricted from our site.  Others will be restricted in the near future.

The “Bad Bots” are a mixed lot as well.  Many we’re sure are innocently created by skilled individuals who mean no harm.  But as a group they have a significant negative effect and will find the door closing over the coming weeks.  If you have automated access to our site this is your opportunity to reach out and establish a legitimate path forward.  We have developed a token that subscribing users can utilize in such scenarios so we know who is requesting the pages.  Our site is not designed to serve 1000 pages per hour to a single user, as some have been requesting, so please throttle your requests even with a subscription and the use of the token.  If you have high-volume needs please contact us.

That gets us to the Web Users and the fact that some economically support the site and others do not.  We have recently been placing our own banners when ads are blocked, asking for user’s support.  We hope most of the ad-blockers will take the hint, but the reality is that some will not.  The situation has progressed to the breaking point, and we will soon be limiting access to users that do not view ads and do not subscribe. This is the only way to keep the site functioning.

In exchange, we are committed to only showing ad formats that we find acceptable and relatively low impact. If you see a pop-up ad on our site, or an auto starting video [with audio], we want to know about it and are committed to eliminating such.  In the end we want to have a better performing site serving pages effectively to a deserving audience.

We’re asking for your support have flexibility in how we move forward. Discussion is welcome.

Note:  If you utilize an automated system to retrieve information from our site, please contact us about the use of tokens in the URL.


4 thoughts on “Attention Bots and Ad Blockers

    • You shouldn’t be blocked as long as you are logged in as a Premium Access user. The key there might be “logged in”. I’ll email you directly. Hugh

  1. good site. just found this and you have been around since 2005! well done and I am no bad bots for sure. happy to view acceptable ads in order to enjoy what you put out here. Thanks.

Leave a Reply

Your email address will not be published. Required fields are marked *