Dealing with the "Keyword not provided" problem in your statistics

This article explains why the proportion of "keyword not provided" visits to most websites is increasing, and gives you options for finding out what keywords people are searching for when they reach your blog.

Why the percentage of not-provided search visits to your blog has increased

If SEO is important for your blog, and if you therefore watch the Stats > Traffic Sources tab in your Blogger dashboard or your Google Analytics results, you'll probably have seen that proportion of your search-visitors whose keyword is "not provided" has gone up a lot recently, to be more-or-less 100% of your Google search traffic. (In the Blogger Stats tab "not provided" isn't shown - but the number of visits per keyword is now massively less than the vists from Google.)

This is no accident: Google is now witholding the keywords that people use, and (says that) this to protect your visitor's privacy. The issue has been widely discussed in sites like SearchEngineLand.

Opinions vary, but many people believe that

"Not knowing keywords has big implications if you use data about what people search for to decide how to develop your blog." [tweet this quote].

For example, I publish listings of the contents of old (ie graphical copyright expired) song-books in a particular niche on one of my blogs. There are far too many songs for me to load the full text or sheet music of all of them. And this is a niche with lots of competition: there are a zillion websites distributing song-lyrics (most illegally). But by watching the search-terms that led people to arrive at certain pages, I can identify particular songs that people were looking for and not finding anywhere else (the so called "long tail" of search keywords). If these songs are now in the public domain, I can make a dedicated page for them, and share what I know - in many cases after doing more research and pulling together information from a range of different sources. Not knowing the keywords that people use to get to the book-listing pages would totally destroy this approach.

What you can do about it

So far I've identified three alternative options for getting data about what my visitors are searching for.

Ask for user-provided information

I've used Google Docs to make a data-collection form, and invited my visitors to use it to tell me about songs they are looking for.

The advantage is that I can ask them for richer information than just the keywords, eg where / when they remembmer it from, multiple snatches of the lyrics, what style the music is, etc.

But the disadvantage - and it's a big one - is that it only works for people who actually get to my site and then go into the other page where this form is kept, and fill in the form. I don't want to go into details - but let's just say that I haven't been run off my feet!

Get data from WebMaster Central

If you have verified your blog in Google Webmaster Tools, then the Search traffic > Search Queries tab shows the queries that have caused your blog to show up in search results pages, as well as how many times this has happened and what position, on average, you had in these pages.

This is richer information than you get from Analytics or Blogger-Stats, which only tell you about people who actually visited your blog.

But the disadvantages are that data is only kept for 90 days, and it only shows the top 2000 keywords. Both of these are issues for me - some of my song-book contents are seasonal - if something is being looked for now, then the moment (week, month) may have passed by the time that I've noticed the trend, researched the song and written it up to a standard that I'm happy to publish. So really I want to checking the logs for nine months ago, so I can research things that are likely to be popular again next year.

Get data from AdWords

Advertising campaigns are the one place where Google is passing the search-keywords through to back-end systems. And because of this, Adwords does have data about what your visitors are searching for - provided you've set it up to collect this data. To get it up:

Firstly, sign up for an AdWords account. You probably have to deposit $10 into the account to get started - but you don't actually need to set up any advertising campaigns or spend any money after that.

Then link your AdWords account to your Google Webmaster Central account.

Once this is done, Adwords will start collecting the search-keywords for your blog. To get at the data:

Log in to AdWords
Select "All Online Campaigns,"
Make an empty campaign (if you haven't got one already)
Go to the "Dimensions" tab
Change "View" to "Paid & organic".

AdWords will display your stats, since you signed up and linked your account. This includes the top search terms that users got to your site with, number of clicks, number of queries and some other measures too.

I'm only just starting to assess how well this will for for my song-listing site - will update this post when I have more specific information about how well it works and whether I can get actionable results from it.