Google re-ranking and personalized search study

Once upon a time me and a few cohorts wondered about just what levels of flux there were in the Google SERPs. Has personalized search really changed the consistency of rankings? It’s an issue that has been spoken about many times in the search world. We set out to see what was up. After two rounds of research we noticed that this was unlikely to be the case. You can learn more in the post; The SEO guide to Google personalized search

It seemed only sensible, given Google’s Psearch expansion, to have another look. And so last December we started a third round. Has anything changed? How much flux is out there? Well, read on and find out what we’re seeing so far…

Google ranking flux and personalized seach study

Now, first things first; the goal wasn’t really to establish how personalized search operates. What we really wanted to know is how much movement is there in the SERPs for a given query type, in a given region (USA in this case). Yes we’re also noting some potential personalized search affects, but movement/flux was the core observation.

We must understand that even though there is some interesting data here, it is by no means a large enough sample to get nitty gritty with IMHO. It is also important to remember this is a specific task from a quasi-transactional query session (task development).

We want to go back and do more from an informational query space and one more suited to respondents (familiar, such as ‘learn SEO’ etc..). This approach, last time out, did show more movement than we saw with this space (we’ve done this round 2x now).

As always, no magic bullets here… Just more links in the chain (pun intended of course).

The Set up

We decided to take a set of queries to build a session (task) in a space that may or may not be familiar to the respondents. The queries we used were;

‘antique lamps’
‘buy antique lamps’
‘buy lamps online’

We also asked them to tell us;

Where they’re located (region/country)
Which browser they’re using
Cleared search history lately?
Is personalized/search history on?
Most common Google app used
If they had searched for ‘lamps’ or furnishings in the last 60 days

The goal being to start understanding what may or may not be influencing the search result flux. While we had some questions to be answered, there was no initial bias (hypothesis) behind the analysis. It should also be noted that the entire collection process only lasted 4 days so that we could try our best to avoid any temporal anomalies.

Today we’re looking at the data from the US. At the end of the test period there was really only enough to do analysis on the US and UK markets. I shall post the UK data in the coming weeks.

US Respondent Stats;

Browsers;

FireFox (FF) 88.46%
Chrome (GCRM) 7.69%
Internet Explorer (MS) 3.85%

Most used Google app;

Gmail (GML) 53.85%
GoogleTalk (GT) 11.54%
Analytics (GA) 11.54%
YouTube (YT) 11.54%
Webmaster Tools (WMT) 7.69%
Google Reader (GR) 3.85%
iGoogle HomePage (iG) 0

Cleared Cache lately?

Yes (Y) 42.31%
No (N) 57.69%

What have we learned?

It’s a good question, one that I don’t think we have a definitive answer to. Let us remember this is but one query type in a limited set. Personalization is more likely to have an effect in a query space where the user is more active in. That being said, there were some interesting elements that emerged so far.

Much like the original round of testing, a year ago, we can say that there is no massive upheaval where SERPs are vastly different from one user to another. This much has been consistent in each of the (3) rounds in both informational and transactional query spaces.

Here are some notes on other questions we’ve had;

Are the top 3 stable?

In the initial task the top 3 placements were rock solid. It was on the secondary queries that extended the task that they would become somewhat instable, (see ‘concurrent tasks’). Now, it wasn’t huge, but there was some small movement that became more prominent with each related query performed.

Ultimately not much has changed as far as aiming for the sweet spot (above the fold). These are easily the most stable listings and it is unlikely these would be affected more than 95% of the time and even then, rare to drop too far down when they do.


	Are top 10 stable? Yes. While the movement always would be found from #4 on down, they did for the most part remain fairly consistent in content if not ranking. As was noted with the top 3, the more tasks in the chain (queries) the more instability would creep in. I’d say it is more common to see a #4 move further below in the top 10, not altogether off of the page. We also noted that 7-10 are at the most risk and the 4-6 spots more consistent as far as a presence in the top 10. Many times it would be a plus/minus 1 or two ranking spots.

Where is the biggest re-rankings?

As noted so far, the biggest area of flux was on the expanded task elements. The first query ran, in most cases, was fairly stable. It was on further related queries in the session that higher levels of instability were noted.

We’re considering running the test again while reversing the order to see if indeed the order of the session queries created the increased movement the further we went into the process. It does seem like this is an important element to watch for in future testing.

What causes SERP flux?

I really have no conclusions at this point. We’ve likely not hit a trigger often enough to see any deeper personalization in play. I believe the next round, using more familiar queries and of the informational nature, there could be stronger Psearch elements present (as there were in the first round, late 08).

For the moment, in case you hadn’t guessed, the actual ongoing task development seems to be an important element. Much of the more gradual movement could easily be attributed to various servers the locales were hitting at the time. Many of the other elements we looked at (related searches, Psearch on/off) didn’t really seem to have a strong bearing in the SERP flux.

My guess? Query analysis will play as important a role as the Psearch does. There does seem to be a task related SERP adaptation from what we’re seeing in this round. The jury is out though until we know more.

Secondary observations

What do strong listings have in common?

Certainly the money listing is #1-3 as these seem to be the most resistant to movement. What was also interesting is that the odd 8th place or 5th even would remain more solid than some above and below them.

Indented listings would often be vulnerable but other than that, the higher the ranking, the more security one would have. Beyond that we’d need to actually dig into each of the sites for further clues.


	Is related searches past 60 days making a difference? Not any more than the avg. general flux. There doesn’t seem to be a huge difference or much of one at all from those with recent related searches to those without. I thought this was surprising as there was some expectation of this playing into SERP flux… It didn’t seem to make much of a difference ultimately.


	Are concurrent tasks creating a difference? This is the most active and interesting area that we’re seeing. All three queries the respondents ran for us were related to the same task. We were essentially building a task towards a goal. It seems that with each concurrent search, we’d see more and more flux.

Are browsers making a difference?

We’re seeing some flux in IE, FF and Chrome. Of interest is that the Chrome SERP was a model SERP (consistency wise). I really can’t see this as being a major element, but we would probably need more data for a conclusive decision.

So far, I’d say that this doesn’t play into it….or at least isn’t much of a signal of note.


	Is Psearch on making a difference? This one I still have to say not really. At least not in the traditional sense of having surfing history enabled. Actually, some of the odder SERP flux came from those that weren’t actively browsing with it turned on. If there is a strong Psearch in play, it wasn’t due to past related searches.


	Is regional location creating flux? While there is some flux, there was no major insertion of listings that could be considered regional. This being said, we’d likely want more regional data and more targted queries to better establish this to be the case.

Other notes;

PPC and Uni results. – it was also interesting to see how the Shopping results popped in and out. While it is beyond the scope if this research, they were interesting and far more irregular than the SERP flux. This does play into one’s placement in the top 10 and is something to look at down the road.

Indented listings came and went (on final query) – another finicky element were the indented listings. It seems those can be more finicky than the average flux/movement in the SERPs. Looking for this in the next round and digging deeper if it persists is another area of interest.

Click data – given the suspicions of query analysis and task related triggers, in the near future introducing random clicks on links (and return/bounce?) might be worth putting into the mix. We haven’t given many signals beyond merely a 3 query task. If we get more evidence of this being an element, deepening the tasks seems a likely next route (past further rounds like this one).

What does it mean for SEOs?

Once upon a time this journey began because I really wanted to know if ‘rankings are dead’ and if ‘everyone gets a different SERP’. A few of the age old, but poorly understood, questions of the search world. So, have they been answered? Well, sort of.

Rankings are dead? – Yes and no. The old days of ranking reports telling us one ranks in X-position, seem to not be a valid approach. Should we be tracking/watching rankings? Most certainly we want to be getting top 10 and then above the fold to ensure the best performance.

I’d also suggest that you might occasionally get people in a targeted region to check SERPs for you now and again.

Everyone sees a different SERP? – Yes and no. The rankings are reasonably consistent as far as the make up of the top 10. Even the movement isn’t massive when there is some.

But are they ALL the same… most certainly not.

We shouldn’t be changing how we approach things ultimately… Above the fold is the real estate that’s prime (what else is new?). It may be the measuring that we will have to adapt. You will need to find ways to check rankings from a few locales and discern a mean average instead of a definitive placement.

What’s next?

We really have only touched the surface so far (with this round) and are going to want to start looking at a few more things including;

Informational – next round we’re returning to the original informational query which is in a more familiar query space which is likely to produce some different effects.

Navigational – we may also consider running a round on a navigational query space to see how it reacts. I am more interested in the informational/transactional spaces, so time permitting on this one.

Reverse order – if the trend continues with concurrent searches (task development) we will want to reverse the order on one to see if that remains the case

Click data – obviously once we get past all of that, we are likely to start introducing click data into the mix to see what effect that has on things.

Temporal watching – another point brought to me by a information retriever was that any changes to Google’s processing would likely be incremental. This means we’ll re-test in 6 mo to see if things have changed.

As you may have started to notice, there are still a ton of elements which are unknown. This is why I am still not making any major calls on what we’re seeing. Please bear in mind that we’ve only 2 rounds last year and one recent set of data. This really only begins to give us some insight; nothing more. The last thing we need are SEOs running around taking any of this as gospel at this point.

In the short term we’re going to sort the UK data from this round and I will be posting about that in the coming weeks… Stay tuned.

Log In

Our Sites

Home - About Us - Consulting - SEO Reports - SEO Programs - SEO Packages - Request for Proposal - Contact Us

TOP

The Reliable SEO services company