Richard Boulton home

Weeknotes 9

18 May 2014

This was a great week.

At the start of this week we released the new version of site search for GOV.UK. This gets rid of the old tabbed interface, and introduces our first "facet" for refining searches. There have been a few small teething troubles, but mostly it's performed as we hoped; it's working at least as well as the old interface, but is a much more standard search interface so we'll be able to build on user interface patterns that people are used to. As you'd expect, this is just the first of many iterations, so it's important that this has laid a nice stable foundation. We're now watching the analytics and feedback from users closely to determine what areas of ranking need work most urgently.

I spent my final two days on second-line support mainly working on getting a plan for upgrading elasticsearch in place. We have several different apps using the same elasticsearch cluster, so any upgrade basically means doing a lot of testing first. On Monday, I was finally fairly confident that we'd managed to make all the apps work with either elasticsearch 0.20 or 0.90, so we upgraded the preview cluster to a recent 0.90 release.

Unfortunately, we couldn't easily perform the upgrade by building a new cluster and then switching to it, because we'd have to make sure that updates during the switchover process from all of the different apps talking to the cluster were not lost. We decided the likelihood of us making a mistake during that process was just too high, so instead opted for an upgrade-with-downtime. Fortunately, because we didn't need to upgrade the apps at the same time, we managed to get the downtime for this to about 1 minute.

We upgraded the cluster on our staging environment on Wednesday, and after that had worked for a day with no problems, we upgraded production first thing on Friday. Thanks to our caching layer, searchers performing high volume searches won't have experienced problems during the downtime, so very few people received an error as a result.

Working on second line was a very interesting insight into how the rest of the GOV.UK teams work, but I was glad to get back to my "home" team for the second half of the week.

I spent the rest of the week working on making our search wrapper incorporate data from the new search-admin tool so that we can force results for certain queries. I tried various strategies for doing this, finally coming up with one which does all the re-ranking in elasticsearch, but has negligible performance impact. This tool will give us great power to fix search issues quickly, but needs careful monitoring so that we don't break rare searches by forcing results for popular searches to the top of them.