We’ve improved the Search Analytics calculations

Earlier this year we released a new Search experience in Happeo that enhanced the overall search experience for users and made it easier to find results. We’ve now improved the Search Analytics calculations to match the changes in the Search behaviour of our users. Read more below.

Here’s a quick reminder on what was introduced with the new Search experience release:

• New search bar
• Quick filters
• New search view
• Search sections and Search filters
• New search algorithm

You can learn all about the new search here. 

After this release, a few of our customers have noticed a decrease in their search success rates. We’ve investigated this, and it turns out that Search is actually a victim of its own success. Read more below.

What happened, in short:

In our Search Analytics, we categorize user searches into three main groups: successful searches, failed searches, and refined searches. These categories together account for 100% of all user interactions with our search system.

Now, with our recent update, users can type a query and get a result that shows up in the search bar itself, leading to an increase of refined searches. This means that more users are modifying their initial search queries to find what they're looking for, simply because it’s more user friendly to do so now (we saw an increase of about 33% in using the search bar to find answers instead of the search results page). 

However, there's an issue with how we define failed searches. It turns out we're being too generous when determining if a search has failed, or even if it’s refined. This generosity leads us to classify certain refined searches as failed, and successful searches as refined. This is why you’ve likely seen your search success percentage decrease since this release. In reality, some of those searches that were initially labelled as failed are actually successful when you consider the refined searches that followed.

For example: 

• Changing “Produkt” to “Product” will count as a failed search. This should be a refined search.
• Changing “Product” to “Product Roadmap” will count as a refined search. 
• If you type “Pro” and results show, then you get distracted for 5 seconds, then next you add “duct” to it to form “product” and click on a result, today it will count as two searches, one refined and the other successful, instead of just one successful search session.

In summary, while it may seem like our successful searches have decreased, it's mainly due to a change in how we categorize and account for user interactions, particularly in the refined and failed search categories. On average, we’ve seen a 10% increase in refined and a 5% increase in failed searches across customers.

The solution

1. Success and failure together will make up 100% of total searches. Refined searches will be calculated separately.
2. We will modify the concept of a “refined” and “gave up” query according to the current behaviour.
   
   - “Gave up” will mean that a user opened the search bar, typed in the query, did not click on any results and left. It can also mean the user did nothing for 2 minutes or longer.
   
   - “Refined” will mean that a user started a query and after that changed that query. If you change the query to a different word completely, it won’t count as a refined search.
3. We will change the Overview section in search analytics to be based on Search Sessions. Terms will still be based on Search Queries. This means you will see your average time to search go up, in some cases rather dramatically – it will however, be more accurate, as a search session encompasses all the queries someone used to click out to their final result.
4. You will still be able to access your old search-logic data. We’ll introduce a dropdown in Search analytics, which will be available until the 1st of April 2024. This allows you to see the data from before this change, although the old logic cannot be applied to new data. After the 1st of April, you can still access that data, but it will have to be through a CSV report which you can download.

Glossary

Search sessions: one search session can have multiple queries in it. Open search bar starts a search session

• Success - a user typed a query (or multiple) into the search bar, found what they were looking for and clicked on the result
• Failure - a user typed a query (or multiple) into the search bar, didn’t find what they were looking for and left

Search queries: any term that you look for in search 

• Success / Opened item - user typed one query and clicked on the result
• Refined query - a user typed a query into the search bar, didn’t find what they were looking for and changed the query (following an algorithm to determine the similarity of the queries) 
• Failed / Gave up query - a user typed a query, and left the Search without clicking on anything, or didn’t do anything for 2 mins