An Age Old Problem
DFU recently engaged in a project with a client to develop a new sitemap, focusing specifically on restructuring the global navigation and labeling conventions. The challenge for this project was the large volume of content: thousands of pages spanning various directories and subdirectories.
The crux of the project was to conduct an open card sort remotely to determine the user’s mental model for the best labels and content groupings on the site.
We’ve conducted dozens of card sorts over the years with OptimalSort, our trusted online application for remote card sorting. Typically, we adhere to a 30-35 card limit, which is the suggested threshold for users’ ability to sort content remotely. However, even a carefully selected sample of 30 cards would not reveal enough information about the full structure of the site.
A New Approach
To balance the large amount of site content against the constraints of remote card sorting, DFU deployed three different versions of an open card sort: one with 35 cards, one with 50, and one with 65. We assumed that there would be a high level of attrition with the 50- and 65-card surveys, but in order to organize this large set of content the boundaries of remote card sorting had to be pushed.
One concern in deploying three card sorts was ensuring that the analysis would be sound. If we staggered the content randomly, we would run the risk of making false correlations across surveys.
Say for example we were organizing items in a grocery store. We’re from Texas, so we might commonly group tortillas with cheese in the 35-card survey. However, if tortillas don’t appear in the 65-card survey, we might group cheese with milk. Without providing overlap between the categories, this would imply a false correlation between tortillas and milk, when in reality, we’re just hungry for quesadillas.
True Correlation:
(Tortillas + Cheese) | (Cheese + Milk)
False Correlation:
(Tortillas + Milk) | (Tortillas + Cheese + Milk)
Accordingly, we started with a base group of 35 cards, and then added 15 pieces of content to create the 50- and 65-card surveys. Adding on to the base of 35 and 50 cards created an overlap of content, which reduced the risk of a faulty analysis.
Winning the War of Attrition
There was not a huge difference between the conversion rates of the 35-card card sort and the 65-card card sort. On average, users finished the 35-card survey only 25% more often than the 65-card survey.
Additionally, the majority of people who completed the 65-card survey provided valid and relevant results. On average, users created 9 categories for the 65-card sort. This makes sense considering the broad amount of content they were asked to organize. The point of diminishing returns occurred when participants used 5 or fewer categories. At that point, the content is sorted at too broad a level to be of use in creating a specific mental model. The same can be said for participants who used 13 or more categories, in which case the categories appear at too granular a level. Most people (75%) created 6-12 category names, which provided meaningful feedback and insight about how to organize the site content.
It Worked!
Once the results rolled in, DFU normalized the categories in each survey separately to investigate whether there would be distinct differences in the groups and category names users created.
Surprisingly, the category names created by participants in each survey were similar, and in some cases identical. After combining the results of the three surveys, DFU made solid recommendations about global category names that were then tested and validated in a closed card sort.
Expanding the number of cards allowed DFU to harness a broader level of content, rather than being limited by conventional card sort wisdom.
Why Not Skip the Small Stuff?
Although the 65-card survey was helpful in determining content groups and category labels, the 35- and 50-card surveys should still be included in this type of exercise. Deploying different-sized surveys in tandem lowers the risk associated with a low conversion rate, which is a possibility with a larger card sort. It’s better to give users an option with a smaller set of content than put the full weight of the research in a single large card sort.
Additionally, the type of content available in the card sort also influences the conversion rate. This website consisted of generalized content that didn’t require expert knowledge. If content is more technical, users will likely find the card sort more challenging.
While a staggered approach to card sorting is necessary to combat low conversion rates, this project demonstrated that users can effectively sort more content remotely than previously anticipated. In the future, when a client has a broad content structure, the blended approach can be used to examine a more robust collection of content.