Card sorting is great. That’s why we built OptimalSort.

But card sorting is not enough. Here’s why.

In user centered design (UCD), the process hinges on the premise that end user feedback will help refine and validate the quality of a design. We see this in practice all the time through the most commonly practiced UCD method - usability testing.

However, usability testing rarely tests the information architecture itself. It will provide some insights into labels that may be confusing or categories that failed to get users’ attention. More often than not however, findings from user testing focuses around the interaction and presentation (user interface) levels. Specific feedback on the information architecture itself is rare.

This makes sense because an information architecture is like a building’s foundations - invisible but central to the building’s strength and viability. People will tell you that they don’t like the colours on the walls or the placement of the windows, but they won’t - they can’t - tell you anything about the foundations.

Card sorting is the most popular research method for the creation of an information architecture. It provides the insight into how people organise and label information. A designer takes this information, applies his or her experience, balances the needs of the organisation and creates an information architecture. Unfortunately, that’s often where it stops.

According to UCD, this is only the start of the process. Feedback needs to be solicited, findings applied to refine the design and this process repeated as much as possible until a happy medium is reached.

Dare I suggest that the reason we don’t do this for information architectures, is because it’s too hard. Any system where an information architect is required suggests that the system has a lot of content and is relatively disorganised. Having to test the effectiveness of such a large system is daunting at best. And it’s daunting because none of it is currently automated.

Testing an IA is actually incredibly simple, albeit long winded. Assign participants with representative tasks and observe where in the information architecture they end up, and note the path they took. Do this with enough tasks and enough people and very soon you’ll notice where your structure is inefficient or if certain labels are misleading. Make changes to either, test again and keep going until you run out of time, money or both.

Unfortunately, doing this is both time intensive and somewhat mind numbing. Fortunately, computers are great at doing mind numbing things really quickly so we don’t have to.

Our team longed for a solution, and recently we did something about it. Treejack is an online software tool that allows you to test your information architecture by treating it like an online survey:

1. Upload your existing or draft information architecture.
2. Create as many representative tasks as you can think of.
3. Email the URL of this “tree test” survey to stakeholders, end users or colleagues.
4. See how well your IA performed by viewing success, confidence and speed measures for each task.

With Treejack, we can tell in a matter of hours what aspects of an IA need work, what is effective and what to change. Additionally with automated reporting, decision makers are now more interested because of the presence of data that is easy to understand.

At the high level, Treejack provides reporting of:

• Task success - did people end up in the category or categories you expected them to?
• Confidence - how sure were they? Was their path direct or indirect?
• Speed - how long did they take?

More detailed reporting shows which categories participants ended up in, the full path taken, any back tracking and the skip rate for each task.

More information can be found on the Optimal Workshop website. A free public version of Treejack will be available from the 16th February 2009.

In the last 3 months we have been gradually making infrastructural changes to OptimalSort. As a result, a number of bugs have emerged in OptimalSort.

We apologise to all our customers for this inconvenience.

We are working hard to resolve all these bugs, and are will soon be releasing fixes for most of them - along with some new enhancements.

The reason for these infrastructural changes is because OptimalSort is now a part of Optimal Workshop. Optimal Workshop is a new platform for information architecture and user experience software tools.

Instead of simply providing you with one tool, such as OptimalSort, we hope to bring you a suite of tools - all in the one place. By mid February 2009 we will have two new tools in Optimal Workshop:

1. Chalkmark. A simple tool to test the effectiveness of any user interface prototype.
2. Treejack. An innovative approach to testing the quality of an information architecture.

We are (of course!) extremely excited about both of these new tools. They represent the ability for web professionals to more easily integrate sophisticated research into their design process and explain findings to stakeholders.

We are looking forward to introducing both tools, and an improved OptimalSort, to you in the coming months.

Once again, thank you for your patience as we iron out any bugs during this transition period.

We have received numerous requests for OptimalSort to support sub-categories (particularly for open card sorts).

Due to a variety of reasons, this is quite challenging for us to implement - but we are looking into doing this. One thought that we would like to suggest, and get your feedback on, is a method that some customers have used to workaround this barrier.

The work around is this. Consider running multiple card sorts iteratively.

To do this, you first run an ordinary card sort to define the top level IA. Once you see the emerging patterns you can then create cards for each of the main categories/content types that participants have suggested. Then you can run concurrent card sorts to define the second level of hierarchy.

The benefit of this is that your analysis will be much simpler and your results much more definitive – particularly at the second level. This may not necessarily be true if you are getting no more than 10 participants but with more participants, analysis of sub-categories will very quickly get mind-boggling.

To celebrate a successful launch of OptimalSort, we are giving away 3 annual OptimalSort subscriptions.

To enter, simply:

• Email 3 people about OptimalSort. Feel free to use the text below to get started.
• In your email, cc promotions@optimalsort.com

You receive a single entry in the draw for every 3 people you refer. Enter as often as you like. Entries close 31st March 2008 and winners will be notified by email on 1st April 2008.

Here are some words you can paste directly into an email:

Hi friends,

To make information easier to find on our website, I’ve been using OptimalSort to run online card sorts.

I thought you might find it interesting. Card sorting is a great technique and OptimalSort has made it a lot easier to run card sorts. Check it out:
http://www.optimalsort.com

After months of being in “beta”, OptimalSort went live as a pay-to-use service on 1st March 2008.

If you are a registered beta user, you should have received emails from us about what this means and how it affects you. Contact us if you have questions about this.

As part of the new look OptimalSort:

• A free account that you can use forever. It is suited to smaller projects (and getting to know OptimalSort) as it is limited to 30 cards and 10 participants per card sort.
• 3 subscription options available - 30 days, 90 days and annual. More information can be found on our pricing page. (If you’re already a user, upgrade by logging in rather than signing up on this page)

There are also a number of new features we have released with this version of OptimalSort, including:

1. Secure SSL. Many customers have asked for increased security so we have introduced SSL encryption for paid customers. This option can be enabled in your Settings page and “forced” on all your users.
2. Additional administrative users. Our customers tend to work in teams and previously had to share login information. With paid subscriptions, you can now create up to 3 administrative logins to the same account.
3. Remove individual responses. It is inevitable that often some results are invalid or unhelpful. You can now remove participant results individually on the Results and Analysis page.
4. Online help.  We have tried to make it even easier for you to use OptimalSort and run card sorts. There is an online help system that provides help for every page and answers the most common questions.
5. Automated analysis. This is the most exciting new feature of all. Many of you use Donna Maurer’s open sort analysis spreadsheet. Previously that meant hours of data entry to populate her spreadsheet with OptimalSort data. We have automated that process so you can now import OptimalSort data into the spreadsheet and immediately see the results. To access this, download it from the Downloads page and follow the onscreen instructions in the Excel file.

There are many more new features planned in the coming weeks. Thank you to everyone who has written in and given us feedback.

To finish this post, we’d love to share some feedback (reproduced with permission) from a customer. We really had to share this one:

Our team was limping along trying to perform remote card sorts with mediocre software until we found OptimalSort. OptimalSort has literally changed the way that our User Experience group does business, allowing us all of the advanced card sort functionality we need to perform tests with anywhere from 15 to 200 users or more.

It’s about as close as you can come to actually sitting down with users in person, with real index cards, and observing them – with the added benefit of being able to test users all over the world, bringing more geographic diversity to our card sort results. We couldn’t be more thrilled with the results we are seeing, nor with the quick responsiveness of the OptimalSort team to any of our questions or concerns.

OptimalSort really has changed our lives. I mean it.

M.McLaughlin, Convio.com

Recently I caught up with Donna Maurer at a training event. We talked card sorting, as you do. Afterwards, I thought that an ‘interview’ with her could help with the questions we are receiving from OptimalSort users. Here’s what she had to say for herself:

Sam: So Donna, most people would have heard of you before but, for those who don’t know you, how would you describe what you do and how you’ve come to achieve celebrity status with regard to card sorting?

Donna: I work as a freelance information architect and interaction designer. I
usually work on big websites, intranets and business application -
either organising all the content (that’s the information architect bit)
or designing the pages, interfaces and way things work (that’s the
interaction designer bit).

When I was learning IA, I heard about card sorting and I got the
impression that it was the ultimate (and easy) technique for creating an
IA. So I tried it and had all sorts of problems. I started blogging
about my problems & questioning the value of the technique.

But as I did more IA work and gained a better understanding of
classification and categorisation theory I started to think about how to
use card sorting more effectively. I realized that card sorting is a
good support technique for the category development process but is not a
replacement for it. Now I don’t rely on card sorting as the only
approach, but one method among many to help me organise things.

Sam: One of the biggest challenges with card sorting is analysis and making sense of results. Where do you suggest people start?

Donna: At the beginning

Seriously, analysis starts before the card sort starts. You have to know
what you want to learn and run the card sort in the right way -
involving an appropriate number of participants and cards. It is
wasteful to run a detailed, remote card sort with hundreds of
participants if you just want to get some initial insights into how
people think. And it’s just as silly to involve 6 people and a small set
of cards if you want to explore all the ways content can be grouped.

If you plan up front, analysis does become easier. In the first example
(when you want some initial insights), run an in-person team-based sort
and don’t collect more data than you need. It then is fairly easy to do
some qualitative analysis and identify the main insights. For the second
example (exploring content groupings in detail), you need to get
organised up front and store your data in a way that lets you do more
in-depth analysis - either with a statistical tool or something like my
card sorting spreadsheet. That way you aren’t overwhelmed with so much
data that you can’t identify patterns.

Sam: Do you ‘eyeball’ the data? I’m sure this can mean different things to different people (and I bet experience counts for a lot). Any practical tips and guidelines on how you ‘eyeball’ data? What do you look for?

Donna: Some of the ‘eyeballing’ process (where you mainly look at what you have
collected) is to spot the ‘interesting’ things - the things that you
didn’t expect to see or that surprise you. It is also to spot the things
that support your ideas. It isn’t about coming up with a structure, but
finding insights and patterns.

You’re right - it becomes easier with experience - you have a better
idea of what to expect and can notice the ‘interesting’ things more
easily. But even without experience, a card sort will help you learn
things you otherwise wouldn’t have.

Sam: Many people expect that card sorts produce site maps. What is your response to that? And what are the outputs from one of your card sort projects?

Donna: My response to that is that you should never let any technique do the
work for you. A technique is a tool to help you learn things, not create
answers.

Sure, a card sort can produce a sitemap - you can collect data, run the
outputs through a statistical tool and make a dendogram - a hierarchical
site map. But you shouldn’t just use that as the sitemap for your
website. The card sort doesn’t consider the business goals of the site,
what users need to achieve and how they are likely to work. It just
shows you some ideas for grouping content - this is hugely valuable,
just not the only thing you need to consider. You have to put that
together with many other things to create a good structure for a site.

Sam: There are a lot of people who love the idea of a card sort but find it hard to convince the powers to be to let them run a card sort. Any tips on how to do this?

Donna: I’m a bit cheeky sometimes. I tell my clients that they can let me make
it up and take the risk that I get it wrong and have to start over. Or
they can let me get some user input, be more likely to get it right the
first time and be able to demonstrate why it is right. That goes for any
type of user research, not just card sorting.

Thanks Donna for the ‘interview’, and some insightful comments. As an aside, Donna is in the process of writing a book on card sorting. I’ve been privileged to read some of her draft chapters - and it’s good! Watch this space. I understand it’s due to be out in late 2007.

Dendrograms and cluster analysis appear to be the defacto for card sort analysis. The popularity of IBM’s EZCalc and USort beta software has further popularised the notion that cluster analysis is the best (and only?) form of analysis for card sorting. (In fact, I’m not sure - is it because of EZCalc that we consider cluster analysis as the only form of analysis?)

We’re not sure this is right. Or at the very least, that there has to be other more useful ways of analysis.

We get asked constantly why we don’t have dendrograms. We have three reasons:

1. Our first and most simple reason is this - we’re not convinced it’s helpful. We have run numerous card sorts in the last 4 years. Not once have we found dendrograms to be particularly helpful. Maybe it’s because we’re not particularly good statisticians, or our confidence intervals were too high. Whatever it is, our results would just not make any sense. So, we gave up on dendrograms. (This is not to say others have not had good results with dendrograms!)
2. Our second objection to dendrograms is more ideological. Dendrograms are too convenient. These three things, stick them together and you have a category (but only if you accept a 90% confidence level and a total of 5 categories). It feels like dendrograms may be popular because they give easy answers. A quantitative answer to what is often a qualitative question. We think it is dangerous to expect card sorts to provide concrete answers. At best, they help you realise you’re asking the right questions.
3. Lastly, we’re not sure about dendrograms because we believe it may hide what is really going on. “Modes” can exist in data. Take for example business customers and consumer customers of a website. Often, two groups of people with different goals are going to give you different perspectives. Entering their data into a mathmatical function can tend to ‘average out’ the results - leaving you with something that does not match the real world. (In much the same way as an average of 3.35 bedrooms on a street of houses doesn’t mean much to anyone).

So there it is. That is why we don’t (currently) have dendrograms. We’re happy to be convinced otherwise. As a parting comment, consider this - why is it that when asked, most people admit to ‘eyeballing the data’ as the analysis method for card sorting?

ps. You can still export our results into Excel or a statistical package and create dendrograms should you wish.

One of the biggest drawback of remote card sorting is qualitative feedback.  When running a physical card sort, a facilitator would typically prompt participants to explain their thought processes and be able to seek clarification.  Of course, that’s no longer possible when you run remote card sorting.  Or is it?

It has been suggested to us that we consider implementing functionality to allow for moderated sessions.  When we looked into it we found it is quite simple to use some existing tools in conjunction with OptimalSort to do this.  Here is how:

1. Sign up for a free account at Gotomeeting.
2. Sign up for a free Skype account.
3. Recruit your participants and organise a time and date.  Give them your Skype user name and  make sure they have a microphone/speakers.
4. Set up a meeting on Gotomeeting.  Send your participant a link to meet with you online.
5. On your computer, have a browser ready on the welcome page of the card sort project you’d like the participant to complete.
6. Once your participant is online and you’re able to talk to them over Skype, you can “hand over controls” to your participant.  This allows them to control your computer and provide comments as needed.  You will also be able to see everything that they’re doing.
7. The participant will complete the card sort while you observe and take notes.

We’ve used this approach a few times as a fast and cost effective way of moderating sessions.   Of course, it tends to ‘bias’ the participant set as typically only more saavy web users would know how to set up Skype on their computer.  However, this minor drawback is often worth the advantages that this method offers.

We’ve had a few people ask us why we’re doing this. The answer’s quite simple - we wanted some software that we ourselves could use.

But more than that, we wanted to walk the talk. Here’s an extract from our latest newsletter:

Creating something that people love to use is easier said than done. Doing it with limited time and resources is even harder. But this is the task we set ourselves 12 weeks ago.

As consultants we provide expert advice, but we seldom get the opportunity to be fully involved in the implementation process. We are aware that it is easier to give advice than to act on it.

So, in order to walk the talk, we decided to design, build and release a new online card sort tool. We set ourselves a small budget, set a 10 week timeframe and formed a team of two to make it happen. These constraints proved to be one of our most valuable assets. The constraints forced focus, focus helped reduce unnecessary complexity and required our decision making to center around our users.

This exercise taught us a lot about what it means to juggle conflicting needs and objectives. Most importantly it also confirmed to us the importance of goal directed design and the necessity to stay ‘intimate’ with users.

Some insights we gained include:

- The value of ‘checking in’ every week. User testing twice a week meant that we had few surprises left when we launched. With the simple (and free!) web tools Gotomeeting.com and Skype, we conducted user testing with people from all around the world. The investment in time was quite literally 2 or 3 hours a week but the insight and peace of mind was invaluable.

- Quick iterations save time. The sooner we learnt we should not be spending time on something, the better. With the feedback we got from our user tests, we were constantly tweaking and refining. I’m sure it drove our developer a little crazy at times, but it meant we were able to keep to our budget and release on time.

- Low tech can be just fine. A lot of time can be wasted in “pretend planning”, or cleverly disguised procrastination. Often the most effective solutions are remarkably low tech. For us, four large sheets of paper and Post-It notes worked perfectly well for tracking our development progress. It took 10 minutes to set up and cost virtually nothing.

- Small can be good. A bigger team and large budget is not necessarily the most effective (or only) way to deliver on projects. A small but well organised team can be surprisingly effective. Sitting next to each other, a shared calendar and regular structured communication were our biggest assets in our team.

The finished product, OptimalSort, was released two weeks ago at the Usability Professionals Association conference in Austin, Texas. OptimalSort significantly reduces the time required to set up, run and analyse results from card sorting as it is all done online.

We have been pleasantly surprised by the response. Hundreds of users have signed up and thousands of participants have completed card sorts in just two short weeks. It seems that creating something people love using is well worth the effort.

OptimalSort has now been in beta for around two weeks. We’ve received an overwhelming response in that time. Many of you have been giving us great feedback and loads of questions. So to start addressing some of these questions, and to let you in on progress, we’ve started this blog.

We’re looking forward to sharing our ideas and learning from you as we continue to make OptimalSort something you enjoy using.