1. With updated prototypes from Matthew (July 21 versions)
   1. Heuristic evaluation
   2. Usability testing

Heuristic Evaluation

• [H1] Visibility of system status
• [H2] Match between system and the real world
• [H3] User control and freedom
• [H4] Consistency and standards
• [H5] Error prevention
• [H6] Recognition rather than recall
• [H7] Flexibility and efficiency of use
• [H8] Aesthetic and minimalist design
• [H9] Help users recognize, diagnose, and recover from errors
• [H10] Help and documentation

Issues Identified

• “Home” state button is potentially unclear, assuming tool opens to available software. Conceptually either available or installed could be the home state. At this level is there even a need for the home button? [H1, H4]
• Unclear as to what these are categories of – user must visually explore the left nav to further their understanding of what is being presented. [H6]
• Categories may not appear clickable. [H1]
• Do available software and installed software have the same right nav pane? If so, this will likely introduce confusion. In the case of installed software, it will require navigating through categories, when it would be simpler as a list view. [H7]
• Lower left white space is unused, which adds to minimalist design but may be repurposed to handle non-vital content that would otherwise be adding complexity to right pane. [H7, H8]
• Top-level category page should not scroll if possible. [H6]
• # of packages is not highly visible and is not associated with relevant elements of the UI (the categories). [H2, H4]
• Packages is jargony. [H2]
• Icons may not be appropriate for the “available software” and “installed software” menu items, as they may not be universally understood for these functions. [H2, H5]
• File/edit/view/help menu bar seems overcomplicated and unnecessary for this system. [H8]
• “Most popular” pane should be limited so it does not scroll, scrolling in two panes adds unnecessary effort. [H7]
• “Most popular” text should be more detailed, i.e. – most popular in games software. [H5, H6]
• List of software should have its own header to clarify what is being presented. [H1]
• Forward/back buttons may be useful as breadcrumbs may not be inline enough to facilitate easy back and forth browsing. [H3, H7]
• Detail page might have a better way of communicating canonical supported or not beyond simply text that would likely be ignored - green light, yellow, red? Along with text? [H5]
• Install button should show state when clicking, changing to in-progress or installed with uninstall button. [H1, H3]
• In progress - relationship between downloading and waiting items may not be clear to users as it requires and understanding of how packages relate to each other. [H2]
• Pause all and resume all may not make sense if they cannot all be downloading at the same time. In that case, only a stop/start downloading button (that changes label based on state) is required. [H4]

User Testing

Summary (and some paraphrased quotes) from interviews with users between 7/22 and 8/11, following script from SeasonOfUsabilityWeek6Notes

Start page prototype

• All participants noticed the right-side category section first
• Everyone mentioned not knowing that the "available software" section was selected (and one thought the categories section was grayed out). Assuming that is largely because of the low fidelity of the prototype but a stronger visual connection would be useful.
  • "The available software jumped out, but I didn’t realize it was selected."
• Seems to be a relatively good grasp of what "available software" is, although one thought it was installed software.
  • "Available software is software that the system has that it can share - software that is already on or available to be downloaded"
  • "The different programs installed on the computer"
• Lack of association between “available software” window and what would be considered the home page (and accessed via the home button). Relationship needs more clarity.
  • "What is selected when you click to the homepage?"
• Searching: "If you knew the name of the software, you would just search for it." One user questioned - "Is the search box automatic because there is no button?"
• Consistent confusion around what "x number of packages" means in this context, even among those with Linux experience.
• File/edit/view/help - one user questioned what they might do and if they were relevant to this tool:
  • "Is view a search feature? View could say show me all the software with the word X in it. But a robust search bar would do the same job. Is view by date or by type or something? Most recent used, recently added – other ways of filtering and presenting the data. No edit, doesn’t make sense. Just need to be able to close it, File doesn’t make sense either."

Browsing and detail page prototypes

• Some issues around the logic behind the selection of the most popular items.
  • "Why are these the most popular? Most popular how?"
• Top scroll bar is confusing, there are two separate scrolling windows with a lot of information, they should have their own identity.
  • "I am not clear about relationship between first and second one. Not clear whether it is a subset or the rest of the list."
  • "I am not sure whether it is all or most of the games. Not sure what the division is between top and bottom section. Not sure what is in bottom section."
• Liked the simple browsing path to get to the item’s page (only a few clicks).

"In progress" prototype

• Participants felt it would list what is being installed
• Clear understanding of cancel. Some confusion or incorrect impressions around what “waiting” means.
  • "Not sure about “waiting” how do you turn it back on? Can it only be done one at a time? Or is it pause? How come there are only pause all buttons but no way to start up the waiting one?"
  • "...waiting for information that it needs to download. Unless there is a queue within the in progress that limits the amount at one time and then it is waiting for others. “waiting” – something is wrong? Might say something more informative. “next in queue”, then “in queue” for the rest. It will happen after other things happen."
  • "“waiting” implies order for download. Like a queue. Control how many, bandwidth, force start all at the same time if you want."
  • "[I am] used to downloading things simultaneously, so why would something be waiting."
• Some felt there was just enough info there to make it clear, others suggested more information and means of control for longer lists of downloads, while another user suggested moving the entire functionality to within the left-hand navigation so it was always visible.
  • "...if there were 15 things it might be better to organize them in a tree or something, so you could collapse and expand. Since you may be downloading a ton of stuff since it’s freely available. Should have headers to be able to sort by date added, etc. Might be like the huge list of seeding torrents. Don’t have to worry about all the stuff that torrents do, but sorting and categories are important."
  • "It would be cool if the bottom had a utorrent style info on active downloads, speed etc, real basic network activity. So you could see the activity as you are browsing."
  • "In progress is an active area you [should be able to] see it while navigating"

How would getting software happen? What happens when you click install?

• User responded with a mix of scenarios, from the typical install process (where you get a “save as” pop-up, then run the install external to the tool). Others correctly assumed it would be added to the in progress queue.
  • "Connects to a web-based resource, like a webpage, to find the software"
  • "Click install, button fades/changes into “added to queue”, behind the scenes it gets added to the in progress list. You stay on the detail page, nothing else changes. It’s free so you’re just going to keep browsing and clicking. Especially if it is just going to queue up to the list. Make sure the button changes immediately."
  • "If you click install, an installation window would come up to download, then it would start up in the in progress section."

How would you get to this tool?

• "Desktop icon, maybe it could be the desktop? You wouldn’t have to log in to get to it. It would be a standalone application."
• "Maybe right at the clock always at the top. And the drop-down ubuntu menu. Like in apple, when you go to software update."
• "Through the control panel… or maybe an icon on your desktop."

What would you call it?

• "Appstore, but that’s already taken. And you’re not buying stuff so that doesn’t seem right. It seems one-way for getting not sharing, and not buying so store isn’t right anyway. Associates “application” with office stuff and not games."
• "Software manager? But only if you can install and uninstall."
• "Add and remove software options"
• "No idea... 'software' can be intimidating"

Design recommendations

• Conceptually differentiate “most popular” by more clearly bounding it (or moving it to the side).
• In progress needs to increase clarity around “waiting” state, by using numbered queue or other representation (or wording) around fixed order.
• Visually link left nav selection more tightly to right panel (one surrounding color or more tabbed appearance)
• Clearly associate “home” state with available software view (?)
• Category headings at the top level need to be more descriptive
• Available software may be clearer as “get software”
• Move # of packages information in-line with the categories and represent numerically only (without mention of “packages”).