Usability Monitoring – Extending Quality of Service Monitoring for Decision Making

The paper presents a new concept, Usability Monitoring, and applies it for situational awareness applications in military Command, Control, Communication, Computers, Intelligence, Surveillance and Reconnaissance systems (C4ISR). Usability Monitoring means taking measurements of technical Quality of Service (QoS) parameters in end systems and comparing them to target values of reference cases. The concept differs from QoS monitoring in the goal and in the placement of measurement points: In QoS monitoring the goal is to verify that the network provides the promised service quality in the user system - network interface. QoS monitoring does not capture the actual end user experience, which is influenced also by the end system, and therefore it does not directly correspond to the service quality that a user sees. Usability Monitoring has exactly this goal. The Observe-Orient-Decide-Act-Loop (OODA) is a decision making concept that is widely used in the network-centric approach and it emphasizes fast decision making. The presented model for Usability Monitoring is based on the OODA-loop. It includes QoS measurements not only in the Act-phase, i.e., can the user perform the actions s/he wants or are there delays and losses that make the system less usable, but also in the Observe-phase, i.e., does the user get the information s/he subscribes to, and in the Orient-phase, i.e., does the user get confusing information and cannot orient, and also the Decide-phase, i.e. is the user able to make right decisions based on the previous steps. The measured technical QoS parameters are connected to user experience by Mean of Score (MOS) functions that are obtained by usability tests.

A case study for partially evaluating the Usability Monitoring concept is from MNE5 MSA (Multinational Experimentation 5, Maritime Situational Awareness) Experimentation Event 3 that was conducted in partnership with the Navy Command Finland, Naval Warfare Centre of Sweden and NATO Allied Command Transformation (ACT) and the Singaporean Armed Forces (SAF) Future Systems Directorate. In the MNE5 MSA experimentation we were able to monitor end user experience, how the user sees the services and is able to work with the current tools and capabilities. In this article we describe how Usability Monitoring was addressed in the MNE5 MSA case study: meters for Usability Monitoring were selected and we investigated what aspects of usability affect the phases of the OODA-loop.