SAVR: Selective Audio Visual Relevance for Explainable Coalition Situational Understanding

Abstract Coalition situational understanding (CSU) involves the use of artificial intelligence (AI) based assets to assist human decision-makers, operating at or near the edge of the network, in responding to rapidly-changing events. CSU exploits data collected from sensors of multiple modalities, for example, visual and acoustic. A user will often need to rely on an AI asset created by a different coalition partner; therefore, the assets must be capable of generating explanations for their outputs to engender user trust. Moreover, the edge setting requires explanations to be as specific and targeted as possible. We demonstrate a novel technique—selective audio visual relevance (SAVR)—designed to support CSU in real-time, edge settings, by providing effective and resource-efficient explanations for AI-based activity detection from videos with audio soundtracks. We illustrate the applicability of SAVR as part of a real-time CSU ‘dashboard’ for monitoring events in an urban terrain setting.
  • Liam Hiley (Cardiff)
  • Harrison Taylor (Cardiff)
  • Jack Furby (Cardiff)
  • Katie Barratt-Powell (Cardiff)
  • Alun Preece (Cardiff)
  • Dave Braines (IBM UK)
  • Richard Tomsett (IBM UK)
  • Supriyo Chakraborty (IBM US)
  • Prudhvi Gurram (ARL)
Date Sep-2020
Venue 4th Annual Fall Meeting of the DAIS ITA, 2020