Overview

“It works really well!” is the expected conclusion of academic publications, technical reports, and project outcomes that seek to demonstrate a designed system (i.e. robot, autonomous vehicle, controller) is functionally sound, safe and/or trustworthy. In this workshop, we dissected and examined under a critical eye each component of the phrase:

• the “It”, namely how the system-under-test has been defined for verification and validation purposes, and how this meets the needs for robotics in sectors such as autonomous vehicles and medical robots;
• the “works”, namely the definition of the system’s tasks or behaviours, including systems that might learn and at industrial scale;
• the “really well”, namely the formulation of metrics, evaluation tools, reproducibility and replicability frameworks, and extensibility to an “industrial grade”; and, furthermore,
• the ability to openly report “failures”. The workshop will bring together experts in robotics, verification and validation, end-user experience, regulations and standards, formal methods, assurance and certification, and industries including healthcare, transportation, aviation, military, and automotive.

We had three sessions with invited talks followed by Q&A, and a fourth session with an interactive panel. We opened new avenues for future interdisciplinary collaboration among participants and expect that this will significantly advance the field of verification of autonomous systems.

This was a hybrid workshop.

Intended Audience

We invited anyone interested in verification of autonomous systems to participate, including

• The new generations of roboticists, new to the field of verification and validation.
• Experts in verification and validation who are exploring challenges in emerging robotics fields, developing state-of-the-art tools and techniques
• Experts in human-robot collaboration, who are studying shared workspace domains and their implications to human safety.
• Experts in evaluation of robots and systems, who can provide insights on opportunities for the development of metrics, methods and tools applied to robotics.
• The community writing norms and standards to guide the development of safe and trustworthy robots and autonomous systems.
• The community working on field robotics and industrial applications, to provide feedback to the academy and to foster new collaborations between industry-academy to tackle challenges.
• Researchers interested in exploring the space of research challenges in the verification of autonomous systems.

We also invited all members from the following IEEE RAS Technical Committees:

• Verification of Autonomous Systems (all co-chairs are workshop organizers)
• Software Engineering for Robotics and Automation (co-chair is workshop organizer)
• Software Engineering for Robotics and Automation (co-chair is workshop organizer)
• Performance Evaluation and Benchmarking of Robotic and Automation Systems (co-chair is invited speaker)
• Cognitive Robotics (workshop organizers are TC members)

Content and Schedule

The workshop is a joint collaboration between the IEEE RAS Technical Committees on Verification of Autonomous Systems and Software Engineering for Robotics and Automation. The workshop brought together experts beyond robotics, in the areas of verification and validation, end-user experience, regulations and standards, formal methods, assurance and certification, and industries including healthcare, transportation, aviation, military, and automotive. Our main goal was to discuss, with a highly self-reflective angle, the state-of-the-art in the domain of verification of autonomous systems, in terms of requirements, techniques, tools, architectures, frameworks, performance and evaluation metrics, reproducibility and replicability, and real-life use-case needs. A major goal was to build common ground on how verification and validation have been applied successfully and unsuccessfully to date, what lessons have been learned from it, and what is yet to be addressed. Additionally, we opened new avenues for future interdisciplinary collaboration among participants and expect that this will significantly advance the field of verification of autonomous systems.

The workshop consisted of four sessions. The first three sessions started with four short invited talks that positioned different perspectives of academy, industry, government and end-users, related to the session’s main topic. After the invited talks, each session had a Q&A from the audience. The workshop organizers moderated the sessions for time keeping and inclusivity.

The last session consisted of a panel of three experts, who addressed topics regarding key challenges in the field of verification in the context of robotics. This included an open-ended discussion with the audience and pre-selected key questions.

The diversity of expertise encouraged a cross-domain perspective in challenges and applications for the future and emergent intelligent and autonomous robotic technologies. The invited talks covered state-of-the-art challenges, developments, combined efforts and collaborations relevant to the topics of each session. Furthermore, the flow of the invited talks ensured a good mix of academic and industrial points of view, as well as theoretical and practical approaches. This encouraged the participants to learn, ideate and conceptualize throughout. Questions were collected digitally before and during the event. An online chat was made available to continue with offline discussions. Notes from the event will be made publicly available after the event as they are received, along with slides and any other produced materials.

Note that this will be the fifth in a series of workshops at major conferences. The previous workshops were at ICRA 2016, IROS 2017, ICRA 2018, and ICRA 2022.

Note that slides and notes are available links in the above agenda and on the TC-VAS official website.

Participation

Awards for travel funds were available to early career researchers and researchers from underrepresented communities.

• We plan to publish a special issue based on the proceedings of the workshop. .
• The available slides from the invited talks will be published openly on the TC’s official IEEE RAS website (Activities section).
• The discussion notes from the sessions and panel will be made available on the TC’s official RAS website (Activities section).
• Available slides and discussion notes will be posted on the Workshop webpage.

Registration

Registration will be through the main conference registration process, at IROS 2023 Registration

IEEE RAS Technical Committee Support

The proposed workshop is supported by the IEEE RAS Technical Committee on Performance Evaluation & Benchmarking of Robotic and Automation Systems (PEBRAS) and the IEEE RAS Technical Committee on Verification of Autonomous Systems (VAS),

Workshop website hosted on robotistry.org