ICRA 2018 Main Page

Verification of Autonomous Systems

Self-driving cars, robotic home assistants, medical robots, and other diverse autonomous systems are becoming a reality worldwide. Their verification—compelling evidence that autonomous systems satisfy their requirements and behave safely—has become increasingly important. As technology progresses, and autonomous systems become more complex, with added intelligence and adaptive capabilities, the challenges of verification increase. Efforts to propose methodologies and to develop tools related to the verification or verified design of autonomous systems are increasing worldwide./p>

Our goal is to bring together experts in verification, autonomous and intelligent systems designers, and leading roboticists to discuss the main challenges in verification of autonomous systems. We will present the state-of-the-art in verification of autonomous systems in terms of methodologies, techniques, ontologies, languages and successful applications. A major goal will be to find common ground to address the challenges relating to design for verification in robotics and verified robotic designs. We hope to open new avenues for future interdisciplinary collaboration among participants and expect that this will significantly advance the state-of-the-art in verification of autonomous systems.

Intended Audience:

• Researchers interested in exploring the space of research challenges in the verification of autonomous systems.
• Researchers and developers interested in verifying their autonomous robots.

Content and Schedule

Note that this will be the third in a series of workshops at major conferences and will follow the format established in the previous workshops. The previous workshops were at ICRA 2016 and IROS 2017.

The proposed event will consist of a main speaker followed by short panelist introductions and panel discussions on sub-topics.

SESSION 1: Industrial Perspectives on Verification of Autonomous Systems

9:00AM - 9:20AM   Speaker: Dejanira Araiza-Illan, V&V Challenges in the Aerospace and Food Industries (pptx)

9:20AM - 9:45AM   Panel: Craig Schlenoff (pptx), Pulkit Kapur, Hadrien Bride (pdf)

9:45AM - 10:30AM   Discussion Notes (pdf)

As industrial exploitation of robotics grows, the need for researchers to understand the goals of industry in this area increases. This session is intended to support a lively discussion about the difference between the needs of the industrial and applications-focused community and the capabilities of the research community.

COFFEE BREAK - 10:30AM - 11:00AM

SESSION 2: Design Tools for Verification

10:00AM - 11:20AM   Speaker: Galen Mullins, Adaptive Sampling Methods for Testing Autonomous Systems (pptx)

11:20AM - 11:45AM   Panel: Signe Redfield (pptx), Zhe Hou (pdf)

11:45AM - 12:30PM   Discussion Notes (pdf)

The focus of this session is design tools and toolchains supporting development of correct-by-construction and bounded performance systems, as well as verification tools to enable verification of otherwise intractable behaviors, algorithms, and decision-making mechanisms and mechanisms to support representation of the tasks the robot is expected to perform.

LUNCH BREAK - 12:30PM - 1:30PM

SESSION 3: Verification of Learning Systems

1:30PM - 1:50PM   Speaker: Don Sofge, Verification of Learning Systems: "Trust, but Verify" (pptx)

1:50PM - 2:15PM   Panel: Robert Fitch (pdf), Will Curran (pptx), Fabio Bonsignorio

2:15PM - 3:00PM   Discussion Notes (pdf)

This session focuses on verification as it applies to more complex autonomous systems that incorporate learning and intelligence in their design.

COFFEE BREAK - 3:00PM - 3:30PM

SESSION 4: Group Discussion

3:30PM - 3:45PM   Summary of Research Challenges

3:45PM - 4:00PM   Ana MacIntosh, Assuring Autonomy (pptx)

4:00PM - 5:00PM   Capturing the Results Discussion Slides and Notes (pptx)

The purpose of this session is two-fold. The first goal is to go over the research challenges discussed in prior meetings and workshops, to categorize them into specific areas where progress has been made, specific areas where research is underway and progress is expected in the near term, and specific areas that need more work, and to identify the specific progress that has been made. Ideally, the output of this portion of the path forward discussion will be a list of specific verification tools and approaches that have been developed, and an updated list of the current open research challenges in this area. The second goal is to discuss how that progress should be captured. This will ideally include discussions about the relative merits of standards, traditional publication venues, and mechanisms such as shared repositories for aggregating links and references and making them publicly available.

Note that slides and notes are available at the links in the above agenda.

Participation

Prior to the Workshop, attendance will be requested with two considerations in mind:

• We plan to invite participants from well established research networks, both academic and industry related, since they can provide greater insight to the current challenges of autonomous systems verification, and provide some real-life first-hand knowledge on a range of applications.
• We also plan to invite early career researchers, including current graduate students, and postdoctoral researchers, and encourage the participation of persons new to the field of verification, such as robotics designers attending the main conference tracks at ICRA.

In terms of publicity and communications, we are pursuing the following courses of action:

• Personal invitations, via email, to academic and industry-based research networks and IEEE lists. These will include our own Verification of Autonomous Systems working group list as well as the Autonomous Intelligent Systems Partnership, the Robotics and Autonomous Systems Group and the EPSRC UK Network for Robotics and Autonomous Systems in the UK, the EURON mailing list in Europe, and various formal and informal international working groups like the NATO Research Task Group on Evaluating the Efficiency of Coordination Mechanisms for Distributed Mobile Sensors and ontology groups working towards IEEE standards. We expect that these invitations will also reach larger research groups in associated universities and institutions, and thus early career researchers.
• E-mails to relevant and more general robotics and verification networks mailing lists such as Robotics Worldwide Digest
• Distributing the Workshop's information through printed and printable resources. The Workshop's information will be formatted as a poster/flier, so that it can be printed and distributed to universities and companies across the world, and attached to the aforementioned e-mails.
• Call for panel participation, for up to two places in each session. The speakers and panel members will select from the applications, giving preference to early career researchers, such as recent PhD graduates and postdoctoral researchers with less than five years in the verification fields. Topics of interest include
  • Verification of autonomous systems
  • Verification research challenges
  • Verification in industry and government
  • Quantification of trust
  • Common evaluation frameworks
  • Task representation and task ontologies
  • Verification of learning systems
  • Verification of IoT devices
• Social media, including Facebook, Twitter, and blogs, to highlight the Workshop and to disseminate the web page link.
• An e-mail-based "question drop", to complement the discussions, as some interested parties will not be able to attend the workshop in person.

We will broadcast the Workshop live if the facilities are provided. Whether a live stream is possible or not, we will publish a written summary of the discussion outcomes and any released presentations to provide a posteriori engagement with both the verification and robotics research communities.

Registration

If this workshop is accepted, registration will be through the main conference registration process, at ICRA 2018

Interaction Among Participants

During the Workshop, we envision multiple types of interaction among participants over the course of the workshop.

• Formal interaction
  • Panelists will provide short summaries of their perspective
  • These perspectives will provide other participants with insightful ideas and questions to further the discussion
• Experienced:Early career interaction
  • Experienced researchers will interact with other early career researchers in the panels
  • Experienced and early career researchers will interact with participants of all levels through questions and discussions
• External interaction
  • Previously collected questions, from the suggested “question drop” format, will serve as a complement for the in person discussion, if time constraints allow it.
• Informal interaction
  • Moderators will actively encourage the participation of audience participants during the discussions
  • There will be a no-host lunch to encourage continued informal discussions between presenters, panelists, and attendees outside of the formal sessions
• Feedback
  • At the end of the workshop, we will ask the participants to indicate how they would like to move forward with the topics discussed. Options will include potential publications, future meetings, and research collaboration opportunities.

Support of an IEEE RAS Technical Committee

The proposed workshop is supported by the IEEE RAS Technical Committee on Performance Evaluation & Benchmarking of Robotic and Automation Systems (PEBRAS).

Contact

Workshop website hosted on robotistry.org