Areas mainly addressed: Robotics and AI/Big Data Value, Ethical, Legal, Social, Economic implications of robotics, Benchmarking 

Main questions to be answered: 

· What actions can we take to involve more European Robotics projects within the ERL framework? 

· What activities do we need to introduce into European Robotics Benchmarking competitions to increase impact and relevance? 

· How can competitions foster progress in benchmarking and research in Robotics? 

WS Description: 2nd part of the Workshop: panel discussion on the topics presented in the 1st part. We will add to the discussion the relevance of robot competitions for benchmarking and research. 

We know Benchmarking will be crucial to a successful, safe robotics industry. The European Robotics League benchmarking methodology is applied in other EC funded projects and we anticipate further large Smart City Robot Competitions to demonstrate robotics in relatable and realistic events. As the ERL becomes self-financing, what actions should we take to increase our reach and relevance? New modes of competition? New ways of interacting with the public as participants in benchmarking activities? New models of funding for teams and travel, prizes and awards? 

We expect to produce a white paper to direct development of European Benchmarking through Competitions, with the contributions of workshop attendees. 

Intended outcome: Workshop expected to create a roadmap or white paper, Workshop discussion topics of common interest, success stories, use cases, etc 

Approach: Trustable and Dependable Intelligent Robots for Smart Societies 

Subtitle: Research Competitions, Benchmarking, Societal and Ethical Issues 

We propose the following: 

• Before the workshop we gather opinions from the communities involved in benchmarking about how reproducibility and benchmarking can be realized and made useful to support European robotics 

• We then gather opinions from that community, and communities involved in funding, legal standards, etc (as far as this is possible and reasonable), in answer these questions 

• We will use POLIS (pol.is)or a similar tool to promote synthesis of knowledge from these opinions, and build an emerging consensus 

• We will share this as a draft white paper with the community before the workshop 

• The workshop will consist of a series of short provocations – briefings by panellists on topics linked to this. There will then be a round table discussion of the community consensus to propose a route map for future work. The pitchers will be the union on 

• After the workshop, the final draft of the white paper will be published. 

Before the workshop we will spread by mailing lists, social networks etc. a number of crucial thought provoking and outright provocative questions and challenges to the community with the objective of raising awareness and stimulating contributions on the topic. Thought provoking video from recent robotics competitions will be shared before and during the event. 

Hybrid format 

PART II Agenda 

– second 80 mn: 

– panel with all of speakers + audience, moderated by the 4 organisers 

(panel discussion contributing to define a roadmap) 

Contributors: Matteo Matteucci, Enrica Zereik, Davide Scaramuzza, Agnes Delaborde, Isabel Ferreira, Tony Downs (NIST), Avrin Guillame, Ricardo Sanz, Antidio Viguria Jiménez, Alin Albu-Schäffer , Francesco Ferro 

Organisers: 

· Matthew Studley, matthew2.studley@uwe.ac.uk, Bristol Robotics Laboratory

· Pedro Lima, pedro.lima@tecnico.ulisboa.pt, ISR/IST, University of Lisbon 

· FabioBonsignorio, fabio.bonsignorio@gmail.com, FER, University of Zagreb and Heron Robots 

· Diego Torricelli, diego.torricelli@csic.es, CSIC

Project cluster relation: Yes: EUROBENCH, SciRoc, Metrics, Natural Intelligence, Coresense … 

Areas mainly addressed: Agri-Food, Maintenance and Inspection, Agile Production, Robotics and AI/Big Data Value, Ethical, Legal, Social, Economic implications of robotics, Cybersecurity in Robotics, Regulation and Standardization for Robotics and AI, Control technology including virtualisation and edge/cloud 

Main questions to be answered: 

· Reproducibility, benchmarking, standards 

· Objective measures of robustness, reliability and agility 

· TRL roadmap for new products 

WS Description: The reproducibility of research results and the objective and operational comparison of different methods is still not mainstream in Robotics, AI and Automation Research. However, in particular thanks to the EU Commission significant progress has been done, This will be a gathering of the TG on Benchmarking and Competions. The SotA will be presented and the road ahead discussed. Info about currently completed and starting projects and results willl be shared. The meeting is open to all. 

Intended outcome: Workshop expected to create a roadmap or white paper, Workshops engaging non-roboticists to understand needs/goals (policy, commercial, technical), Workshop discussion topics of common interest, success stories, use cases, etc, Workshop on recent developments in technology or applications 

Approach: Trustable and Dependable Intelligent Robots for Smart Societies 

Subtitle: Research Competitions, Benchmarking, Societal and Ethical Issues 

This is the first part of a two-parts workshop. 

We propose the following: 

• Before the workshop we gather opinions from the communities involved in benchmarking about how reproducibility and benchmarking can be realized and made useful to support European robotics 

• We then gather opinions from that community, and communities involved in funding, legal standards, etc (as far as this is possible and reasonable), in answering these questions 

• We will use POLIS (pol.is) or a similar tool to promote synthesis of knowledge from these opinions, and build an emerging consensus 

• We will share this as a draft white paper with the community before the workshop 

• The workshop will consist of a series of short provocations – briefings by panellists on topics linked to this. There will then be a round table discussion of the community consensus to propose a route map for future work. The pitchers will be the union on 

• After the workshop, the final draft of the white paper will be published. 

Before the workshop we will spread by mailing lists, social networks etc. a number of crucial thought provoking and outright provocative questions and challenges to the community with the objective of raising awareness and stimulating contributions on the topic. Thought provoking video from recent robotics competitions will be shared before and during the event. 

Hybrid Event 

PART I Agenda 

– first 80 mn: 

– intro + summary of the white paper (MS): 15 mn 

– 10 pitches by invited speakers of 5 mn each + spare time: 65 mn 

(presentations + Q&A) 

Contributors: We have a long list: Matteo Matteucci, Enrica Zereik, Davide Scaramuzza, Agnes Delaborde, Isabel Ferreira, Tony Downs (NIST), Avrin Guillame, Ricardo Sanz, Antidio Viguria Jiménez, Alin Albu-Schäffer , Francesco Ferro 

Further information: Will follow up and put on the web the WS as soon as approved. 

Organisers: 

· Fabio Bonsignorio, fabio.bonsignorio@gmail.com, Heron Robots and FER, UNIVERSITY OF ZAGREB

· DIEGO TORRICELLI, diego.torricelli@csic.es, CSIC 

· PedroLima, pedro.lima@tecnico.ulisboa.pt, ISR/IST University of Lisboa 

· Studley Matthew, Matthew2.Studley@uwe.ac.uk, BRL, University of the West of England