CS Colloquium: Coline Devin ’15

Details “Robot Foundation Models: Physical Commonsense, Few-shot Learning and Task Mastery” Humans have the remarkable ability to perform new physical skills from only one or a few examples, adapt on the fly to unexpected object physics and creatively recover from small mistakes by constantly sensing and replanning. Traditional robot automation pipelines have typically focused obtaining reliability by fully controlling the environment. Learning-based approaches have made significant progress in robotics, but often require large amounts of data or fail to reach high enough success rates for commercial use. Generalist AI has focused on training large-scaling robot foundation models on general human manipulation data to provide robots with broad abilities in comprehending task intent, and to adapt in real time to the unexpected variation of the real world. Their new model Gen-1.5, trained on over 500,000 hours of real-world interaction data, can learn new tasks from just a few demonstration, and, with order(hours) of data, can master a new task, exceeding 99% success rates on several tasks. Speaker Coline Devin ’15 is a technical staff member at Generalist AI. She was previously staff research scientist at Google DeepMind, co-leading the Gemini Robotics projects. She earned her PhD from UC Berkeley on compositionality in robot learning, was advised by Professors Sergey Levine, Pieter Abbeel and Trevor Darrell, and was supported by the NSF Graduate Research Fellowship.