I think what has to happen is a government standard autonomous system driving test. A system has to pass dozens of tests to be certified for use on a public road. Also the system needs an accident recording system to verify what it did before the accident was in compliance with the testing. If it failed to work right, manufacturer gets sued. Failed because of poor maintenance or improper driver override, driver/insurance gets sued.
Right now I suspect these systems are hesitant to slow down and go into a potential avoidance mode because it makes the system seem "jumpy" or nervous. If I see someone on a bike approaching the road perpendicular at a high rate of speed even 100' away I will atleast get ready to do something if needed. If its a kid dragging their feet and looking scared then obviously I would stop or slow long before they got in front of me. I'd guess right now, an automated system does a bad job of identifying when something coming up looks wrong, especially approaching the road perpendicularly. Or it can't recognize when it doesn't have enough visual cues to maintain its current speed safely. Sure lots of people drive less safely than is optimal too of course, but they can't be engineered to be safe where an automated system can.
Right now I suspect these systems are hesitant to slow down and go into a potential avoidance mode because it makes the system seem "jumpy" or nervous. If I see someone on a bike approaching the road perpendicular at a high rate of speed even 100' away I will atleast get ready to do something if needed. If its a kid dragging their feet and looking scared then obviously I would stop or slow long before they got in front of me. I'd guess right now, an automated system does a bad job of identifying when something coming up looks wrong, especially approaching the road perpendicularly. Or it can't recognize when it doesn't have enough visual cues to maintain its current speed safely. Sure lots of people drive less safely than is optimal too of course, but they can't be engineered to be safe where an automated system can.