Between what they witnessed and their captain letting himself die, they both solved to in no way choose a kid aboard their ship except if they completely had to, which is the true cause Shanks didn't take Luffy together with him.