Yeah, a flour and water mixture is a great breeding ground for all sorts of nasties. Fortunately, the waste product of lactobacillus is lactic acid, which lowers the pH of the starter to levels inhospitable to other nasties. But it takes a while for the pH to rise up, so other things will grow in there for a while. Although I'd be concerned if it was really nasty, since much of the badness of nasty bacteria is not the infection potential, but the toxic waste products, which won't "die off" as the acidity of the starter increases.