Few points, from my limited knowledge as a civilian:
First of all, what AlaskaMike said is worth repeating: "stealth" doesn't mean invisible; it means hard to pick up and track.
When a stealth aircraft flies by you, you could see it if you were looking hard enough. It would just be a tiny blip for a fraction of a second, whereas a non-stealth aircraft would be a big blip that is easy to track consistently.
Could you track tiny blips with a good radar? Sure. But then you'd be tracking EVERYTHING in the sky for dozens or hundreds of miles around you that reflects such a tiny amount of radar signal -- birds, insects, random objects, etc. If your radar shows you that much clutter, it's useless, which is why they all have filters so that they show only the stuff that's not likely to be a bird.
If the stealth aircraft is REALLY close to your radar station (or infrared sensors, or eyeballs in a clear sky, or ears when all is quiet), then yeah, you'll pick it up. That's why a major piece of "stealth" is good mission planning (e.g. not flying right over top of things that could detect you).
The B-2 in particular also has the advantage of being able to fly at very, very high altitudes. Even if you could pick it up, it's that much less likely that your planes or missiles could ever get to it before it's out of range.
The SR-71 had a bigger radar signature than a B-2 and a MUCH larger heat signature, but flew even higher and MUCH faster, so the result was similar. Yeah, maybe you can detect it, but even if you can, there's nothing you can do about it.
The F-117 (not F-111) that was shot down had flown the same relatively predictable path at about the same time for a few consecutive nights. People had caught glimpses of it, so they knew what to look for and where to set up their sensors. Pretty sure this mistake hasn't been made since, not to mention that other stealth technologies have gotten MUCH better since then.