Somewhat amusing Cisco bug

[OVERKILL]

Figured you guys might get a kick out of this one:

So I was installing a 1921 ISR G2 today at a satellite location. Requirements are pretty simple: Internet, WiFi, phone service (VPN). I pre-configured the unit last week and dropped it off, but didn't have time to activate the setup, so that was on the docket for today.

Connection is 100/100 fibre, but authentication, like with Bell's service (this isn't Bell) is PPPoE. Not a big deal. Usually.

Get everything configured, link won't come up. Odd. Figure maybe it requires CHAP instead of PAP, give that a shot, nadda.

OK, triple-check the username/password, they are correct. Huh. Try PPPoE directly from my MBP, comes up whammo, no problem. Ugh.

Turn on debugging. Nothing weird with authentication, OK, PAP is correct. But it is hanging up somewhere on the ipcp process, never gets an IP.... But it works fine on the laptop.

So, figuring it might just be an IOS bug, I hop on to the Cisco website, grab the latest IOS release for the ISR, fire it onto the TFTP server, grab it, set it active, then reload. Boom, link comes up immediately. Wasted a solid hour troubleshooting the stupid thing only to find out it was shipped with a buggy image. Wasn't overly pleased, but did find the fact that they would ship it like that, which would leave somebody utterly screwed without a SmartNET contract to get the newer image, amusing.

 

[simple_gifts]

j/k

 

[JHZR2]

You sure it wasnt counterfeit?

 

[OVERKILL]

Originally Posted By: JHZR2
You sure it wasnt counterfeit?

100%.

Cisco sometimes ships buggy software. This is the first time I've actually been impacted by one of the bugs though.

 

[webfors]

heh, Cisco and bugs... yep.

We have a 2960 that reboots every time you plug into the console. Fixed with latest firmware. Not fun when in the middle of a simple change implementation and you cause a reboot of a primary device (yes, we have no remote mgmt of these switches lol).

 

[FordBroncoVWJeta]

I purchased a new HP laptop last summer. It would always turn off after 3-4 hours of use randomly. HP claimed nothing could ever be found wrong and could never duplicate it (of course). Then I got on HP and found a new BIOS update that dropped in March. Finally fixed it. Really disappointed me that HP couldn't figure out why my laptop was randomly restarting. After 9 months me computer doesn't randomly restart.

 

[wwillson]

Every version of IOS has bugs. At work we spend millions of dollars regression testing IOS before it can be run on PE and P routers. I believe we find more bugs than anyone else in the world. I did find a PE a while ago that has an uptime of over 10 years, so IOS can be reliable.

 

[OVERKILL]

Originally Posted By: wwillson
Every version of IOS has bugs. At work we spend millions of dollars regression testing IOS before it can be run on PE and P routers. I believe we find more bugs than anyone else in the world. I did find a PE a while ago that has an uptime of over 10 years, so IOS can be reliable.


Yup, a quick glance at any of the open and resolved caveats lists can be quiet amusing

IOS can be extremely reliable, particularly if the usage profile isn't impacted by any of the open caveats. I have a few ISR's that are at the 4 and 5 year marks. Realistically, nobody has bug-free firmware, but the way Cisco documents them, it certainly makes them see prolific, though I doubt in reality they are any worse, and likely, probably better, than other vendors.

 

[Jooksing]

Can't be Cisco's fault.... Blame walmart... I kid I kid.

Guys I work with installs $4k radio equipment. I would guestimate 1 out of 300 does not work right and needs a firmware flash to get it going. Sometimes they can't be fixed and get sent back to the manufacturer.

 

[brianl703]

Originally Posted By: OVERKILL
Wasn't overly pleased, but did find the fact that they would ship it like that, which would leave somebody utterly screwed without a SmartNET contract to get the newer image, amusing.


That wouldn't be covered under warranty?

Sounds amusing. About as amusing as Dell sending 4 broken switches in a row. Maybe Cisco does the same stuff, wouldn't doubt it. I worked for a place that had Cisco server that needed a replacement hard drive...what did they send....a 2 year old drive. Probably had some bad sectors on it, they zero'd it out, called it good, and sent it out as a replacement.

What I find amusing is that I have a better chance of getting a reliable replacement part buying it on Ebay than I do direct from Dell.

 

[OVERKILL]

Originally Posted By: brianl703
Originally Posted By: OVERKILL
Wasn't overly pleased, but did find the fact that they would ship it like that, which would leave somebody utterly screwed without a SmartNET contract to get the newer image, amusing.


That wouldn't be covered under warranty?

Sounds amusing. About as amusing as Dell sending 4 broken switches in a row. Maybe Cisco does the same stuff, wouldn't doubt it. I worked for a place that had Cisco server that needed a replacement hard drive...what did they send....a 2 year old drive. Probably had some bad sectors on it, they zero'd it out, called it good, and sent it out as a replacement.

What I find amusing is that I have a better chance of getting a reliable replacement part buying it on Ebay than I do direct from Dell.



I've never had a defective piece of Cisco Enterprise gear. I've had a few SPA phones, but those are basically a re-branded Linksys product. Generally their hardware is excellent, but as Wayne and I discussed, IOS can have some interesting bugs. The saving grace is that usually the bugs are obscure or require some very specific circumstances not experienced in typical use to get triggered. This is one of those instances where that wasn't the case.

Yes, they would probably be able to get an image through Cisco's support system. But that would be a drawn-out affair. With an active SmartNET, you just login and download, takes seconds. If this was a time-sensitive deployment, waiting on support would be a huge PITA, particularly for something that is a glaring defect in the software.

I don't have any dealings with DELL directly, but many of the healthcare facilities I work with do. I've seen workstations come out of the box and not work, parts missing....etc. So your experience doesn't surprise me unfortunately. I would categorize my experience with HP Enterprise as better than what I've observed with DELL.

 

[nthach]

I remember setting up a Cisco 851W with a bridged AT&T DSL connection(before U-Verse) and it was an OK ordeal as long as you didn't need "static" IPs, which were sticky IPs in AT&T speak. The PPPoE details lived on the 2Wire gateway in bridged mode, and it was an matter of setting up the WAN interface on the Cisco.

Now, if you needed to get a sticky IP working it was an messy process. I hate AT&T.

 

[OVERKILL]

851W is going back! That's an old girl. I had an 851 that I pulled from a clinic years and years ago, thing had an amazing uptime, probably close to the decade or so Wayne mentioned earlier. It was a VPN endpoint for a satellite ultrasound location.

Generally with DSL and a modem in bridge mode, the PPPoE details live on the WAN-facing device (like the Cisco in your instance) and the bridge is just an intermediary media converter with the PVC/VCI details setup on it to allow the authentication and IP details to be exchanged to bring the connection up. Now, a local ISP used to provide a Cisco 867 w/integrated DSL that they would setup similar to the manner in which you've described, it did the DSL subscription and then you used a static IP on your piece of WAN-facing gear with the device performing the bridge also acting as the gateway. So you were responsible for NAT/PAT, you just didn't have to worry about PPPoE.

 

[nthach]

I think I recalled setting up a Cisco 827 with PPPoE on the router itself, I had to enter the username/password to get it to authenticate with AT&T and pass packets. But that was a simple DHCP single-IP line.

It's been a while since I touched Cisco anything, but now AT&T moved on from PPPoE to 802.1X-based authentication for U-Verse ADSL2/VDSL2+. I think there's someone on DSLR or Reddit who managed to grab the certificates from a 2Wire/Pace gateway and got a 3rd party modem to work, or did some creative routing to only allow the U-Verse gateway to pass management traffic.