Quote:
Originally Posted by madwi
(Post 3750617)
It was DNS wasn't it.
:icon17:
|
TL:DR - moron with rf guns screwing with IP configurations because he held a job as a help desk tech for a pc repair shop a decade ago. Massive loops.
Long Version - Last week, my Access Points started going down, producing latency and overall being a mess. I chalked it up to be a crappy Arruba network with tons of deadzones and just general crap hardware. I reset the AP's which is what really started the issues beginning.
Once restarted, all RF guns began conflicting with other equipment on wifi with my network. Straight to the MAC addresses. The conflicts, pending where they were.... began to create loops with small switches I have deployed in the building because finding proper RJ jacks is a nightmare here.
I was never aware of this though because my main switches are all unmanaged and straight out of 1990. I had no way of knowing the amount of loops that were happening. At the same time, i had no way of knowing the MAC problems because the RF Guns are running Windows CE 5.0. Once the loops started breaking main equipment connected physically to my network, everything basically started shutting down.
I got techs on site and we began disagnosing literally everything. We tested the switches, my stacks, the core, the fiber itself, the AP's, the jacks... basically everything that had electricity.
Once all of that failed, I was finally given control of the faking firewall and stacked switches. I performed a stupid amount of packet traces via WireShark and noticed packets either being chopped in half or just completely getting lost.
With that figured out, i began looking at the UDP of where they were coming from. Most stemmed from the RF guns. By that time, i had lead techs on site reconing my network way better then i ever could with more equipment I could ever hope to afford. They came back with the same diagnosis. So we began to pull each patch from each port and see what what went down, what stayed up, and if anything was less stressed.
3 main switches later, the bare network was working properly. So we are now in the process of migrating everything to managed Meraki switches with proper tracking of everything so I will have a full scope of my network here.
That's about where we are as of right now. Slowing bringing things back online, changing main infrastructure, and just generally pissing off every single user. It's awesome.
My only question is.... who needs a new IT guy? I can't keep up at this rate. I'm still running on literally just coffee and red bull. Last actual meal was over 48 hours ago at this point.
