Texas A&M University manually fixes thousands of campus devices affected by CrowdStrike outage

COLLEGE STATION, Texas (KBTX) – The CrowdStrike IT outage forced Texas A&M University to cancel classes Friday as IT professionals manually fixed thousands of impacted devices.

The outage was caused by a faulty update sent out by U.S. cybersecurity firm CrowdStrike, which manages security for nearly 30,000 companies worldwide.

It was a routine software update gone very wrong.

After the update went out, across the world people turned on their computers to find the ‘blue screen of death,’ an error in the computer’s operating system that causes a system crash.

That error can only be fixed by deleting the update file, which IT professionals are forced to do manually, one computer at a time.

”We had to boot the machine in something called safe mode, which is a very limited Windows level, so you’re actually trying to load the minimal amount of data that the PC needs to run. At that point you can go in and delete the file that CrowdStrike told us to delete,” City of College Station Chief Security Officer Sam Rivera said.

At Texas A&M, its IT department has been working since 4 a.m. to fix the issue.

Peter Lange, the university’s Vice President of Operations, said as of Friday afternoon about 80% of their servers are back online, but questions still remain.

“There should have been a whole bunch of testing and clearly, clearly there wasn’t,” he said.

Lange expects Texas A&M to resume its normal operations over the weekend. As new student conferences are currently going on, taking the pressure off of the academic side allowed them to get needed computers up and running.

“We had to get computer labs where they register for those classes back online ASAP,” Lange added.

Texas A&M has around 700 people in its IT department, manually fixing around 1,000 computers an hour.

Lange warned users about possible phishing attempts in the coming days. According to him, there have been reports of scammers posing as CrowdStrike to gather personal information.

”Give me your credit card or give me your personal information and we can fix it. That is not happening. Nobody from CrowdStrike is reaching out to do anything. Talk to your IT guys,” he said.

While there were concerns about security and data, Rivera said their devices seem relatively unscathed.

“First thing goes to, ‘Who did this and how bad are we going to be affected?’ And today I think we were, I was pleasantly surprised that there wasn’t any data loss at all for us,” he said.

Rivera, who has been an IT professional for over 40 years, plans to do a deeper dive of his own on the faulty file. According to him, CrowdStrike sends out routine updates every 15 to 30 minutes.

This one just went wrong.

“I can’t speak for them, I know that they test out major updates but these are not, this wasn’t a major update,” Rivera said.

Rivera is confident the City of College Station is back up and running, with everything back to normal by Monday.

The firm has since apologized and sent out a fix for the update. According to a statement put out by CrowdStrike, the outage was not a cyberattack or a security breach.