SolarWinds Network Breach Case Study Part 1

In this episode of The Cybersecurity Insider podcast, host Yigal Behar talks about the infamous SolarWinds Network Breach Case Study with his guest, Seth Melendez, president of Waregeek Solutions, an IT solutions provider specializing in cybersecurity and other IT issues for businesses of all sizes. 

Together, Yigal and Seth dissect the SolarWinds breach issue as well as dabble into other noteworthy cybersecurity topics.

A Cybersecurity Crisis Unfolds

Yigal sets the stage for the podcast by spotlighting the recent cyber events involving FireEye, SolarWinds, and Microsoft. 

Yigal values Seth’s technical expertise and believes their open dialogue will help them understand the situation and provide valuable insights to their listeners. 

Yigal then presents information from Cybersecurity and Infrastructure Security Agency (CISA) alerts to guide their conversation and help them better understand how to assist their customers. He and Seth start to discuss SolarWinds or FireEye.

According to Seth, the full details of the cyber incidents are still emerging, as FireEye conducts a post-mortem review. He commends FireEye for their transparency and willingness to investigate themselves despite being a victim of a compromised software company. 

He commends their commitment to isolating the investigation team to ensure unbiased reporting. He contrasts this with SolarWinds, from whom he expects no public acknowledgment or transparency. 

Seth contrasts the differing responses of FireEye and SolarWinds to the cyber incident.  FireEye’s transparency, while potentially harmful, benefits their clients. Conversely, SolarWinds’ silence hurts numerous government and corporate vendors who installed their compromised software. 

Seth also uses an analogy of finding a burglar’s gloves in one’s home to illustrate the potential for a major breach even without visible evidence. He clarifies that SolarWinds has 18,000 out of their 33,000 customers who received the compromised update.

SolarWinds’ Orion Platform Is More Than Just Monitoring

Yigal then shifts the conversation to the SolarWinds Orion platform, which affects all mentioned versions. He explains that Orion is a suite of tools that allows enterprises to manage their network, comparing it to a plumber ensuring the smooth flow of water or sewage. The platform provides functionalities like traffic monitoring, CPU and memory utilization tracking, and configuration management. 

The impact of the breach goes beyond monitoring, as the Orion platform allows for actions based on the monitoring data. Yigal recommends disabling and uninstalling the services, reinstalling Windows, and rebuilding machines from scratch. 

While restoring from backups before the update is an option, Yigal suggests rebuilding everything as the safest course of action until SolarWinds releases a stable version without vulnerabilities.

Going Back To Basics

Yigal suggests going back to basics, especially after the FireEye incident. He insists on the need for internal reviews to protect their own network and that of their customers, as they are part of the supply chain. He points out the risks associated with using third-party vendors and the importance of due diligence in ensuring the security of their services.

Yigal advises changing all passwords—especially admin passwords—to 30 characters and implementing separation of duties. Administrative accounts should not be used as email addresses. 

Seth agrees, sharing his experience at a large corporation where standard and admin accounts were strictly separated. He believes that even in small IT departments, separating duties and access is necessary for security, preventing accidental or unauthorized actions.

Yigal shares an anecdote about his wife asking for the admin password on her laptop to install software. He firmly explains to her that she doesn’t need it and shouldn’t have access to it. He reminds her of a past incident where she had more rights and made a mistake. Yigal also mentions how he educated his wife on identifying spam messages and emails, and now she is better equipped to recognize them. 

Protecting Against Human Error

Yigal and Seth then discuss the importance of protecting oneself from their own devices and mistakes. They joke about disliking the waiting room feature in a Zoom meeting, but they know it’s there for safety. 

Seth believes in fixing and restoring things after a problem happens instead of just trying to prevent it, so he shares his method of creating an install repository for clients that allows them to install software only from validated sources.

This approach prevents unauthorized installations from USB drives or CDs. However, it still requires an account with admin rights to install the software. Seth suggests creating a separate domain account and revoking access after installation. This may only be feasible for some but he argues that software repositories exist for internal use, citing examples like Microsoft, Google, and Apple stores. He says these repositories validate software and enable automatic installation with specific rights.

Whodunnit? Cracking The SolarWinds Network Breach

Yigal and Seth—still deep in conversation about the SolarWinds network breach—note the likely culprits behind the attack, with fingers pointing toward Russia or China. They believe that it’s difficult to pinpoint the exact perpetrator, as it could be any one of numerous countries or even smaller hacker groups.

The focus of the conversation then shifts to the implications of the breach and how to prevent similar attacks in the future. Seth says that understanding the vulnerabilities that were exploited, rather than dwelling on the identity of the attackers. He likens it to a burglary, stating that the priority should be fixing the broken lock rather than solely focusing on catching the burglar.

They dive into the technical details of the breach, noting that it had been ongoing for months, possibly even years. The sophistication of the attack and the patience exhibited by the perpetrators suggest the involvement of a nation-state with ample resources. The podcasters express a sense of urgency in addressing the broken systems and protecting against similar attacks.

Rethinking Penetration Testing Strategies

Yigal and Seth discuss the limitations of penetration testing. Seth points out that clients often restrict pen testing to specific entry points like a website, which may not be the most effective approach. He suggests that there are better ways to assess security vulnerabilities, where a website breach doesn’t always mean data is at risk.

Yet, he warns of a huge risk: even if the website itself holds no valuable data, malicious code could be injected. This code could then infect users when they visit the compromised page, creating a broader threat beyond the immediate target. Seth stresses that while a company might believe they are insulated from risk, their customers could still be harmed.

Supply Chain Attacks In A Growing Threat Landscape

As the conversation progresses, Yigal describes a shift in the threat landscape. He mentions a tactical change where attackers no longer go directly from point A to point B. Instead, they might navigate through multiple points, leveraging indirect paths to reach their ultimate target. 

This method echoes the concept of supply chain attacks, which Seth argues have been recognized and discussed widely in recent years. Seth draws parallels between modern cyber threats and historical intelligence tactics. 

He cites how intelligence agencies, particularly Russian ones, have long used indirect methods to infiltrate systems. For example, he illustrates how spies might target a seemingly innocuous person, like a maid at an embassy, to gain access. This strategic thinking, he asserts, is not new; it has been a part of criminal and intelligence operations for decades.

He laments that the IT security field has largely overlooked these age-old strategies. Seth believes that understanding these methodologies is best for developing strong defenses against current cyber threats, as they reflect a blend of traditional intelligence tactics and modern technology. 

Third-Party Risks

Even the best companies can be hacked. Seth Melendez uses the example of a top company, like the Yankees of the business world, getting hacked through a less secure vendor. This shows how even successful companies can be at risk due to the vulnerabilities of those they work with.

According to Seth, it doesn’t matter how well a company protects itself, it can still get hacked if the companies it works with have weak security. This shows how connected businesses are today—one company’s problem can become another’s.

He expresses concern that organizations often place too much trust in their partners without sufficient verification. This leads to the mantra of “trust, but verify,” which he insists should be the guiding principle in cybersecurity. Seth advocates for a zero-trust approach, where every third-party connection is scrutinized and validated. He explains that when introducing any external software, particularly network monitoring tools, companies must ensure these tools have the least amount of access necessary to perform their functions.

Seth warns that even monitoring tools can pose risks, as they can collect sensitive data that might be exploited. He discusses the protocols involved, such as Simple Network Management Protocol (SNMP) and Windows Management Instrumentation (WMI), which are commonly used in network management but can also be manipulated if not properly secured.

To stay safe, companies need more than just monitoring. They also need to double-check everything to make sure their systems are secure. This means spending money on extra tools and people to make sure nothing bad is happening. It’s really important to be careful and have strong checks in place to stop problems from spreading to other companies.

The Human Element In Cybersecurity, Layered Security & Recovery

Yigal poses the question to Seth about the easiest way for attackers to gain entry into a system. Seth believes it often comes down to people and culture, citing that technology alone cannot be the sole focus of security. He explains how even with strong technology, human error, and cultural shortcomings can still be exploited, much like a door lock that can be broken.

To address this, Seth says it’s important to have many layers of security, like an onion. This means having different ways to protect data and devices, not just by stopping attacks, but also by finding them quickly and having a good plan to get things back to normal. 

Yigal chimes in, introducing the concept of layered security, often referred to as the “onion approach.” He explains that this involves creating multiple layers of protection around data and devices, not only to shield them but also to detect intrusions. He maintains that detection mechanisms, such as remediation and recovery, are a crucial part of this layered security strategy.

Seth then uses his website as an example, saying he focuses on fixing things fast if something goes wrong, rather than trying to stop every single attack. “My biggest thing is that I have that ready to go. We have that ability to monitor and detect that. But if they do something like that, I can recover quickly. I don’t have to wait, and then I can figure out, looking at my logs, who might have been affected or who might have been, you know, suspect to that…you know, subjected to that. So, it’s that methodology is what we want to go through is how do we recover from something quickly, right? And test that ability to recover quickly.”

Cybersecurity Challenge For Small Businesses

Yigal shifts the focus from large corporations to smaller businesses, questioning whether a small company with limited IT resources would have the time or budget to conduct thorough testing and recovery exercises. He acknowledges the realities faced by these businesses, emphasizing that while ideal security practices exist, not everyone can afford them.

Seth accepts Yigal’s point and stresses the importance of honesty and transparency. If businesses are aware of the risks but choose not to invest in testing and recovery, they become part of the problem. He points out that even he, as a cybersecurity professional, only has backups dating back nine months; this poses the challenges of maintaining comprehensive backups.

Both podcasters agree on how important it is to have backups of data, especially when hackers lock files and demand money to get them back. They also say that it’s important to have data archived offline, so they can’t be hacked too.

They also agree that companies need to focus on both stopping hacks and being able to fix things if they do get hacked. They think it’s important to spend more money on resolving issues quickly because it’s impossible to stop every breach or attack. 

Yigal points out the imbalance in cybersecurity spending, where many companies prioritize firewalls and antivirus solutions over detection and response mechanisms. He observes that most of the budget is allocated towards prevention, leaving little for detection and remediation.

Antivirus, AI & SOCs

The conversation turns to the effectiveness of antivirus software, a topic they have debated before. While Seth has always been skeptical about antivirus, Yigal has now come around to Seth’s point of view, with a caveat.

Seth insists that antivirus is essentially useless, but Yigal clarifies his stance. He agrees that antivirus is mostly ineffective against unknown or “zero-day” threats, which are created by a small percentage of highly skilled hackers. However, he believes that antivirus can still play a role in detecting and dealing with known threats, which make up the majority of cyberattacks.

Seth notes that security companies have become much faster at responding to new threats, particularly with the advancements in artificial intelligence (AI) and the involvement of security operation centers (SOCs). These companies can now quickly identify and release solutions for malware and viruses within hours of their emergence.

Yigal brings a dose of realism to the conversation, sharing that not all news is good news. He points out that even with advanced technologies like AI, much of threat detection still relies on older methods like heuristics.

Seth explains how recent security breaches were discovered not through sophisticated algorithms but by analysts noticing unusual activity within their software. Whether it was unexpected network traffic or anomalies in data patterns, something didn’t seem right. This triggered an investigation, leading to the discovery of the breaches.

Seth uses this as an example to reinforce his point about antivirus software. While he acknowledges that antivirus can be a useful tool, he emphasizes that it’s just one layer of protection. Relying solely on antivirus is a risky strategy, as it won’t catch everything. He compares it to relying solely on a front door lock to secure a house. It’s a good security measure, but it’s not foolproof.

Both Yigal and Seth agree: don’t put all your trust in antivirus. It’s one tool in a larger security arsenal, and a layered approach is essential for comprehensive protection.

Limitations Of Traditional Antivirus

Yigal shares a recent experience where he transitioned clients away from a popular antivirus solution due to its inability to detect even known threats. This move resulted in an influx of security alerts for the clients, which surprised them as they hadn’t received any alerts in the previous three years.

Yigal attributes this to the limitations of signature-based detection engines used by many antivirus solutions. These engines solely focus on known signatures and fail to detect other suspicious activities like unusual communication patterns, IP addresses, or changes to files and processes.

AI & ML For Better Threat Detection

To address this, Yigal stresses the importance of Security Information and Event Management (SIEM) systems, which are enhanced by AI and machine learning (ML) capabilities. He specifically mentions endpoint protection software that monitors user behavior, learning their habits and flagging anomalies like unusual activity outside of normal working hours.

This way of using AI helps security experts see more possible problems by looking at how people usually use their computers and noticing when something is different. If something seems strange, it gets sent to experts who look into it more closely and tell the user if there might be a problem.

Yigal says that this new technology is a good way to address the problems of old antivirus software. This shows how things are changing—instead of signature-based detection, people have a more holistic approach that leverages AI and ML for better threat detection and response.

Power Of Data Correlation & Threat Hunting

Yigal reveals that he has a new solution to share, prompting Seth’s curiosity. Yigal explains the importance of correlating data to enhance security measures. He describes a scenario where an IP address accesses multiple assets within a network. By tracking these activities, he suggests that one can identify patterns and anomalies that might indicate suspicious behavior.

Rather than asking if or when a breach will occur, organizations should adopt a mindset that assumes they have already been compromised. Yigal articulates that by accepting the possibility of being hacked, teams can proactively engage in threat hunting, a practice that involves actively searching for signs of breaches or vulnerabilities within their systems.

This way of thinking pushes cybersecurity experts to look for indicators of compromise so they can act quickly if something happens. Yigal’s ideas show how cybersecurity is always changing, and we need to be ready and careful to stay safe online.

AI-Driven Security

Yigal explains how the information gathered through various security tools can be utilized for threat hunting, a proactive approach to identifying potential breaches. Relying solely on traditional security information and event management (SIEM) systems and antivirus solutions is no longer sufficient.

Seth recalls his past experiences dealing with System Center Operations Manager (SCOM)  and the overwhelming amount of alerts generated by these systems, making it difficult to identify real threats. He contrasts this with modern approaches that leverage AI and machine learning to analyze data and pinpoint anomalies.

He cites ChatFortress as an example of a company that uses AI to sift through vast amounts of data, filtering out noise and identifying potential threats. This allows security analysts to focus their efforts on investigating real risks rather than getting bogged down in a sea of alerts. 

The future of cybersecurity lies in combining AI and machine learning with threat hunting to create a more efficient and effective defense against evolving cyber threats. Seth mentions the concept of leveraging AI and machine learning for threat hunting is what he considers to be the way forward. He adds, “With AI now, it’s a little bit more personal because you can look at the pattern of the person as a variable. Um, so… but, and then these services aren’t that expensive right now.”

According to Yigal, it’s best to use these tools to automate tasks, but also stresses the need for human oversight. 

Supply Chain Risks & Digital Certificates

Yigal switches gears on supply chain security, specifically focusing on the SolarWinds incident. He clarifies that SolarWinds DLLs (Dynamic Link Libraries), which are essentially programs, are digitally signed by certificate authorities.

He explains how these certificates are used to verify the legitimacy of software to ensure that the files are created by trusted sources. 

As for Seth, he uses DocuSign as an example to illustrate the concept of digital certificates. He explains that when a company like his obtains a certificate, it serves as a way to verify the authenticity of files and software they create. It’s a way of saying, “We made this, and it’s legitimate.”

He compares it to a notary public, who traditionally validates the authenticity of physical documents. In the digital age, this process is replicated through services like DocuSign. Yigal describes how DocuSign creates a unique code, similar to an algorithm of numbers when a document is signed electronically. This code, hidden beneath the visible signature, is sent off for verification.

Microsoft Vulnerabilities & The Importance Of Patching

Yigal then discusses a vulnerability in Microsoft’s digital signature, which could potentially be exploited by adversaries. He mentions that while the exact details of the vulnerability are unclear, it raises concerns about the integrity of digitally signed files. He also brings up another Microsoft vulnerability, Netlogon, which allows attackers to move laterally within a network once they gain a foothold.

He urges patching systems, applications, and drivers to prevent such exploits. Seth adds that even firmware updates should be considered, as vulnerabilities can exist at the hardware level.

Responding to a Breach

The conversation then shifts to responding to a potential breach. Yigal suggests approaching the situation as if the system has already been compromised and focusing on damage control. 

He and Seth recommend disconnecting all computers from the network, removing potentially compromised software, and rebuilding systems from scratch. They also advise resetting hardware, such as routers, to factory settings to make sure they are not compromised.

Yigal continues to change all passwords, including those for admin accounts and service accounts in Active Directory. He recommends reviewing domain users and local users, and disabling any unfamiliar accounts as a precaution.

Clever Workaround

Seth shares about a time when he and his colleagues bypassed a company policy that regularly removed their admin access. They created a dummy account with a generic name and instructions not to delete it, which was never questioned. 

“So, one of the guys went in and created an account. The account didn’t have a name, it just said, you know, instead of “admin” or whatever—I forget what the name was—but it said it was a security account and “Do not delete,” and then it just said “Microsoft.” And, literally, that account never got deleted or changed. Someone went in there, looked at it, and said, “Oh, okay, let’s remove the changes on this. Let’s keep the password forever.” And the funny thing is, myself and three other guys were using that account.”

Steps To Take In Case of A Breach

Yigal then reiterates the steps to take in case of a suspected breach: change all passwords to complex, long ones (at least 17 characters), rotate them regularly, separate duties and roles, disable unused accounts, and monitor and block any suspicious traffic, including any communication with SolarWinds.

Interested in learning more about the SolarWinds Network Breach Case Study? Click Part 2 of this conversation!

See more of The Cybersecurity Insider podcast episodes on YouTube, Apple, and Spotify.

Part #1 :SolarWinds breach, what can we say about this