The Compliance and Security Connection

Virtualization continues to grow, so how do you control the sprawl?

According to recent research from IDC, "the pace of adoption of virtualized servers is incredibly rapid among organizations that are using virtualization, with 35% of servers purchased in 2007 being virtualized and 52% of those bought in 2008 expected to be so." Now, while this was research of the European market, it is pretty easy to assume that these increases are probably representative of the US market as well, with possibly higher percentages. However, while these numbers reflect the number of servers purchased, it doesn't begin to capture the number of Virtual Machines being deployed on those servers.

The recent edition of InformationWeek featured an article by Joe Hernick and Lorna Garey discussing why it is "Time to Halt Runaway VM Sprawl." At the very beginning of the article, Hernick and Garey ask the question, "How will you audit for compliance if you don't know where all your production VMs reside, or even how many you have?"

As good as that statement is, it really only addresses have of the compliance concern surrounding virtualization. In the IDC study, their research showed that "the majority of virtualization is still for test and development and for network server applications." In many companies, this likely means that some customer data, including potential data that falls under compliance regulations, is stored in virtual test environments.

In addition, the IDC research also found that "virtualization is growing as a datacenter strategy in itself rather than as part of other projects." Why is that significant? It means in a growing number of IT departments, not only is the test environment virtualized, so is the production environment and, in all likelihood, so is the back-up data center used for disaster recovery and business continuity purposes. This means you need to understand how VMs are configured in each of these three virtualization environments if they hold data that falls under any of a growing number of government and industry regulations.

If you are just transitioning from physical  to virtual servers, Hernick and Garey offer these recommendations:

• Admit you need to put a policy in place for how physical to virtual migrations are to be executed
• Live by the credo that "a virtual server is still a server", with all the policies and security and management concerns of a physical box
• Build a mission statement laying out the organizational goals for virtualization
• Inventory your physical and virtualized environments (Ecora Auditor Pro can be a tremendous help here)
• Overlay compliance and data security policies and organizational and management requirements (again, Ecora Auditor Pro and its Executive Dashboard and Compliance Center can be a real asset here)
• Put strict controls in place on VM creation (Auditor can also help in identifying unapproved VMs that may be deployed)

As IT becomes more and more dependent on virtual environments, especially with the availability of Microsoft's Hyper-V in addition to VMware ESX, as well as the growing number of other players in the virtualization space, it will become increasingly more critical to be able to quickly access the number of VMs in your environment, understand how they are configured, and be able to track changes to those configurations on an ongoing basis.

Contributed by Mark Tordoff

HP Chief Security Strategist Shares 5 Steps to Safer Virtual Servers

Earlier this month, at the Black Hat conference, Chris Whitener, chief security strategist with HP Secure Advantage, shared the following five steps to safer virtual servers, in a story that was captured by Infoworld.

1. Protect your host operating system by using server hardening tools and methodologies

2. Ensure that your host OS is as secure as the guest operating system

3. Security policies in the host OS should reflect requirements of individual virtual machines

4. Manage virtual processes more like you already manage your physical resources

5. Stay vigilant about securely managing the physical infrastructure

A challenge for many organizations is how to easily and cost-effectively manage the configurations on both the virtual servers themselves and the virtual machines they host.

Ecora Auditor Pro is capable of collecting and reporting on configuration attributes in both environments, as well as comparing physical environments to systems in the virtual environment.

As importantly, policies and rules regarding proper configuration attribute settings can be created and compared against actual infrastructure configurations on a scheduled basis. To see examples of reports related to both virtual and physical servers that can be generated using Ecora Auditor Pro, you can go to http://reports.ecora.com and view several samples.

Without the ability to constantly understand how systems are configured, virtual or not, and compare them with established configuration standards, it is more than just virtually impossible to secure your virtual server environment.

Contributed by Mark Tordoff

More Thoughts on Virtualization Security

After yesterday's post on The Burton Group's Five Immutable Laws of Virtualization Security, I received Mike Rothman's daily update where he linked to his most recent Mike Rothman Security Report on eBizq.com where he shared his own thoughts on securing virtualization.

Mike focused on the hypervisor, application issues and operational issues. Here are some excerpts that stood out to me.

Hypervisor

"The hypervisor is actually another operating system and thus needs to be secured like an operating system. If the hypervisor is owned, which is called hyperjacking, it's game over for all the servers that run on top of the hypervisor and given that virtual machines (VMs) can be moved dynamically between hypervisors running on separate machines, one jacked VM puts your entire data center at risk."

Mike's conclusion was"the entire industry needs to keep pushing VMWare and the other virtualization players to ensure they have as effective a security research and quick response patching capability as the core operating system vendors."

Application Issues

"There isn't anything specific to virtualization that needs to be done within the application to make it more secure...there is a lot more at stake in a virtualized environment. Remember, a vulnerable application running on the hypervisor could allow root access to OS, which in turn allows that VM to launch attacks against other virtual machines."

Operational Issues

Here, Mike reinforces the complexity caused because "it’s so easy to just spin up another VM at will." He suggested that  "each VM needs to have a standard build, which should include some automated mechanism (meaning some kind of agent) for configuration management and/or patching."

Mike also sends a warning to intrusion prevention vendors when he points out that "VMs running on the same machine will communicate with each other without communicating through the actual 'network'."

His input for application developers is to think of virtualization similarly to SOA. "A lot of the underlying resources that drive applications like data and computing resources are virtual and therefore largely unknown. So part of building applications continues to be relying on "faith" that the data you are calling is the right data and that the requests that you are responding to are authentic and authorized requests'" according to Rothman.

As the adoption of virtualization continues to escalate in production environments, security issues are going to become more and more important. Much like we've seen with compliance issues, namely SOX and PCI, there will be an ever increasing wave of marketing pitches offering solutions for addressing virtualization security. Make sure you get the facts before spending money on a solution that may be more "virtual" than reality...pun intended.

Contributed by Mark Tordoff

Burton Group Identifies the Five Immutable Laws of Virtualization Security

The Burton Group recently released a Burton Group Overview, “Attacking and Defending Virtual Environments.” While you do need a client log-in to access the report, Burton Group analyst Pete Lindstrom did share the following excerpt highlighting what they've called the Five Immutable Laws of Virtualization Security on a recent post to the Burton Group blog.

When combining the standard risk principles with an understanding of the use cases of virtualization, a set of immutable laws can be derived to assist in securing virtual environments:

Law 1: Attacks against the OS and applications of a physical system have the exact same damage potential against a duplicate virtual system.

The point here is that, for all intents and purposes, a VM acts like a physical system, and in most environments that means it can be attacked in exactly the same way. The suggestion that because VMs run in user mode and are therefore more secure is insignificant if an attack remains within the VM and has access to the VM’s content and programs as if it is in kernel mode. So, when comparing a physical system with an OS and applications to an identically configured VM, all existing attacks will work the same way. Importantly, many attacks occur in the application stack, often against sensitive information itself. Any data on the VM may be stolen, and if the VM has network access it may be used as a stepping stone to attack other systems just as a physical system might be so used. The only attack that may be original and different is one that breaks the virtualization mechanisms and compromises additional VMs.

It is important to note that although VM and non-VM attacks are largely the same, the ability to recover from an attack may be significantly increased with VMs, and so the response and recovery may be different.

Law 2: A VM has higher risk than its counterpart physical system that is running the exact same OS and applications and is configured identically.

This corollary to the first rule above introduces the effect of the additional attack surface from the hypervisor and VMM. Because the VMM monitors and responds to a VM, it becomes susceptible to attack itself. The same is true for hypervisors. Any addition of software on a platform necessarily increases its risk if all other things are equal (i.e., there are no attack surface reductions performed in conjunction with the migration). Of course, all other things aren’t equal, so it is important to recognize the basically negative impact of the virtual environment on risk and to pay more attention to offsetting that risk in other ways.

Law 3: VMs can be more secure than related physical systems providing the same functional service to an organization when they separate functionality and content that are combined on a physical system.

When two processes share the same memory space, an attack against one process can impact the other. That is, system ownership by an attacker trumps any other security. One of the ways to benefit from virtualization is to separate various functions into their own isolated operating environments. An even better way is to separate data as well. Either type of separation provides an opportunity to reduce the risk added by the virtualization software.

Law 4: A set of VMs aggregated on the same physical system can only be made more secure than its physical, separate counterparts by modifying the configurations of the VMs to offset the increased risk introduced by the hypervisor.

While separating resources reduces risk, aggregating resources will initially increase risk (Rule 2) and so must be reconfigured or hardened to attain the same level of risk or a reduced level (Rule 3). Turning off services, adding controls, and separating content can reduce that risk.

Law 5: A system containing a “trusted” VM on an “untrusted” host has a higher risk level than a system containing a “trusted” host with an “untrusted” VM.

Broadly, “trusted” systems are managed and have protection controls deployed, while “untrusted” components may or may not be managed but are generally considered to have weaker protection and are therefore more susceptible to attack.

In American football, the general concept of “getting low” while blocking is an advantage to the blocker. In attacking systems, the same is true. Attacks at lower levels have higher risk than those at higher levels. This is primarily due to the ability to spoof or trick higher-level programs into believing assertions about trust and authenticity.

This rule is particularly important with Type 2 virtualization (OS-hosted virtual machine) because the host OS has a ready attack surface. Because client-side architectures involve both of these scenarios, it is important to recognize and understand the differences. Given this rule, it is important for deployments of trusted VMs in untrusted environments to consider the implications and harden the VM image accordingly.

Pete and the Burton Group have made some very salient points in this post. One area that they didn't speak of, at least in the post, was how the ease of VM creation as opposed to the cost and time of provisioning a physical server has led to significant increases in the number of overall servers for organizations deploying virtual environments.

In a similar vein, physical environments offer some physical evidence of security (I can see the server box, the router, the firewall and have some ability to know who may be physically interacting with the hardware, what any alerts or other diagnostics that appear physically on the console might be indicating, etc.), but the same is not true in a virtual environment. It is much more important to have tools to manage what is happening in a virtual environment, especially with the ability to change resources dynamically and easily turn VMs on and off, if you will.

As Lindstrom said in a release on Marketwire, "Ultimately, whether virtualization is a bane or boon for security depends on how the systems are configured, deployed and managed."

Contributed by Mark Tordoff

Concerns About Security of Virtual Environments

Virtualization is one hot topic. Whether it's a blog post, a white paper, or a webinar, no topic draws more attention than one involving some aspect of virtualization. Given the value it provides in reducing the number of physical servers, the energy savings, and the flexibility to respond to demands on the system, it should be no surprise that a lot of people are trying to learn as much as possible about how virtualization might benefit their organization.

However, David Frith, a senior consultant with Siemens Enterprise Communications Limited, shares some security-related cautions in a recent Continuity Central post.

Frith raises several issues that are worthy of consideration, whether you're still considering migrating to a virtual environment or have already taken the plunge.

1. "Full security analysis of many [virtualization] vendor offerings reveals large areas of unexplored code in which could lurk potential flaws, this is an ‘known unknown’ since the lack of live deployments until recently has resulted in little testing."
2. "Attacks on virtual systems can come from an extension of older forms of attack such as Denial of Service (DoS), buffer overflows, spyware, rootkits and/or Trojans – all prone to lurk beneath guest operating systems."
3. "New specific attacks include those from worms, guest hopping, Hypervisor malware and Hyperjacking all involving the Hypervisor itself being exploited and used to subvert each VM it controls."
4. "Existing security technologies typically revolve around static and IP based controls (be they firewalls, IDS’s, VLAN’s etc.), however the tracking of IP or static based identifiers is no longer sufficient."
5. "Existing [vendor] tools largely fall short ... in their ability to monitor the whole enterprise, integrate with other tools and to keep track of and detect VMs to limit their spread.
6. "Control of inter-virtual data needs to be monitored, with suspicious traffic reported and/or escalated."
7. "Communications between virtual components need to be safeguarded with built-in encryption, digital signatures and hardware based root certificates provided by technologies such as the Trusted Computing initiative TPM (Trusted Platform Module) offering built in security, tamper detection and exploit prevention."
8. "Management tools need to map interdependencies and data flows ensuring that, with all the complexity, administrators do not lose an understanding of their environment."
9. "Patching tools are also required. The need to introduce timely patches is ever more critical to reduce attack surfaces and ensure best-practice compliance. New technologies such as inline patch proxying and application correction (modifying data in midstream) have been developed to help mitigate such issues."

A recent post by Carl Weinschenk on virtualization security also references a recent post by Slavik Markovich, the CTO of Sentrigo,who laid out the problems to IT Business Edge earlier this month. As Weinschenk remarked, "The vulnerabilities were stark, even to a non-expert: The new architecture is not compatible with the old approaches to security."

In his remarks, Markovich comments that virtualization vendors themselves are not up to speed in addressing concerns similar to those raised by Frith.

Until other security vendors or the virtualization vendors themselves can catch up, IT departments should carefully consider each of these concerns and how they might best be addressed. With the number of installed virtual systems expected to double by 2012 and with the potential that a breach might yield data from multiple systems instead of just one, the efforts of cyber criminals will undoubtedly be more focused on exploring the vulnerabilities of virtualization and how they might be exploited.

Contributed by Mark Tordoff

What's Ahead for Virtualization

'Tis the season for surveys, prognostications, and general guesses at what the year ahead might bring.

Amy Newman of ServerWatch weighed in on what she thinks the year ahead will bring to virtualization.

She starts by referencing some IDC research that indicates the number of server units shipped in the third quarter of 2007 grew 1.5 percent less than it did for the previous period in 2006. As Newman queried, is this an initial sign of the migration from physical to virtual server environments? Or, perhaps it is just a sign of the tight purse strings in most organizations around capital IT expenses.

That being said, here are seven trends Newman is anticipating in 2008.

1. Security Will Loom Large
While virtualized envionments haven't been singled out specifically for exploitation, that will most likely change with increasing adoption. Vice president and Gartner fellow, Neil MacDonald was quoted in the article as saying, "You have 10 workloads and you merge them onto one, that's a very attractive target for a bad guy. Now, if I compromise just one thin layer I get all 10 machines. ... And this layer will be targeted."

2. Easy Money Won't Be Quite So Easy Anymore
The virtualization field is getting crowded and major players who were observing from the sidelines have finally jumped in the game. Newman says "ideas will need to be more solid to receive funding, and venture capital firms may be ready to cash out, whether through a sale or IPO."

3. Value-Add Will Be Critical
With multiple virtualization vendors now in the fray, choosing a vendor will have less to do with their hypervisor technology and more to do with what they offer for improved performance, integration, usability, and management. Newman referenced the recent announcement by VMware, a traditional go-it-alone type of company, of a partnership with SAP as an example of the types of deals you can expect to see between the virtualization vendors and ISVs.

4. Vendor Consolidation Will Continue
When it comes to consolidation around virtualization, Newman recommends watching the hardware side carefully. She says "theories about Intel, Sun Microsystems and IBM going on shopping sprees have surfaced in the blogosphere in recent months."

5. Microsoft Will Make Waves

Microsoft released Hyper-V, the hypervisor formerly known as Viridian, for public beta last week.

Newman believes Hyper-V will have an enormous and direct impact on the virtualization landscape in 2008. "More organizations deploy Microsoft than any other operating system. With Hyper-V a standard component, enterprises still on the virtual fence will be able to dip a toe in the water, easily and inexpensively," she said.

6. Storage and Servers Will Converge
Convergence has been a buzzword in the dot-com era. Newman said last week's product news from VMware and Virtual Iron offers ample evidence of the melding of storage and server infrastructures.

7. I/O and Automation Will Become Increasingly Critical
In the article, Newman said "when you're moving workloads around and planning out deployments, it's important to have an understanding of your needs and limitations. As more workloads are moved around virtually, it's even more paramount." She cited a recent survey from Xsigo Systems to highlight the concerns surrounding  I/O problems and the price HP paid for Opsware as an example of the importance of automation.

While none of these may be completely surprising, I think Amy has made solid observations, particularly as virtualization moves from being "the latest thing" to a platform more and more organizations are relying on to deliver business value.

Contributed by Mark Tordoff

Virtualization Security Concerns - Truth or Fiction?

I wanted to take a moment today to catch up on a story written by Denise Dubie a few weeks ago for Network World. Dubie takes on some IT concerns that might be hindering some organizations from deploying virtual servers in their environment.

Dubie tackles four specific concerns: virtual-machine escapes, patching challenges, running virtual machines in a DMZ, and the relative newness of the technology as a target for hackers.

1. Virtual-machine escape

Virtual-machine escape is the phrase defining an attack where a hypervisor attack could potentially infect virtual machines that reside on the same physical host. In this scenario, if a virtual machine is able to "escape" the isolated environment where it resides and interact with the parent hypervisor, an attacker could potentially use the access to the hypervisor to control the remaining virtual machines on that system.

While current users admit the possibility for this may exist, this type of attack has yet to be seen and there are steps to prevent it. For example, Tim Antonowicz, from Bowdoin College, sequesters virtual machines in resource clusters to limit this threat.

2. Patching Challenges

Virtual-server sprawl is the concern with staying on top of patches. As Dubie states, "IT managers agree that patching is critical in virtual environments, but the real difference between virtual and physical-server patching isn't a security issue, it's about volume." The key here is having an automated solution for patching, as manual efforts may soon not be capable of keeping up with server growth. "Virtual environments can grow too fast without physical constraints," Antonowicz was quoted as saying.

3. Virtual Machines in a DMZ

The concern is running mission-critical servers in the DMZ, but, according to Burton Group's Pete Lindstrom, "You can run virtualization inside the DMZ as long as the firewall or separating device is physical. And, in most cases, as long as you are separating out resources, you are good to go."

In Antonowicz's case, he sets up his environment so "each cluster has its own set of resources and accessess so you can't get from one to the other..."

4. The Relative Newness of the Technology as a Target for Hackers

While most new technologies are susceptible to flaws, virtualization has appeared to be fairly stable in that respect. Part of it is the technology is really derived from established platforms and, as Peter Christy, principal at Internet Research Group said, "a hypervisor is a small piece of code that represents a small and limited surface area, which is easier to make more secure than 80 million lines of code."

The bottom line - If you think through where virtualization will provide you value and carefully plan your security needs, you should not have any exceptional security concerns that would prevent you from considering virtualization if it makes business sense.

Contributed by Mark Tordoff 

Opinions Run Wild on Future of Virtualization

Earlier this week, I shared some thoughts on Oracle's announcement of their free version of the Xen hyervisor. In the last few days, writers, bloggers and analysts have shared opinions that have run the gamut.

The VMware is dead opinion was championed by Steven J. Vaughan-Nichols in Is VMware a Dead Duck? Vaughan-Nichols' opening salvo was "Today, VMware has about 80 percent of the virtualization market. By 2011, I think it may be lucky to have 8 percent of the market." He goes on to justify his comments by implying that Xen OpenVZ, KVM, VirtualBox and UML will have a similar impact on VMware that Linux did on Unix. And, if that's not enough to cut VMware's market share off at the knees, the efforts of Microsoft, Citrix, Red Hat and now Oracle will take care of the rest. Vaughan-Nichols says any efforts VMware makes at preserving its market share with value-added services will be "too little, too late."

However, if you're not quite ready to jettison your VMware stock, at least based on Oracle's announcement, you have plenty of company who think the Oracle announcement is nothing more than marketing smoke and mirrors. According to a post on ZDNet by Larry Dignan, "analysts called Oracle’s virtualization announcement “virtual FUD” and noted it was reminiscent to the company’s Unbreakable Linux announcement a year ago." You can read further comments from several leading analysts in Larry's post.

VMware, in response, took the high road, as you might expect. Parag Patel, vice president of alliances with VMware, is quoted as saying, "VMware ESX Server has been tuned extensively to run both databases and other mission critical applications well. Real-world Oracle database performance can run near-native when running on VMware Infrastructure on paravirtualized Linux."

"VMware will continue to work closely with Oracle because we believe the majority of customers want the flexibility and choice offered by an Oracle solution on VMware Infrastructure.”

Is the sky falling around VMware, or do they still have a bright horizon in front of them? There's lots of opinions, but it's clear that recent activity will not allow them to sit on their laurels celebrating their ample stock increases of the past few months.

Contributed by Mark Tordoff

There's a New Competitor for King of the Virtualization Market

Oracle announced yesterday at their user's conference in San Francisco that it is jumping into the virtualization market. And quite a leap it was. According to an article in the Boston Globe, customers can download "Oracle VM" for free starting today, according to the company. Oracle will sell service contracts for the product ranging from $499 to $999 per year.

Oracle is the latest to join the fray to knock off VMware's dominance of the server virtualization market. It is not a surprise that more companies want to enter this market, especially given VMware's IPO in August was the hottest technology initial public offering in years. The news from Oracle did drop VMware's stock price 8%, but at $80.36 a share, it still remains far above its initial public offering price of $29 a share.

Oracle is touting that its offering is "three times more efficient than competitors' offerings", although I'm not sure how they are measuring that or even how you can claim that without even being deployed in end users' production environments yet.

That being said, the interesting question will be who are the guinea pigs that are going to step up and actually give this a try. Free can go a couple of ways. You certainly will have some people that have held off investing in virtualization that may seize this opportunity, but you also have a lot of people who tend to shy away from "free" products, fearing that there must be a catch or simply a compromise on features.

Yet, Microsoft has certainly managed to knock the stuffing out of the patch management market with their release of WSUS a few years back. Although, there's really no comparison between the relative simplicity of deploying patches and the challenges of implementing and managing a virtualized environment.

The good news in all this for consumers is competition usually breeds innovation. With another major player entering the field, you can expect that VMware, who was already watching Microsoft's moves, will certainly not take Oracle's announcement lightly.

Contributed by Mark Tordoff