Home Applications My Challenge to Citrix’s Simon Crosby and VMware’s Stephen Herrod

My Challenge to Citrix’s Simon Crosby and VMware’s Stephen Herrod


In his blog post, Simon Crosby has a good point when he chastises VMware for using questionable methodologies in their own performance study, and for refusing to allow third party independent benchmarks to do performance comparisons. Simon presents the following graph in his blog post:


The graph above shows how end user response time rises for four environments as the number of concurrent users is increased (in this case a virtualized XenAPP server). VMware’s license restrictions prevent the identification of ESX as either Brand X or Brand Y, and I think Simon is absolutely correct when chastising Mr. Herrod about VMware’s policy of keeping the truth hidden in the shadows of obfuscation, instead of exposing it to the clarity of daylight.

However, I think both companies could take one additional step that would make such comparisons truly useful. The problem with every performance comparison that Citrix has done over the years (going back to MF and CPS days) is that Citrix always runs a script that runs a well known application like Word or Excel. There are two issues with this approach. The first is that the choice of the scripted application rarely mirrors the kind of applications that enterprises run in production. Enterprises run a mix of applications; some web based (IE as the client), some two-tier client server with fat clients (Cerner Powerchart – a popular health care application by example), n-tier with a fat client (SAP), and then standalone Office type applications that really only communicate with the file system. Running just one application cannot approximate the workload of a real mix applications like the ones listed above. The second problem is that no script can mirror what a combination of real end users will do across a mix of applications. Scripts are way to predicatble (end users are not), they wait the same amount of time between actions (end users do not), and they do not do stupid things with applications that no one ever intended (end users – at least some of them do this quite frequently).
My challenge to both companies is two-fold. Get open and get real. Simon addresses the first point about opening the process and the results to third parties who can objectively construct tests and objectively communicate results. My point is that that tests also need to mirror how these products get used in production by real end users. This means that these kinds of tests cannot be done in a lab, but must be done in real customer environments with instrumentation designed to capture end user experience for each and every end user and every application all of the time (not fake instrumentation based upon a script).
This challenge is also not just about how to benchmark the performance of the respective virtualization platforms. It is also about what the performance management strategy is for each company. VMware has at least decided to try to address this issue with the acquisition of B-hive (which will be called AppSpeed in vCenter when vSphere is released). Whether or not AppSpeed can address this issue across the application mix detailed above in production (I have my doubts) is a subject for an entire article which I will write when vSphere is released. However while VMware is making an effort to embrace production response time as the metric by which to judge applications performance and end user experience in virtualized environments, the only management offering that Citrix has is based upon EdgeSight which takes a woefully inadequate resource utilization based approach to measuring performance. In my opinion both companies need to step up their efforts in this regard, either through internal development or via partnerships with one of the many third party vendors focused upon these issues.
So, one more time. To both gentlemen and both companies. Get open and get real!
APM Experts
DABCC DABCC.com, the world leader in sharing the finest Virtualization & Cloud news and support resources. #Citrix, #VMware, #Microsoft, #Mobility and much more! Brought to you by @douglasabrown & team!

Featured Resources:

Related Articles:


White Papers

    Application Lifecycle Management with Stratusphere UX – White Paper

    Enterprises today are faced with many challenges, and among those at the top of the list is the struggle surrounding the design, deployment, management and operations that support desktop applications. The demand for applications is increasing at an exponential rate, and organizations are being forced to consider platforms beyond physical, virtual and cloud-based environments. Users […]


      Download Commvault VM Backup and Recovery: end-to-end VM backup, recovery and cloud management

      Commvault’s ability to provide end-to-end VM backup, recovery and cloud management creates a significantly better way to build, protect and optimize VMs throughout their lifecycle. Our best-in-class software for VM backup, recovery and cloud management delivers a number of significant benefits, including: VM recovery with live recovery options; backup to and in the cloud; custom-fit […]

      On-Demand Webinars

        What’s Going on in EUC Printing – A Technical Deep Dive!

        The IGEL Community and ThinPrint invite you to watch the following technical deep dive webinar. The agenda is to technically bring you up to speed on what’s going on in the EUC Printing space today along with a deep dive into new methods, technologies, printing scenarios and a discussion on why printing still matters. You […]

        Latest Videos

          Views All IT News on DABCC.com
          Views All IT Videos on DABCC.com
          Win big $$, visit ITBaller.com for more info!

          Visit Our Sponsors