Virtualization

[masteen]

I know it's been the hot buzzword in IT for a couple years now, but the company I work for is just now looking into it. So I've repurposed one of our old servers, scavenged 5GB of RAM from it's cousins, and am installing 6x147GB SCSI drives in RAID 5 so my coworker and I can set up some virtual machines and get familiar with this before we're expected to maintain this in the production network.

So any of you nerds using VM have any tips, tricks, amusing anecdotes, or real horror show stories?

[Aabidano]

Why are they wanting to run VMs?

Definitely has it's place but it's not a perpetual motion machine the vendors pitch it as, and many management types think it is

They took lessons from MS on marketing.

[Bubba Grizz]

We reduced our server farm from 160 servers down to about 37 by using VM. It still scares me though to think that if the machine hosting those servers fails...

[Canelek]

We've been doing the "footprint reduction" thing over the last year. Got a couple really solid VMware and storage guys. ESX is pretty tight, especially with Raw Disk Mapping and the SAN.

That said, it is not for everything. I am a SQL Server DBA (Production) and I have quite a few boxes that require much, much more than the 8 CPU limit on ESX VMware instances. I have some batch and datawarehouse servers that run HP 580s with 128 GB RAM and 4 X 6-core CPUs.

[Hesten]

We got around 1900 servers, and so far managed to virtualize around 300....our plan is to be down to 800 servers or so in 2012.
And its rare when you cant virtualize a server, due to hardware demands, usually only with very heavy database use, or if some absolutely moronic company decides to use some external hardware (yes, still happens)

[Aabidano]

Seen lots of places going back to mainframe-ish architectures, sometimes w\ terminals to get away from the explosion of windows servers everyone seems to get. A company down the road went from thousands of Sun & MS systems to a dozen globally redundant IBM boxes.

Worked on a job in Buffalo that replaced a basketball court sized data center with two systems + a backup across town.

[masteen]

I know none of our current applications are that demanding because most of the boxen in our racks are run by asian gerbils working tiny abaci. For us, it's not so much about the footprint reduction, but more about ease of management.

The audit last week hammered us because our servers are so fucking far behind on their updates from MS, but because we're a bank, we can't take many of these machines down for extended maint. because they supply data to online banking. It would be nice to be able to migrate VMs from physical box to box while we patch behind them.

[Canelek]

PEE2VEE!

[pyrella]

First and foremost - get shared storage for them - you can set up dirt cheap iscsi with a box with a bunch of harddrives and a linux distro (openfiler, etc) that will do the trick for testing and development. Then you can go to pro iscsi or FC solutions. It's good to tier like cheap iscsi for archival stuff (not your back up box, that's your hardest working machine) - SATA for low end file servers and the like, and FC or SSD for OS's and databases etc.

Shared storage lets you lose any of the VM hosts and be able to migrate your client vm's in state and resume - in our test 2-4 seconds of lost pings, depending on how many are going at once. VMotion is the shit.

There are other solutions out there besides VMWare, but none are as robost with as large of an active community that can give you example scripts, hints, tips, and share your same horror stories so you can actually fix it. Also it's converter is usually smooth as silk - however the generic fix for anything that doesn't want to convert properly is to use some third party backup like acronis or ghost or something, and create an image of the client, create a new box in the VM environment and restore in to it. Also be careful with anything that has SATA or SAS drives in it and specify in your hardware options to force the LSI logic scsi controller.

[Hesten]

I know none of our current applications are that demanding because most of the boxen in our racks are run by asian gerbils working tiny abaci. For us, it's not so much about the footprint reduction, but more about ease of management.

The audit last week hammered us because our servers are so fucking far behind on their updates from MS, but because we're a bank, we can't take many of these machines down for extended maint. because they supply data to online banking. It would be nice to be able to migrate VMs from physical box to box while we patch behind them.

Hehe, join the club, our 1900+ servers run 15 hospitals, we got around 30 NT4 servers that can not be upgraded, we got 900+ programs, half of them undocumented, or close to it.
As for patch level, we finally got our bosses to see the light and allow us to start on it, and give us money for it this summer, after a conficker virus outbreak (small one, more of a nuisiance, but good to show the bosses that its serious), a virus that got in through a security hole fixed THREE years ago by MS in a patch

[Zaelath]

I know none of our current applications are that demanding because most of the boxen in our racks are run by asian gerbils working tiny abaci. For us, it's not so much about the footprint reduction, but more about ease of management.

The audit last week hammered us because our servers are so fucking far behind on their updates from MS, but because we're a bank, we can't take many of these machines down for extended maint. because they supply data to online banking. It would be nice to be able to migrate VMs from physical box to box while we patch behind them.

Hehe, join the club, our 1900+ servers run 15 hospitals, we got around 30 NT4 servers that can not be upgraded, we got 900+ programs, half of them undocumented, or close to it.
As for patch level, we finally got our bosses to see the light and allow us to start on it, and give us money for it this summer, after a conficker virus outbreak (small one, more of a nuisiance, but good to show the bosses that its serious), a virus that got in through a security hole fixed THREE years ago by MS in a patch

900+ programs? And people wonder why we get shirty when we suggest their network needs to be managed rather than run like a hippy commune.

[Hesten]

Well, we had 10 years of anarchy, where the IT bosses were kings of their own little realm (and still act like they are, its only 4 days ago since i found out they took a test server we gave them for one purpose, and made it a production server for another system, and didnt tell us, and now complains we didnt monitor if it was down. Gonna play hardball on that one and demand the payment for a production server, test servers are free, production are not).
And then every departments doctors could also buy what they liked, and they did...that have caused some issues now that we made a single IT department to run the servers for all 15 hospitals . Its called "a challenge"

[Aabidano]

We're looking at picking up a map server, one of MS's prerequisites is that you buy their server.... It's a nicer product and would be a better fit with some other stuff it's being integrated with but I'm not willing to fight to get support for yet another production box.

We'll likely go with something we can run in a VM on ITs' HW or native on one of our Suns.

[masteen]

First and foremost - get shared storage for them - you can set up dirt cheap iscsi with a box with a bunch of harddrives and a linux distro (openfiler, etc) that will do the trick for testing and development. Then you can go to pro iscsi or FC solutions. It's good to tier like cheap iscsi for archival stuff (not your back up box, that's your hardest working machine) - SATA for low end file servers and the like, and FC or SSD for OS's and databases etc.

Shared storage lets you lose any of the VM hosts and be able to migrate your client vm's in state and resume - in our test 2-4 seconds of lost pings, depending on how many are going at once. VMotion is the shit.

There are other solutions out there besides VMWare, but none are as robost with as large of an active community that can give you example scripts, hints, tips, and share your same horror stories so you can actually fix it. Also it's converter is usually smooth as silk - however the generic fix for anything that doesn't want to convert properly is to use some third party backup like acronis or ghost or something, and create an image of the client, create a new box in the VM environment and restore in to it. Also be careful with anything that has SATA or SAS drives in it and specify in your hardware options to force the LSI logic scsi controller.

That's awesome, we've been playing with ghost for a couple months now, so making images shouldn't be too traumatic if we have any conversions fail.