Recommended Posts

Hi,

So I've got a HP DL160 G6 running with an HP SmartArray p410i 1GB super-capacitor RAM-based cache, and am using 3*15K RPM 73GB SAS drives in a RAID5 configuration and I have a single 500GB WD enterprise SATA drive connected too...

Now I know in comparison to the Dell PERC6 that the HP SA is a steaming pile of trash, it's much slower for reasons I've never been able to work out despite having a much better spec.

But this always gets me, I back up my VMs using the SSH server in ESXi and I have downtime of 50 minutes to transfer a 22GB VM, which is absolutely bloody rediculous (it's still not finished copying) and I'm just at a loss as to why. The 500GB drive isn't anything amazing, just a standard 7200 RPM drive but to transfer a VM at less than 7MBps is just... I'm speechless. I can transfer files faster on a 32 bit SCSI card faster than this.

 

Does anyone have a similar setup or any tips for this? (It's ESXi 5.1 and both file-systems are VMFS-5) I've got 4 more VMs to backup after this then apply some system updates and it's looking like it's going to take the whole day. I don't think it's a hardware speed problem, I'm thinking it's down to ###### poor drivers from HP for the p410i (hpaucli is [in comparison to dell's PERC utilities] a complete joke) or something up with ESXi but don't really know how I can go about test either or speeding it up.

Is the vmkern network on the same or a different physical port from the interface on your primary vSwitch?

 

I read that performance is awful unless this is separated (although I have never tried).

  On 24/01/2015 at 12:18, Fahim S. said:

Is the vmkern network on the same or a different physical port from the interface on your primary vSwitch?

 

I read that performance is awful unless this is separated (although I have never tried).

Ah, I don't mean I'm coping the data over SSH, I just mean I'm connected via SSH to do the file copying. It's going direct disk-to-disk on the same host.

At 11:58 I started copying a 10GB VM, 25 minutes later it's still not finished, so this copy speed is definitely slower than 7MBps.

How exactly are you doing this copy.. Are you going to the datastore and downloading the vm disk?

 

How is your vmkern - is it shared with another nic.. I noticed a huge increase in performance when broke out vmkern to its own port group on its own nic..

 

post-14624-0-74552400-1422101386.png

 

So here I started a download of vm - clicked go at 6:07:30..  Its downloading now, This is off a HP N40L with cheap nics added, the vmkern is using the built in nic I do believe.. If I look at my network performance for my pc I downloading the file too - getting pretty decent network util

 

post-14624-0-94754700-1422101722.png

 

Ok done...  looks like 20min

 

post-14624-0-40134400-1422102622.png

 

See the time created, and then last modified time..  So lets call it 34GB / 20 min = 1.7GB a min = 28MBps, which clearly is not full speed of my network..  I normally see double or triple that from the nas on the same esxi..  But it is inline with with how the vmkern works, etc..

 

edit:  So your dong disk to disk copy on your esxi host?  via cli command..  Let me test that with this same 34GB file..  I have a SSD datastore and the 250GB disk it came with as datastore as well..  BRB

 

Ok so started at 6:38:30 and so far its copied 3.8GB in 3 minutes.. 

so at 10 min mark bit over 12GB

 

/vmfs/volumes/54c39196-0f3ec6fc-3df2-001f29541714/test # ls -la
total 12049416
drwxr-xr-x    1 root     root           420 Jan 24 12:38 .
drwxr-xr-t    1 root     root          1400 Jan 24 12:37 ..
-rw-------    1 root     root     12343582720 Jan 24 12:48 w7x64-clean-flat.vmdk
/vmfs/volumes/54c39196-0f3ec6fc-3df2-001f29541714/test #

 

So I would have to say, seems like a bit slower than the download copy..  But that 250GB disk is pretty old crappy disk ;)

 

So at 20min, 24GB roughly looks like about 20MBps which yeah is like 3x what your seeing and this is just the controller that that comes with the N40L..

Not an answer to your problem but you could use ghettoVCB which is a free backup script for ESXi and works quite well. Bit fiddly to set up but once done it's fine. 

 

I have it scripted so I just log into the host and run a script to do the backup then once done I can copy the backup files without having to take down any of the servers.

I've now shut down all VMs except one (my W7 remote management VM) and I've taken a screenshot of it... Something really is not right here.

aNS6w1g.png

EDIT: Changed around the graph output (bit hard on a small VNC screen) and it's apparently reading at 6MBps from the main drive and writing to the backup drive at 15MBps... I can't understand how it's writing twice the data it's reading!

 

 

  On 24/01/2015 at 12:45, Depicus said:

Not an answer to your problem but you could use ghettoVCB which is a free backup script for ESXi and works quite well. Bit fiddly to set up but once done it's fine. 

 

I have it scripted so I just log into the host and run a script to do the backup then once done I can copy the backup files without having to take down any of the servers.

I can't do that as I've got snapshots disabled and have all changes written to the disk as they're performed.

Edited by n_K

Ok so it finished

 

/vmfs/volumes/54c39196-0f3ec6fc-3df2-001f29541714/test # stat w7x64-clean-flat.vmdk
  File: w7x64-clean-flat.vmdk
  Size: 34359738368     Blocks: 67108864   IO Block: 131072 regular file
Device: 831f0b1fc2871364h/9448282774282113892d  Inode: 4225796     Links: 1
Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-01-24 12:38:34.000000000
Modify: 2015-01-24 13:06:49.000000000
Change: 2015-01-24 13:06:49.000000000
 

 

So we got 28 min for 34GB, roughly 20MBps, which yeah is blowing you away on crappier hardware.... Hmmmmmm??

you could try dd command vs cp, doing a test now looks like 4.9GB in 3 minutes vs the 3.8 with the cp command, let me try uping the bs from 1M

 

edit:  Well using dd seems to get me the speeds I saw with download..  About 28MBps vs the 20 was seeing with cp.

 

edit2:  ok -- seems cp has really be depreciated for a while on esxi.. your suppose to use vmfsktools command..

 

http://www.vmware.com/pdf/esx_3p_scvcons.pdf

For performance and data placement reasons, do not use scp or cp; instead, use vmkfstools, the Virtual Machine Importer tool from VMware, or the SDK APIs to manipulate your virtual disks. You should see very significant performance improvements if you use the recommended tools.

 

So doing a copy of that same vm using

/vmfs/volumes/535605bc-d0c25a0d-7cf0-001f29541714/w7 # vmkfstools -i /vmfs/volumes/datastore0/w7/w7x64-clean.vmdk /vmfs/volumes/datastore1/test/test.vmdk
Destination disk format: VMFS zeroedthick
Cloning disk '/vmfs/volumes/datastore0/w7/w7x64-clean.vmdk'...
Clone: 100% done.
 

was done in 4.25 min or 133MBps

 

/vmfs/volumes/54c39196-0f3ec6fc-3df2-001f29541714/test # stat test.vmdk
  File: test.vmdk
  Size: 514             Blocks: 0          IO Block: 131072 regular file
Device: 831f0b1fc2871364h/9448282774282113892d  Inode: 8420100     Links: 1
Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-01-24 13:38:46.000000000
Modify: 2015-01-24 13:42:57.000000000
Change: 2015-01-24 13:42:57.000000000
 

 

post-14624-0-99362500-1422107393.png

 

Argh the dd is an annoying busybox version, just started running it with all the ibs and out set to 16MB and will see what happens with an 8GB file!

Did not know that about cp on ESXi, thanks budman! I'll see if DD increases the speed and if not will retry with that command, I'm assuming it doesn't bother copying the blank space and speeds up transfer that way, the VM I'm copying now is 90% utilised so it probably wouldn't save much time.

 

EDIT: OK no there is definitely something not right, 3Gbps link speeds between the SATA/SAS disks and the controller, 0.2GB copied in 30 seconds.

Edited by n_K

Dude see my edit.. Use vmkfstools -i src dst

 

My test shows a SCREAMING difference..   What you got to loose..  It sure can not be any slower than your dd or cp commands ;)

  On 24/01/2015 at 12:52, n_K said:

I can't do that as I've got snapshots disabled and have all changes written to the disk as they're performed.

 

Curios as to why you disabled snapshots.

Just tried on a 23GB disk after the 8GB DD finished budman;

/vmfs/volumes/508aa94d-fbcf15ba-0faf-68b599b49d30/Jan 24 2015/W7 # vmkfstools -i /vmfs/volumes/Main/Windows\ 7\ Pro
fessional-N\ x64/Windows\ 7\ Professional-N\ x64.vmdk /vmfs/volumes/Backup\ Disk/Jan\ 24\ 2015/W7/
Destination disk format: VMFS zeroedthick
Cloning disk '/vmfs/volumes/Main/Windows 7 Professional-N x64/Windows 7 Professional-N x64.vmdk'...
Failed to clone disk: The file already exists (39).
Ignore, I'm being dense and not putting in the filename!

 

  On 24/01/2015 at 14:11, Depicus said:

Curios as to why you disabled snapshots.

Uses space which I don't have that much of.

Yeah that is much better ;)  Should be a helpful thread for other people I think.. I don't normally move files between datastores

 

Now not sure on what your original was..  Was it thick, or thin?  Notice it defaults to thickzero'd -- so if was thin before, your backup isn't.  if you want to maintain thin you can do -d thin on the end.  But that took about double the time to copy..  But I would think even 10 minutes for you would be much better than what you were seeing.

Not so sure its really a performance booster doing that any more.. If on SSD datastore makes no difference for sure.. So your storage is local, is it VAAI ??  Do you see hardware acceleration when you look at your datastores?

 

There are lots of variables at play when it comes to performance - a we see in this example using a deprecated common way that many people would do can have huge performance implications.. Comes down to your requirements.  I for sure have my storage over provisioned for sure.. Small datastore..  And play with lots of vms, I don't see any reason to suck up all the space with zeros ;)

This topic is now closed to further replies.
  • Posts

    • KDE brings Wayland PiP to Plasma 6.5, adds finishing touches to 6.4 as release nears by David Uzondu The KDE team has released its This Week in Plasma update, showing the final polish being applied to Plasma 6.4 ahead of its June 17 release. Last week, the KDE team brought performance upgrades, and this week the team is continuing that with improvements like faster loading for System Monitor components in Plasma 6.4. Future work for Plasma 6.5 is already underway, and it includes a feature that many have probably been waiting for: proper Picture-in-Picture support on Wayland. This uses an experimental version of the Wayland PiP protocol, which means applications like Firefox that also implement it can finally display PiP windows correctly. It is a long-overdue addition that moves the Wayland session closer to feature parity with X11. The devs also merged KWin's Background Contrast effect into the Blur effect. Virtual desktops can now be re-ordered from the Pager widget, a feature previously missing. Invert and Zoom settings have been moved into the Accessibility page, which is a more sensible place for them than the Desktop Effects page was. The team also brought consistency to the Breeze application style, with animated effects for checkboxes and radio buttons now working in QtQuick-based apps. Other small cleanups include standardizing the section headers in the Disks & Devices, Networks, and Bluetooth widgets. For those who do a lot of screen recording, Spectacle now makes it much clearer how to stop a recording, both in its notifications and shortcut names. As for the immediate future, Plasma 6.4 and its first point release are getting accessibility and user interface tweaks. The team improved text contrast for labels used in secondary roles throughout Plasma, making things like brightness indicators much easier to read. The Kicker Application Menu in 6.4 can now scroll horizontally when a search returns a ton of results, so you can actually see all of them. The team also delivered some stability improvements in Plasma 6.4.0, most notably fixing a long-standing issue where adding widgets to oversized panels could freeze the entire shell. Discover also got a much-needed fix for a crash that occurred when suggesting replacements for unsupported Flatpak apps. On the usability side, dragging files into a Folder View widget no longer causes glitchy visuals, and Open and Save dialogs from Flatpak-based browsers now properly allow the preview pane to open. Printing from Flatpak GTK apps now respects correct sizing, and installing or removing apps no longer wipes out your search input in Kicker or Kickoff while you're using it. Other notable fixes include: Selection rectangles on the desktop now render properly when using custom fonts or sizes (Plasma 6.3.6) A crash in System Monitor charts used by apps and Plasma components has been resolved (Frameworks 6.15) Switching process views in System Monitor no longer causes crashes (Frameworks 6.16) Open and Save dialogs no longer close when hovering over specific files (Frameworks 6.16) A thumbnailer crash on X11 caused by certain widget styles has been fixed (KDE Gear 25.04.3) Frameworks 6.15 also speeds up System Monitor by delaying tree view arrow loading There are still 3 high-priority Plasma bugs holding out, and the list of quick-win "15-minute bugs" has grown to 23.
    • Hasleo Backup Suite Free 5.4.2.0 by Razvan Serea Hasleo Backup Suite Free is a free Windows backup and restore software, which embeds backup, restore and cloning features, it is designed for Windows operating system users and can be used on both Windows PCs and Servers. The backup and restore feature of Hasleo Backup Suite can help you back up and restore the Windows operating systems, disks, partitions and files (folders) to protect the security of your Windows operating system and personal data. The cloning feature of Hasleo Backup Suite can help you migrate Windows to another disk, or easily upgrade a disk to an SSD or a larger capacity disk. System Backup & Restore / Disk/Partition Backup & Restore Backup Windows operating system and boot-related partitions, including user settings, drivers and applications installed in these partitions, which ensures that you can quickly restore your Windows operating system once it crashes. Viruses, power failure, or other unknown reasons may cause data loss, so it is a good habit to regularly back up the drive that stores important files, you can at least recover lost files from the backup image files in the event of a disaster. System Clone / Disk Clone / Partition Clone Migrate the Windows operating system from one disk to another SSD or larger disk without reinstalling Windows, applications and drivers. Clone entire disk to another disk and ensure that the contents of the source disk and the destination disk are exactly the same. Clone a partition completely to the specified location on the current disk or another disk and ensure that the data will not be changed. File Backup & Restore Back up specified files(folders) instead of the entire drive to another location to protect your data, so you can quickly restore files(folders) from the backup image files when needed. Incremental/Differential/Full Backup Different backup modes are supported, you can flexibly choose data protection schemes, which can improve backup performance and save storage space while ensuring data security. Delta Restore Delta restore uses advanced delta detection technology to check the changed blocks on the destination drive and restore only the changed blocks, so it has a faster restore speed than the traditional full restore. Universal Restore This feature can help us restore the Windows operating system to computers with different hardware and ensure that Windows can work normally without any hardware compatibility issues. Hasleo Backup Suite 5.4.2.0 changelog: Added backup image delete feature Added storage path management feature Improved file backup feature Show application notifications in Windows Notification Center Various other bug fixes and feature improvements Download: Hasleo Backup Suite 5.4.2.0 | 34.4 MB (Freeware) Links: Hasleo Backup Suite Website | Hasleo Backup Suite Guide | Screenshot Get alerted to all of our Software updates on Twitter at @NeowinSoftware
    • Fresh leak suggests OnePlus Pad Lite in the works, key specs revealed by Sagar Naresh Bhavsar OnePlus recently unveiled the mid-range OnePlus 13s and the OnePlus Pad 3 Android tablet in India (the tablet is also coming to the U.S.). Now, it appears that the company is working on another tablet, this time, a budget model called the OnePlus Pad Lite. Supposedly, it will be the successor to the OnePlus Pad Go, which launched in the U.K., India, and European countries. According to the leaked images, OnePlus Pad Lite follows the same design elements as earlier OnePlus tablets. The tablet is shown to have a circular camera at the center of the rear panel, with the logo sitting in the middle. The USB-C port and speaker grills are located on the right side frame of the tablet. The alleged OnePlus Pad Lite measures 254.9 x 166.5 x 7.4mm and weighs 539 grams, which is slightly taller, wider, and thinner, but weighs less compared to the OnePlus Pad Go. The tablet, courtesy of 91Mobiles and OnLeaks, is displayed in blue, which could be the only color option. One image shows the tablet with a cover that doubles as a kickstand, but may be sold separately. Gallery: OnePlus Pad Lite Based on the leaked specs, the OnePlus Pad Lite could feature an 11-inch LCD 1920x1080 resolution 90Hz display. It could be powered by the MediaTek Helio G100 processor and paired with the Mali G57 GPU. The tablet may come with 6GB of RAM and 128GB of onboard storage, with more storage and RAM options available at launch. Reportedly, the OnePlus Pad Lite could feature a 5MP primary camera and a 5MP selfie camera. Under the hood, it could be juiced by a 9,340 mAh battery. On the software side, OnePlus Pad Lite may come with OxygenOS 15.0.1 based on Android 15 out of the box. There is no clarity on which market OnePlus plans to launch the OnePlus Pad Lite in. However, the report suggests that it could be priced under ₹20,000 (roughly $231). Images by 91mobiles x OnLeaks
  • Recent Achievements

    • One Month Later
      5i3zi1 earned a badge
      One Month Later
    • Week One Done
      5i3zi1 earned a badge
      Week One Done
    • Week One Done
      julien02 earned a badge
      Week One Done
    • One Year In
      Drewidian1 earned a badge
      One Year In
    • Explorer
      Case_f went up a rank
      Explorer
  • Popular Contributors

    1. 1
      +primortal
      544
    2. 2
      ATLien_0
      227
    3. 3
      +FloatingFatMan
      160
    4. 4
      Michael Scrip
      113
    5. 5
      +Edouard
      102
  • Tell a friend

    Love Neowin? Tell a friend!