Recommended Posts

Hi,

So I've got a HP DL160 G6 running with an HP SmartArray p410i 1GB super-capacitor RAM-based cache, and am using 3*15K RPM 73GB SAS drives in a RAID5 configuration and I have a single 500GB WD enterprise SATA drive connected too...

Now I know in comparison to the Dell PERC6 that the HP SA is a steaming pile of trash, it's much slower for reasons I've never been able to work out despite having a much better spec.

But this always gets me, I back up my VMs using the SSH server in ESXi and I have downtime of 50 minutes to transfer a 22GB VM, which is absolutely bloody rediculous (it's still not finished copying) and I'm just at a loss as to why. The 500GB drive isn't anything amazing, just a standard 7200 RPM drive but to transfer a VM at less than 7MBps is just... I'm speechless. I can transfer files faster on a 32 bit SCSI card faster than this.

 

Does anyone have a similar setup or any tips for this? (It's ESXi 5.1 and both file-systems are VMFS-5) I've got 4 more VMs to backup after this then apply some system updates and it's looking like it's going to take the whole day. I don't think it's a hardware speed problem, I'm thinking it's down to ###### poor drivers from HP for the p410i (hpaucli is [in comparison to dell's PERC utilities] a complete joke) or something up with ESXi but don't really know how I can go about test either or speeding it up.

Is the vmkern network on the same or a different physical port from the interface on your primary vSwitch?

 

I read that performance is awful unless this is separated (although I have never tried).

  On 24/01/2015 at 12:18, Fahim S. said:

Is the vmkern network on the same or a different physical port from the interface on your primary vSwitch?

 

I read that performance is awful unless this is separated (although I have never tried).

Ah, I don't mean I'm coping the data over SSH, I just mean I'm connected via SSH to do the file copying. It's going direct disk-to-disk on the same host.

At 11:58 I started copying a 10GB VM, 25 minutes later it's still not finished, so this copy speed is definitely slower than 7MBps.

How exactly are you doing this copy.. Are you going to the datastore and downloading the vm disk?

 

How is your vmkern - is it shared with another nic.. I noticed a huge increase in performance when broke out vmkern to its own port group on its own nic..

 

post-14624-0-74552400-1422101386.png

 

So here I started a download of vm - clicked go at 6:07:30..  Its downloading now, This is off a HP N40L with cheap nics added, the vmkern is using the built in nic I do believe.. If I look at my network performance for my pc I downloading the file too - getting pretty decent network util

 

post-14624-0-94754700-1422101722.png

 

Ok done...  looks like 20min

 

post-14624-0-40134400-1422102622.png

 

See the time created, and then last modified time..  So lets call it 34GB / 20 min = 1.7GB a min = 28MBps, which clearly is not full speed of my network..  I normally see double or triple that from the nas on the same esxi..  But it is inline with with how the vmkern works, etc..

 

edit:  So your dong disk to disk copy on your esxi host?  via cli command..  Let me test that with this same 34GB file..  I have a SSD datastore and the 250GB disk it came with as datastore as well..  BRB

 

Ok so started at 6:38:30 and so far its copied 3.8GB in 3 minutes.. 

so at 10 min mark bit over 12GB

 

/vmfs/volumes/54c39196-0f3ec6fc-3df2-001f29541714/test # ls -la
total 12049416
drwxr-xr-x    1 root     root           420 Jan 24 12:38 .
drwxr-xr-t    1 root     root          1400 Jan 24 12:37 ..
-rw-------    1 root     root     12343582720 Jan 24 12:48 w7x64-clean-flat.vmdk
/vmfs/volumes/54c39196-0f3ec6fc-3df2-001f29541714/test #

 

So I would have to say, seems like a bit slower than the download copy..  But that 250GB disk is pretty old crappy disk ;)

 

So at 20min, 24GB roughly looks like about 20MBps which yeah is like 3x what your seeing and this is just the controller that that comes with the N40L..

Not an answer to your problem but you could use ghettoVCB which is a free backup script for ESXi and works quite well. Bit fiddly to set up but once done it's fine. 

 

I have it scripted so I just log into the host and run a script to do the backup then once done I can copy the backup files without having to take down any of the servers.

I've now shut down all VMs except one (my W7 remote management VM) and I've taken a screenshot of it... Something really is not right here.

aNS6w1g.png

EDIT: Changed around the graph output (bit hard on a small VNC screen) and it's apparently reading at 6MBps from the main drive and writing to the backup drive at 15MBps... I can't understand how it's writing twice the data it's reading!

 

 

  On 24/01/2015 at 12:45, Depicus said:

Not an answer to your problem but you could use ghettoVCB which is a free backup script for ESXi and works quite well. Bit fiddly to set up but once done it's fine. 

 

I have it scripted so I just log into the host and run a script to do the backup then once done I can copy the backup files without having to take down any of the servers.

I can't do that as I've got snapshots disabled and have all changes written to the disk as they're performed.

Edited by n_K

Ok so it finished

 

/vmfs/volumes/54c39196-0f3ec6fc-3df2-001f29541714/test # stat w7x64-clean-flat.vmdk
  File: w7x64-clean-flat.vmdk
  Size: 34359738368     Blocks: 67108864   IO Block: 131072 regular file
Device: 831f0b1fc2871364h/9448282774282113892d  Inode: 4225796     Links: 1
Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-01-24 12:38:34.000000000
Modify: 2015-01-24 13:06:49.000000000
Change: 2015-01-24 13:06:49.000000000
 

 

So we got 28 min for 34GB, roughly 20MBps, which yeah is blowing you away on crappier hardware.... Hmmmmmm??

you could try dd command vs cp, doing a test now looks like 4.9GB in 3 minutes vs the 3.8 with the cp command, let me try uping the bs from 1M

 

edit:  Well using dd seems to get me the speeds I saw with download..  About 28MBps vs the 20 was seeing with cp.

 

edit2:  ok -- seems cp has really be depreciated for a while on esxi.. your suppose to use vmfsktools command..

 

http://www.vmware.com/pdf/esx_3p_scvcons.pdf

For performance and data placement reasons, do not use scp or cp; instead, use vmkfstools, the Virtual Machine Importer tool from VMware, or the SDK APIs to manipulate your virtual disks. You should see very significant performance improvements if you use the recommended tools.

 

So doing a copy of that same vm using

/vmfs/volumes/535605bc-d0c25a0d-7cf0-001f29541714/w7 # vmkfstools -i /vmfs/volumes/datastore0/w7/w7x64-clean.vmdk /vmfs/volumes/datastore1/test/test.vmdk
Destination disk format: VMFS zeroedthick
Cloning disk '/vmfs/volumes/datastore0/w7/w7x64-clean.vmdk'...
Clone: 100% done.
 

was done in 4.25 min or 133MBps

 

/vmfs/volumes/54c39196-0f3ec6fc-3df2-001f29541714/test # stat test.vmdk
  File: test.vmdk
  Size: 514             Blocks: 0          IO Block: 131072 regular file
Device: 831f0b1fc2871364h/9448282774282113892d  Inode: 8420100     Links: 1
Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-01-24 13:38:46.000000000
Modify: 2015-01-24 13:42:57.000000000
Change: 2015-01-24 13:42:57.000000000
 

 

post-14624-0-99362500-1422107393.png

 

Argh the dd is an annoying busybox version, just started running it with all the ibs and out set to 16MB and will see what happens with an 8GB file!

Did not know that about cp on ESXi, thanks budman! I'll see if DD increases the speed and if not will retry with that command, I'm assuming it doesn't bother copying the blank space and speeds up transfer that way, the VM I'm copying now is 90% utilised so it probably wouldn't save much time.

 

EDIT: OK no there is definitely something not right, 3Gbps link speeds between the SATA/SAS disks and the controller, 0.2GB copied in 30 seconds.

Edited by n_K

Dude see my edit.. Use vmkfstools -i src dst

 

My test shows a SCREAMING difference..   What you got to loose..  It sure can not be any slower than your dd or cp commands ;)

  On 24/01/2015 at 12:52, n_K said:

I can't do that as I've got snapshots disabled and have all changes written to the disk as they're performed.

 

Curios as to why you disabled snapshots.

Just tried on a 23GB disk after the 8GB DD finished budman;

/vmfs/volumes/508aa94d-fbcf15ba-0faf-68b599b49d30/Jan 24 2015/W7 # vmkfstools -i /vmfs/volumes/Main/Windows\ 7\ Pro
fessional-N\ x64/Windows\ 7\ Professional-N\ x64.vmdk /vmfs/volumes/Backup\ Disk/Jan\ 24\ 2015/W7/
Destination disk format: VMFS zeroedthick
Cloning disk '/vmfs/volumes/Main/Windows 7 Professional-N x64/Windows 7 Professional-N x64.vmdk'...
Failed to clone disk: The file already exists (39).
Ignore, I'm being dense and not putting in the filename!

 

  On 24/01/2015 at 14:11, Depicus said:

Curios as to why you disabled snapshots.

Uses space which I don't have that much of.

Yeah that is much better ;)  Should be a helpful thread for other people I think.. I don't normally move files between datastores

 

Now not sure on what your original was..  Was it thick, or thin?  Notice it defaults to thickzero'd -- so if was thin before, your backup isn't.  if you want to maintain thin you can do -d thin on the end.  But that took about double the time to copy..  But I would think even 10 minutes for you would be much better than what you were seeing.

Not so sure its really a performance booster doing that any more.. If on SSD datastore makes no difference for sure.. So your storage is local, is it VAAI ??  Do you see hardware acceleration when you look at your datastores?

 

There are lots of variables at play when it comes to performance - a we see in this example using a deprecated common way that many people would do can have huge performance implications.. Comes down to your requirements.  I for sure have my storage over provisioned for sure.. Small datastore..  And play with lots of vms, I don't see any reason to suck up all the space with zeros ;)

This topic is now closed to further replies.
  • Posts

    • Edge for Business gets secure password deployment for organizations by Paul Hill Microsoft Edge for Business now offers organizations secure password deployments as a generally available feature, the Redmond giant said. Instead of users sharing passwords on sticky notes or via email to access certain websites or tools, admins can deploy encrypted shared passwords to specific users within their organization. When a user receives a password, it is stored in their Edge password manager and can be used to log into websites seamlessly using autofill. Microsoft has made this enterprise-grade solution available to customers at no additional cost. How it works and the user experience Administrators have to manage the feature via the Microsoft Edge management service within the Microsoft 365 admin center. From there, they can add, update, and revoke credentials for specific user groups through configuration policies. Once an admin has set it up and shared passwords with users, the users will see the passwords in their Edge password manager and can be used with autofill on corresponding websites. The passwords are tied to work profiles in Edge on managed Windows devices to limit their misuse. Further boosting security, the shared passwords cannot actually be viewed, edited, or deleted (unless the website allows), or exported from the password manager. This is a good addition for security because if an unauthorized user gains physical access to the computer, they cannot learn what the password is. Administrators reading this do need to be aware of an important caveat related to developer tools. A motivated user who wants to reveal the passwords can do so by digging into the developer tools, for this reason, you should consider restricting access to the developer tools by configuring the DeveloperToolsAvailability policy. The underlying security and encryption Microsoft’s new secure passwords feature has been built using the Information Protection SDK. The passwords are encrypted and the encryption is tied to Entra identities which lets organizations enforce them without manual key management. The decryption of the passwords takes place at runtime using the same SDK, validating the user’s identity. Availability and getting started Secure password deployment is available through the Edge management service in the Microsoft 365 admin center. Once in the admin center, you should choose an existing configuration policy or create a new one. Inside the policy, go to the Customization Settings tab and then to the Secure password deployment page. To use this feature you must have a Microsoft 365 Business Premium, E3, or E5 subscription. The feature also requires the Edge admin or Global admin role. Source: Microsoft
    • Is it though?  I built a new rig a few months ago and it was literally impossible to get one without RGB, but within 10 minutes of setting it up, I turned all that crap off.  It was REALLY distracting, and who needs additional heat INSIDE a PC? It's popular on YouTube for sure, it's neat looking and whatnot, but it's about as practical as a coffee cup with a hole in it. As for the price, a non-enthusiast would just see something priced way above what they can get from a retailer brand new...
    • RollBack Rx Pro 12.9 Build 2710971022 by Razvan Serea RollBack Rx is a robust system restore utility that enables home users and IT professionals to easily restore a PC to a time before certain events occurred. In essence, it turns your PC into a Instant Time Machine. Regardless of what happens to your PC your can quickly and easily restore your PC to a previous time. Making it easy to rescue you from any PC disaster - saving time, money and PC trouble. Windows System Restore only restores Windows system files and some program files. In addition, if Windows crashes to a point were Windows itself can not boot up (ie. BSOD*) you would not be able to access your Windows System Restore points. In contrast, the RollBack Rx technology works at the sector level of the hard drive and restores everything! - right down to the last byte of data. It sits below Windows. So even if Windows crashes, there’s a sub-console (mini OS) that boots prior to windows. This allows you to access Rollback Rx and go back to a point in time when your system was working trouble-free. Key Features Go back to any previous point in time within seconds. Go back minutes, hours, days, weeks, or even months to any previous snapshot. Does not affect computer performance, uses minimal system resources. Supports unlimited snapshots. Creates a complete system snapshot without having to restart the system. Reverse any system crash within seconds (even if Windows cannot startup). Back out of any failed program, OS updates, and botched updates. Recover from any malware or virus attack within seconds. Works with VMWare and Virtual Machines, both as a host or within the virtual machine as a client. Supports Multi-boot, Multi OS workstations. Lock snapshots to prevent deletion. Intuitive GUI based snapshot manager. Explore, browse, and retrieve files and folders from any snapshot. Drag and drop them into your active system. Roll backwards as well as forwards to any available system snapshot. Allows users to safely test any software. Fast, 100% complete uninstaller. Retrieve files from a crashed PC, even if Windows cannot boot. Access control – manage levels of multiple user and administrative privileges. Automatically schedule snapshots to be taken on a fixed schedule or upon execution of specific files (ie. setup.exe) as well as manually. 256 bit AES snapshot encryption. Prevent unauthorized data theft in case of a stolen laptop. Group Management and Enterprise Network Administration Control (FREE utility). Comes with Stealth Mode where you can hide the RollBack Rx tray icon and splash screen (seen during bootup) Change the startup hotkey for sub-console access (default is HOME). Built-in snapshot defragmenter which will optimize system resources and recover free space. Option to keep files and folders unchanged when you roll-back. Advanced setup configuration wizard for system administrators which will set deployment options and predefined RollBack Rx settings. Offers detailed program operation logging. Supports all industry-standard deployment options including silent installations and pre-installation configuration. Explore RollBack Rx Pro with a 14-day trial, fully functional on Windows 11, 10, 8, and Windows 7 SP1** (32 and 64-bit). RollBack Rx Pro 12.9 Build 2710971022 changelog: General Add PnpLockdown in shieldm.inf Fix registry exclusion problem in Windows 11 24H2 release Add detailed logging for file filter driver Add detailed logging for Windows update Add time stamp to kernel drivers Change kernel driver and Win32 IRP structure Other small bug fixes / typos reported through tech support Endpoint Manager Add client report dashboard Add sound effect when receiving a EPM message. Keep EPM message history Fix bug that oversized Windows symbol files cannot be downloaded Download: RollBack Rx Pro 12.9 | 61.0 MB (Shareware) View: RollBack Rx Home Page Get alerted to all of our Software updates on Twitter at @NeowinSoftware
    • Universal Media Server 14.12.1 by Razvan Serea Universal Media Server is a DLNA-compliant UPnP Media Server. UMS was started by SubJunk, an official developer of PMS, in order to ensure greater stability and file-compatibility. The program streams or transcodes many different media formats with little or no configuration. It is powered by MEncoder, FFmpeg, tsMuxeR, AviSynth, MediaInfo and more, which combine to offer support for a wide range of media formats. Because it is written in Java, Universal Media Server supports all major operating systems, with versions for Windows, Linux and Mac OS X. To see a comparison of popular media servers, click here. Universal Media Server 14.12.1 changelog: General Added status page to readme Fixed videos not being marked as fully played (#5373) (thanks, @Fredo1650!) Fixed adding YouTube channels from handle URLs (URLs with @ in them) Fixed handling special characters on Linux (#5100) (thanks, @LaTeteDansLesEtoiles!) Fixed directory browsing crash (#5189) (thanks, @jt-gilkeson!) Fixed FFmpeg on Linux x86_64 and arm64 (#5465) (thanks, @KanjiMonster!) Fixed logspam like "Could not hydrate device or its services from descriptor" (#5292) (thanks, MTOakey!) Fixed broken YouTube video playback Fixed web interface E2E testing on CI using outdated code because of overeager caching Fixed broken video playback when burning subtitles to H.265 via FFmpeg (#5486) Improved logging Translation updates via Crowdin Chinese (Simplified) (59%) (thanks, 無情天!) Dutch (41%) (thanks, Matthias!) Hungarian (86%) (thanks, Zoltán Rózsa!) Japanese (69%) (thanks, Yukihuru!) Download: Universal Media Server 14.12.1 | 203.0 MB (Open Source) Download: Other operating systems View: Universal Media Server Website | Screenshot Get alerted to all of our Software updates on Twitter at @NeowinSoftware
    • You sign your rights to reddit when you write on their platform. Free labour for them to make money. The AI companies should also take advantage of that free labour.
  • Recent Achievements

    • Week One Done
      somar86 earned a badge
      Week One Done
    • One Month Later
      somar86 earned a badge
      One Month Later
    • Apprentice
      Adrian Williams went up a rank
      Apprentice
    • Reacting Well
      BashOrgRu earned a badge
      Reacting Well
    • Collaborator
      CHUNWEI earned a badge
      Collaborator
  • Popular Contributors

    1. 1
      +primortal
      510
    2. 2
      ATLien_0
      260
    3. 3
      +Edouard
      190
    4. 4
      +FloatingFatMan
      175
    5. 5
      snowy owl
      133
  • Tell a friend

    Love Neowin? Tell a friend!