Recommended Posts

Hi,

So I've got a HP DL160 G6 running with an HP SmartArray p410i 1GB super-capacitor RAM-based cache, and am using 3*15K RPM 73GB SAS drives in a RAID5 configuration and I have a single 500GB WD enterprise SATA drive connected too...

Now I know in comparison to the Dell PERC6 that the HP SA is a steaming pile of trash, it's much slower for reasons I've never been able to work out despite having a much better spec.

But this always gets me, I back up my VMs using the SSH server in ESXi and I have downtime of 50 minutes to transfer a 22GB VM, which is absolutely bloody rediculous (it's still not finished copying) and I'm just at a loss as to why. The 500GB drive isn't anything amazing, just a standard 7200 RPM drive but to transfer a VM at less than 7MBps is just... I'm speechless. I can transfer files faster on a 32 bit SCSI card faster than this.

 

Does anyone have a similar setup or any tips for this? (It's ESXi 5.1 and both file-systems are VMFS-5) I've got 4 more VMs to backup after this then apply some system updates and it's looking like it's going to take the whole day. I don't think it's a hardware speed problem, I'm thinking it's down to ###### poor drivers from HP for the p410i (hpaucli is [in comparison to dell's PERC utilities] a complete joke) or something up with ESXi but don't really know how I can go about test either or speeding it up.

Is the vmkern network on the same or a different physical port from the interface on your primary vSwitch?

 

I read that performance is awful unless this is separated (although I have never tried).

  On 24/01/2015 at 12:18, Fahim S. said:

Is the vmkern network on the same or a different physical port from the interface on your primary vSwitch?

 

I read that performance is awful unless this is separated (although I have never tried).

Ah, I don't mean I'm coping the data over SSH, I just mean I'm connected via SSH to do the file copying. It's going direct disk-to-disk on the same host.

At 11:58 I started copying a 10GB VM, 25 minutes later it's still not finished, so this copy speed is definitely slower than 7MBps.

How exactly are you doing this copy.. Are you going to the datastore and downloading the vm disk?

 

How is your vmkern - is it shared with another nic.. I noticed a huge increase in performance when broke out vmkern to its own port group on its own nic..

 

post-14624-0-74552400-1422101386.png

 

So here I started a download of vm - clicked go at 6:07:30..  Its downloading now, This is off a HP N40L with cheap nics added, the vmkern is using the built in nic I do believe.. If I look at my network performance for my pc I downloading the file too - getting pretty decent network util

 

post-14624-0-94754700-1422101722.png

 

Ok done...  looks like 20min

 

post-14624-0-40134400-1422102622.png

 

See the time created, and then last modified time..  So lets call it 34GB / 20 min = 1.7GB a min = 28MBps, which clearly is not full speed of my network..  I normally see double or triple that from the nas on the same esxi..  But it is inline with with how the vmkern works, etc..

 

edit:  So your dong disk to disk copy on your esxi host?  via cli command..  Let me test that with this same 34GB file..  I have a SSD datastore and the 250GB disk it came with as datastore as well..  BRB

 

Ok so started at 6:38:30 and so far its copied 3.8GB in 3 minutes.. 

so at 10 min mark bit over 12GB

 

/vmfs/volumes/54c39196-0f3ec6fc-3df2-001f29541714/test # ls -la
total 12049416
drwxr-xr-x    1 root     root           420 Jan 24 12:38 .
drwxr-xr-t    1 root     root          1400 Jan 24 12:37 ..
-rw-------    1 root     root     12343582720 Jan 24 12:48 w7x64-clean-flat.vmdk
/vmfs/volumes/54c39196-0f3ec6fc-3df2-001f29541714/test #

 

So I would have to say, seems like a bit slower than the download copy..  But that 250GB disk is pretty old crappy disk ;)

 

So at 20min, 24GB roughly looks like about 20MBps which yeah is like 3x what your seeing and this is just the controller that that comes with the N40L..

Not an answer to your problem but you could use ghettoVCB which is a free backup script for ESXi and works quite well. Bit fiddly to set up but once done it's fine. 

 

I have it scripted so I just log into the host and run a script to do the backup then once done I can copy the backup files without having to take down any of the servers.

I've now shut down all VMs except one (my W7 remote management VM) and I've taken a screenshot of it... Something really is not right here.

aNS6w1g.png

EDIT: Changed around the graph output (bit hard on a small VNC screen) and it's apparently reading at 6MBps from the main drive and writing to the backup drive at 15MBps... I can't understand how it's writing twice the data it's reading!

 

 

  On 24/01/2015 at 12:45, Depicus said:

Not an answer to your problem but you could use ghettoVCB which is a free backup script for ESXi and works quite well. Bit fiddly to set up but once done it's fine. 

 

I have it scripted so I just log into the host and run a script to do the backup then once done I can copy the backup files without having to take down any of the servers.

I can't do that as I've got snapshots disabled and have all changes written to the disk as they're performed.

Edited by n_K

Ok so it finished

 

/vmfs/volumes/54c39196-0f3ec6fc-3df2-001f29541714/test # stat w7x64-clean-flat.vmdk
  File: w7x64-clean-flat.vmdk
  Size: 34359738368     Blocks: 67108864   IO Block: 131072 regular file
Device: 831f0b1fc2871364h/9448282774282113892d  Inode: 4225796     Links: 1
Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-01-24 12:38:34.000000000
Modify: 2015-01-24 13:06:49.000000000
Change: 2015-01-24 13:06:49.000000000
 

 

So we got 28 min for 34GB, roughly 20MBps, which yeah is blowing you away on crappier hardware.... Hmmmmmm??

you could try dd command vs cp, doing a test now looks like 4.9GB in 3 minutes vs the 3.8 with the cp command, let me try uping the bs from 1M

 

edit:  Well using dd seems to get me the speeds I saw with download..  About 28MBps vs the 20 was seeing with cp.

 

edit2:  ok -- seems cp has really be depreciated for a while on esxi.. your suppose to use vmfsktools command..

 

http://www.vmware.com/pdf/esx_3p_scvcons.pdf

For performance and data placement reasons, do not use scp or cp; instead, use vmkfstools, the Virtual Machine Importer tool from VMware, or the SDK APIs to manipulate your virtual disks. You should see very significant performance improvements if you use the recommended tools.

 

So doing a copy of that same vm using

/vmfs/volumes/535605bc-d0c25a0d-7cf0-001f29541714/w7 # vmkfstools -i /vmfs/volumes/datastore0/w7/w7x64-clean.vmdk /vmfs/volumes/datastore1/test/test.vmdk
Destination disk format: VMFS zeroedthick
Cloning disk '/vmfs/volumes/datastore0/w7/w7x64-clean.vmdk'...
Clone: 100% done.
 

was done in 4.25 min or 133MBps

 

/vmfs/volumes/54c39196-0f3ec6fc-3df2-001f29541714/test # stat test.vmdk
  File: test.vmdk
  Size: 514             Blocks: 0          IO Block: 131072 regular file
Device: 831f0b1fc2871364h/9448282774282113892d  Inode: 8420100     Links: 1
Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-01-24 13:38:46.000000000
Modify: 2015-01-24 13:42:57.000000000
Change: 2015-01-24 13:42:57.000000000
 

 

post-14624-0-99362500-1422107393.png

 

Argh the dd is an annoying busybox version, just started running it with all the ibs and out set to 16MB and will see what happens with an 8GB file!

Did not know that about cp on ESXi, thanks budman! I'll see if DD increases the speed and if not will retry with that command, I'm assuming it doesn't bother copying the blank space and speeds up transfer that way, the VM I'm copying now is 90% utilised so it probably wouldn't save much time.

 

EDIT: OK no there is definitely something not right, 3Gbps link speeds between the SATA/SAS disks and the controller, 0.2GB copied in 30 seconds.

Edited by n_K

Dude see my edit.. Use vmkfstools -i src dst

 

My test shows a SCREAMING difference..   What you got to loose..  It sure can not be any slower than your dd or cp commands ;)

  On 24/01/2015 at 12:52, n_K said:

I can't do that as I've got snapshots disabled and have all changes written to the disk as they're performed.

 

Curios as to why you disabled snapshots.

Just tried on a 23GB disk after the 8GB DD finished budman;

/vmfs/volumes/508aa94d-fbcf15ba-0faf-68b599b49d30/Jan 24 2015/W7 # vmkfstools -i /vmfs/volumes/Main/Windows\ 7\ Pro
fessional-N\ x64/Windows\ 7\ Professional-N\ x64.vmdk /vmfs/volumes/Backup\ Disk/Jan\ 24\ 2015/W7/
Destination disk format: VMFS zeroedthick
Cloning disk '/vmfs/volumes/Main/Windows 7 Professional-N x64/Windows 7 Professional-N x64.vmdk'...
Failed to clone disk: The file already exists (39).
Ignore, I'm being dense and not putting in the filename!

 

  On 24/01/2015 at 14:11, Depicus said:

Curios as to why you disabled snapshots.

Uses space which I don't have that much of.

Yeah that is much better ;)  Should be a helpful thread for other people I think.. I don't normally move files between datastores

 

Now not sure on what your original was..  Was it thick, or thin?  Notice it defaults to thickzero'd -- so if was thin before, your backup isn't.  if you want to maintain thin you can do -d thin on the end.  But that took about double the time to copy..  But I would think even 10 minutes for you would be much better than what you were seeing.

Not so sure its really a performance booster doing that any more.. If on SSD datastore makes no difference for sure.. So your storage is local, is it VAAI ??  Do you see hardware acceleration when you look at your datastores?

 

There are lots of variables at play when it comes to performance - a we see in this example using a deprecated common way that many people would do can have huge performance implications.. Comes down to your requirements.  I for sure have my storage over provisioned for sure.. Small datastore..  And play with lots of vms, I don't see any reason to suck up all the space with zeros ;)

This topic is now closed to further replies.
  • Posts

    • This super-powerful GaN charger with four ports is 50% off by Taras Buria Some time ago, I reviewed the Cuktech 10, a powerful 100W GaN charger with three ports. It left positive impressions, and since then, it has served me as my primary charger for my phone, watch, laptop, and tablet. Recently, Cuktech offered me the opportunity to take a look at the model 15, a more powerful 140W GaN charger, this time, with four ports. Right now, this powerful charger is available with a massive discount at just $50.99 (with coupon applied). The Cuktech 15 is the same as the Cuktech 10, just slightly bigger, more powerful, and with one extra port. It has a light metallic finish, but overall, it retains the brand's identity and features. The four ports are well spaced out, and the black plastic insert with a cyan rim has a ribbed texture for better grips. Although I live in Europe, Cuktech sent me a US variant with a retractable plug. Okay, I guess. The ability to retract the plug makes it extra portable, which is nice. A high-power 240W five-foot cable is also included. The charger measures 3.11 x 2.56 x 1.26 inches and weighs 0.737 lbs. The Cuktech 15 has four ports: two high-power Type-C ports, one Type-C port with a lower output, and one Type-A port. The first two Type-C ports can deliver a full 140W in single-port mode (PD 3.1 supported), which is nice—no asterisk or caveats here with combined power or something. If you need the full 140W for one device, you get it. Ports are capable of working in the following modes: Single-port Type-C1 / C2: 140W max 5V 2A, 5V 3A, 9V 3A, 11V 6.1A, 12V 3A, 15V 3A, 20V 5A, 28V 5A Type-C3: 33W max 5V 2A, 5V 2.4A, 9V 2A, 12V 1.5A, 11V 3A Type-A: 18W max 5V 2A, 5V 3A, 9V 2A, 12V 1.5A Multi-Port Type-C1 + C2: 100W + 33W or 65W + 65W Type-C1/C2 + Type-C3: 100W + 33W Type-C1/C2 + Type-A: 100W + 18W Type-C1 + Type-C2 + Type-C3: 65W + 60W + 7.5W or 45W + 45W + 18W Type-C1 + Type-C2 + Type-C3 + Type-A: 65W + 60W + 7.5W As you can see, the charger is pretty robust, and it can power two pretty powerful laptops at once and even have enough oomph to charge a smartphone, albeit at a lower power. Another thing worth mentioning is that the Cuktech 15 delivers "clean" power with pretty low pulsations at about 25-50 mV. The rule of thumb is that the lower the pulsations, the better the charger is for your device's battery health. In this area, Cuktech's charger does not disappoint, and they deliver way better results than 100 mV, which is considered a standard for a good charger. Cuktech uses gallium nitride technology, which enables smaller, more powerful and efficient charging. Speaking of efficiency, the charger is rated for 78% average or 64% at a 10% load. When charging at 120W, I received an average of 80-85%, which is good. Of course, when charging at peak power, it gets hot, but not too much. The Cuktech 15 140W usually costs $99.99, which is undoubtedly not cheap. However, right now, you can get it for half the price, which is a very good deal, considering you get a high-quality charger with plenty of ports and very high power output. Like with the Cuktech 10, you cannot go wrong with this one. CUKTECH 15 140W four-port GaN charger - $50.99 | 30% off + a 20% off coupon As an Amazon Associate, we earn from qualifying purchases.
    • Awesome book, just hope they don't screw up Rocky and the story
    • KDE's KClock is getting Wayland Picture-In-Picture support by David Uzondu The KClock app for KDE Plasma over the years has received a number of updates, like better integration with KRunner and a dedicated background service (kclockd) for managing alarms. Now, it looks like KDE devs want to add something cool: pop-out timers using the new Picture-in-Picture protocol for Wayland. Image: Kai Uwe Broulik This all started from a simple observation. Kai Uwe Broulik, a KDE developer, saw someone using a small timer window during a presentation and thought it was a good idea. The problem is that achieving this kind of "always on top" behavior is handled differently between the old X11 display server and the newer Wayland. With X11, an application could pretty much do whatever it wanted. If a program wanted to draw a drop-down menu, it would just create a borderless window, place it in a specific spot, and grab all user input. Wayland operates on a different philosophy. As Broulik notes, under Wayland, the application describes what it wants, and the compositor gets to decide how to handle it. A drop-down menu is an XDG Popup. The application tells the compositor which button spawned it, and the compositor handles the placement and behavior. This is much more secure and consistent. It also means an application cannot just decide to keep its window on top of everything else. This restriction prevents a web browser from implementing an overlay video player under Wayland. To get around this in a standardized way, a proper Wayland support model for Picture-in-Picture, or PiP, was needed. Enter the xx-pip-v1 protocol. It is a new protocol designed specifically for creating floating PiP windows, and KWin, Plasma's compositor, recently gained support for it. Because it is an experimental protocol, its use is gated behind an environment variable, KWIN_WAYLAND_SUPPORT_XX_PIP_V1. A new protocol is fine for demos, but it needs a real application to find its weaknesses. So Broulik implemented it in KClock. This work allows KClock to offer pop-out timers and even a pop-out stopwatch in a small PiP window. The user could get system-wide options to control where the PiP window appears, or if it appears at all, and have that setting apply to every single application that uses the protocol. You can check out the merge request on GitLab for more technical details about this feature.
    • Nothing of course; it is just a classic syndrome of MAGA pathology.
    • They use it to assessment's in the UK already. Have a friend who said they basically feed response's and prompt during assessment's for many mental health conditions now (essentially tick boxes these days enough ticks and you got it) the AI can prompt questions to ask if its unsure how to score something. It's really good at recognising fractures and broken bones also.
  • Recent Achievements

    • One Month Later
      CHUNWEI earned a badge
      One Month Later
    • Week One Done
      TIGOSS earned a badge
      Week One Done
    • First Post
      henryj earned a badge
      First Post
    • First Post
      CarolynHelen earned a badge
      First Post
    • Reacting Well
      henryj earned a badge
      Reacting Well
  • Popular Contributors

    1. 1
      +primortal
      479
    2. 2
      +FloatingFatMan
      196
    3. 3
      ATLien_0
      163
    4. 4
      Xenon
      81
    5. 5
      Som
      77
  • Tell a friend

    Love Neowin? Tell a friend!