Recommended Posts

Going through the logs, found this:

2012-05-15T13:45:52.954Z: <<throttled>> Storage I/O Control: connection with vobd failed, error code: -1 errno: 2[/CODE]

Not sure what it is, but it's mentioned CONSTANTLY!

Additionally:

[CODE]
2012-05-16T07:37:08.952Z: [scsiCorrelator] 64332112349us: [vob.scsi.device.io.latency.high] Device t10.ATA_____SAMSUNG_HD502IJ_________________________S13TJDWQ620722______ performance has deteriorated. I/O latency increased from average value of 22658 microseconds to 684718 microseconds.

2012-05-16T07:37:08.952Z: [scsiCorrelator] 64336113058us: [esx.problem.scsi.device.io.latency.high] Device t10.ATA_____SAMSUNG_HD502IJ_________________________S13TJDWQ620722______ performance has deteriorated. I/O latency increased from average value of 22658 microseconds to 684718 microseconds.

2012-05-16T08:03:08.708Z: [scsiCorrelator] 65891868893us: [vob.scsi.device.io.latency.improved] Device t10.ATA_____SAMSUNG_HD502IJ_________________________S13TJDWQ620722______ performance has improved. I/O latency reduced from 684718 microseconds to 174335 microseconds.
[/CODE]

More:

My Windows 7 VM is basically completely unresponsive from an I/O standpoint. System is very very slow while copying a VMDK between Datastores. Yet the HDD Indicator light on the Server itself is only flashing once in a while, indicating that minimal I/O is happening on the disk itself.

Sorry mate i am still a bit of a noobie with this stuff but what are you running to see this? I can check and see if my system is doing the same thing? On the ESXi installation i tried that and imported the VM's and saw the same issue.

Right so the issue is happening on an ESXi Level, not a VM Level. So further diagnostics on the VMs are pointless.

I've setup a NFS Share on my File Server and attempted to copy over an ISO from one of my DataStores, the speed was averaging about 5% of the 1Gbps connection, 20 minutes for a 3GB file. Everything else can max out this Servers connection easily. I'll attempt to boot a VM over the network later and measure performance again, this will 100% prove is this is a networking issue or an I/O issue as I'll be able to monitor I/O directly on the 2008R2 server.

I'm going to try moving the 'Management Network' to a different LAN and see if that provides me with better performance. Then this weekend I'll install ESXi again on the 250GB Disk that's included with the server and test the same.

Sorry mate i am still a bit of a noobie with this stuff but what are you running to see this? I can check and see if my system is doing the same thing? On the ESXi installation i tried that and imported the VM's and saw the same issue.

SSHing into the server, and running cat on the files in /var/log

Have you considered that maybe there is just an incompatability with your hard drive and ESXi? Have you tried some other type of drive to rule that out? Obviously it is giving you some kind of I/O latency error. What drive is Budman using in his as he seems to have no problems and everything else is identical?

This is such a weird problem. I don't understand how yours can be not working with identical hardware to budmans. I think honestly that your drives are faulty in some way but I have no way to back that up it's just a gut feeling. I would seriously try some new drives with a fresh installation of ESXi and two copies of Windows. It's time to start ruling stuff out as tweaking settings isn't fixing it.

Late entrant to the thread but have been following it on and off.

This may be completely useless, but your mentioning NFS on this page got me thinking - What if you get rid of NFS and use something else? IIRC, the NFS client is v3 for VSphere 5 which means all the traffic for the ESX host uses just 1 single TCP connection.

Tried another disk, and deffo a problem with ESX, Same Disk, same hardware with native windows 7 i get about 80 MBPS, with ESXi i get about 10MBPS :-(

ESXi 5, uploading to a datastore via WAN 100Mbps link from home at around 5-6MB/s. Not seeing the issues reported here, DL380G6 servers mostly.

Its a given that moving files to the datastore is going to be slow -- this is all over the forums on vmware, etc.

But thats not really the issue we are troubleshooting, the issue comes down to this.

between 2 vms, on the same host, same vswitch using the vmxnet3 drivers -- he is seeing LOW iperf speeds in the range of 300mbps, on the other hand on the exact same hardware N40L

Same test, 2 windows vms on the same host, same vswitch using vmxnet3 drivers I see like 1.+ Gbps

this should have nothing to do with disks, etc.

Boom - All non-essential VMs moved to NFS and essential VMs moved to the 500GB Samsung Drive.

I'll zero-fill the SSD and 250GB HDD to ensure disk health and reinstall ESXi onto the 250GB Drive this weekend.

If that fails, I'll get a saucepan, put it on my head, and whack it with a wooden spoon until I no longer care about the performance issues.

Guys!!! It's a CPU issue!

Windows 7 Virtual Machine - iperf -s -w 256k

Galactica Physical Machine - iperf -c 10.0.1.11 -w 256k

Speed = Approx 400Mbits/sec

CPU Use = 100% on Windows 7 VM during Transfer.

pfSesne VM - same situation!!!!

So - why is Budman's performance better in this instance? What would cause a VM to use 100% CPU time for a basic network transfer?

Do you have your VM's limited to what they can use?

This could be the reason your seeing the limit.

Let me run another iperf and check what my vms cpu does and what it maxes out at, etc.

Ok -- here is client, server and host while doing iperf and getting 1.3+Gbps

client

post-14624-0-66086900-1337275902.jpg

server

post-14624-0-49541500-1337275757.jpg

Host

post-14624-0-68526400-1337275770.jpg

And remember we went over this before, my vms only have 1 cpu assigned to them. and here you can see not restricting their usage on the host

post-14624-0-45999700-1337276022.jpg

What does your cpu look like -- are you actually hitting 100% and just staying there.. That would explain why your seeing a limit on the speed.

Hey,

My VMs only have one core accessible to them. Done a complete reinstall today onto the Physical Hard Drive. So my setup is essentially identical to yours now! Setup the 8GB SSD as a Cache drive.

Can you SSH into your pfSense VM and run top and iperf server on it and let me know what CPU% you get during the transfer. If you get similar issues over pfSense to what I get on Winodws that further helps diagnose this.

Additionally how exactly do you have CPU setup in VM settings?

Strange - the Domain Controller VM doesn't get maxed out like the Windows 7 VM does when performing Network Tasks. Did you make any registry tweaks to Networking? I believe you mentioned something about disabling the IPv6 and Tunneling adapters at one point? the Server 2008 R2 is able to get approx 690Mbits/sec compared with 400ish from the Windows 7 VM.

Edit: Server 2003 R2 was able to achieve 772Mbits/sec

here is my win7 client to pfsense running iperf server

post-14624-0-42024700-1337278877.jpg

But I have never seen great performance to and from windows with pfsense, but as you see my windows client is only see about 50% cpu and 500Mbps -- which is much better than yours????

My pfsense is using the e1000 drivers because I was having an issue with access to outside vpn from my laptop when pfsense used the vmxnet3 drivers.

Have not had time to dig into why.. But I think it might be related to the way pfsense labels the interfaces while using this drivers vmx3f0 and vmx3f1 -- which looks to cause a problem with the vlan stuff, and might have be related??? All I know for sure is an ipsec\L2TP vpn to an outside server from a client on my network doesn't work with the vmxnet3 drivers.

Maybe I have tweaked my windows install different than you? I was playing with adapter settings the other day and netsh global options, RSS, Chimney offload, etc I was able to get 1.7Gbps the other day while playing. Now only seeing like 1.3 -- I might of reset something or made another tweak without check performance?

Your limits seem real close to that vmware thread I saw about the guy seeing a 300Mbps limit on his vswitches.

Maybe we should use a specific linux live cd or something as our benchmark for vswitch performance.. Could just fire up 2 exact vms that way and compare performance.

I believe I am running a different version of pfsense than you for one thing. I am running 2.1 32bit, with latest gitsync - its running 8.3 freebsd, etc.

networking settings are different in 2k8 vs w7 for sure -- even just out of the gate.

But yeah I have disabled all the ipv6 stuff on my windows vms. But I do run it on my workstation, and my linux vm and pfsense. But sure its been correctly setup ;) ie the teredo and 6to4 **** disabled because not using that!

as I mentioned in previous post - maybe picking a linux live distro to boot would be a better apples to apples test.

YEAAAAAAHHHHH!!!

post-180771-0-31327900-1337279754.png

Enabled VMCI and Communication between VMs just went through the roof!

That combined with the Server 2003 R2 general networking performance means this is somehow now only effecting the Windows 7 VM. And that I can more than live with.

So, for the VMs dealing with downloading I'm going to allocate 2 CPUs so one can handle the Downloading and Networking while the other can handle FileSystem and File Operations (PAR+RAR).

Now I've just gotta figure out how to get Server 2003 R2 to actually see the two CPUs without doing a full reinstall.

as I mentioned in previous post - maybe picking a linux live distro to boot would be a better apples to apples test.

Sure - Arch might be the best bet because we can turn basically everything off on a tiny install. Want me to build a VMDK and send you the package?

nice

C:\Windows\System32>iperf -c storage.local.lan -w 256k

------------------------------------------------------------

Client connecting to storage.local.lan, TCP port 5001

TCP window size: 256 KByte

------------------------------------------------------------

[268] local 192.168.1.216 port 49182 connected with 192.168.1.8 port 5001

[ ID] Interval Transfer Bandwidth

[268] 0.0-10.0 sec 1.95 GBytes 1.67 Gbits/sec

Now I need to put my tweaks back - I had backed them all out, atleast the ones I could remember and was seeing like 1.15Gbps -- now its freaking rockin!! Lets see what happens when I put some of the tweaks back.. Lets see if can get that 1.8 or 2 mark ;)

ok check this out -- still playing with settings

C:\Windows\System32>iperf -c storage.local.lan -w 256k

------------------------------------------------------------

Client connecting to storage.local.lan, TCP port 5001

TCP window size: 256 KByte

------------------------------------------------------------

[268] local 192.168.1.216 port 49307 connected with 192.168.1.8 port 5001

[ ID] Interval Transfer Bandwidth

[268] 0.0-10.0 sec 2.17 GBytes 1.86 Gbits/sec

But seems disable interrupt moderation on the virtual nic sure doesn't hurt ;)

This topic is now closed to further replies.
  • Posts

    • The quantum search for Time's origin had an equally mind-boggling conclusion by Sayan Sen Image by Steve Johnson via Pexels A theoretical study from researchers at the University of Surrey suggested that the direction of time may not be fundamentally fixed in certain quantum systems. The work, published in Scientific Reports, examined how the “arrow of time” could emerge from microscopic physics and found that time-reversal symmetry can remain intact even in models used to describe processes such as energy loss and thermalisation. The arrow of time refers to the observed one-way direction from past to future in everyday life. In macroscopic processes, this is easy to see. Spilled milk spreads across a table and does not gather back into a glass, and heat flows from hotter objects to colder ones. These processes shape the common sense idea that time moves in a single direction. However, at the level of fundamental physics, many equations do not prefer a direction of time. Time-reversal symmetry means that the same physical laws can describe a system whether time moves forward or backward. This has made it difficult to explain why irreversible behaviour appears in the large-scale world even when the underlying rules do not require it. Dr Andrea Rocco, Associate Professor in Physics and Mathematical Biology at the University of Surrey, described this contrast: "One way to explain this is when you look at a process like spilt milk spreading across a table, it's clear that time is moving forward. But if you were to play that in reverse, like a movie, you'd immediately know something was wrong – it would be hard to believe milk could just gather back into a glass. However, there are processes, such as the motion of a pendulum, that look just as believable in reverse. The puzzle is that, at the most fundamental level, the laws of physics resemble the pendulum; they do not account for irreversible processes. Our findings suggest that while our common experience tells us that time only moves one way, we are just unaware that the opposite direction would have been equally possible." The study focused on open quantum systems, which are quantum systems that interact with a surrounding environment. This environment, often described as a heat bath, can exchange energy and information with the system. The researchers used this framework to study how a direction of time might appear even when the underlying physics does not enforce one. A key part of the analysis involved the Markov approximation. This is a simplification used in many models where the system is assumed not to retain memory of its past states. The idea is that changes depend only on the current state, not on earlier history. This is commonly used when studying thermalisation, which is the process where a system settles into equilibrium with its environment. The study also used concepts such as master equations, including the Lindblad and Pauli equations, which describe how probabilities of different quantum states change over time. Another related model discussed was quantum Brownian motion, which describes the random-like movement of a quantum particle interacting continuously with its environment. In these descriptions, a “memory kernel” can appear, which is a mathematical term that accounts for how past states influence current behaviour. The researchers found that applying the Markov approximation did not break time-reversal symmetry. Even when the system interacted with an effectively infinite heat bath, the resulting equations of motion remained symmetric in time. This meant that the same mathematical description could, in principle, run forward or backward in time without contradiction. The study further showed that standard frameworks used in open quantum systems, including quantum Brownian motion and master equations like the Lindblad and Pauli forms, could be written in a time-symmetric way. These equations are typically used to describe processes that look irreversible, such as dissipation and thermalisation, but the results suggested they can also be interpreted as allowing evolution in both time directions. Thomas Guff, Research Fellow in Quantum Thermodynamics, said: "The surprising part of this project was that even after making the standard simplifying assumption to our equations describing open quantum systems, the equations still behaved the same way whether the system was moving forwards or backwards in time. When we carefully worked through the maths, we found that this behaviour had to be the case because a key part of the equation, the "memory kernel," is symmetrical in time. We also found a small but important detail which is usually overlooked – a time discontinuous factor emerged that kept the time-symmetry property intact. It’s unusual to see such a mathematical mechanism in a physics equation because it's not continuous, and it was very surprising to see it appear so naturally." The researchers also noted that deriving a one-way arrow of time from time-reversal symmetric microscopic dynamics remains an open problem across fields such as thermodynamics, statistical mechanics, particle physics, and cosmology. Their results suggested that some standard descriptions of irreversible behaviour in open quantum systems may be better understood using a time-symmetric formulation of Markovianity. According to the study, processes such as thermalisation, which are usually treated as irreversible, could in theory be described in a way that allows evolution in either time direction under the same rules. This does not imply that time reversal occurs in everyday life, but rather that the underlying equations do not strictly enforce a single direction. Overall, the findings suggested that the perceived direction of time may emerge from how physical systems are modelled and approximated, rather than from a fundamental asymmetry in the laws themselves. The researchers noted that this perspective could have implications for ongoing work in quantum mechanics, thermodynamics, and cosmology on the origin of time’s arrow. Source: University of Surrey, Nature This article was generated with some help from AI and reviewed by an editor. Under Section 107 of the Copyright Act 1976, this material is used for the purpose of news reporting. Fair use is a use permitted by copyright statute that might otherwise be infringing
    • A bit premature... 100% Marketing. Bizarre.
    • A $300 price hike is insane! No one is going to want to pay that much!
    • Since the 1st one flopped, there is really no reason to make another one. It's just losing money left and right.
  • Recent Achievements

    • Reacting Well
      BizSAR earned a badge
      Reacting Well
    • First Post
      AndreaB earned a badge
      First Post
    • Week One Done
      Huge Trailer earned a badge
      Week One Done
    • Week One Done
      Classifyskilleducation earned a badge
      Week One Done
    • One Month Later
      eurospharma62 earned a badge
      One Month Later
  • Popular Contributors

    1. 1
      +primortal
      581
    2. 2
      +Edouard
      182
    3. 3
      PsYcHoKiLLa
      75
    4. 4
      Michael Scrip
      73
    5. 5
      neufuse
      64
  • Tell a friend

    Love Neowin? Tell a friend!