Recommended Posts

Going through the logs, found this:

2012-05-15T13:45:52.954Z: <<throttled>> Storage I/O Control: connection with vobd failed, error code: -1 errno: 2[/CODE]

Not sure what it is, but it's mentioned CONSTANTLY!

Additionally:

[CODE]
2012-05-16T07:37:08.952Z: [scsiCorrelator] 64332112349us: [vob.scsi.device.io.latency.high] Device t10.ATA_____SAMSUNG_HD502IJ_________________________S13TJDWQ620722______ performance has deteriorated. I/O latency increased from average value of 22658 microseconds to 684718 microseconds.

2012-05-16T07:37:08.952Z: [scsiCorrelator] 64336113058us: [esx.problem.scsi.device.io.latency.high] Device t10.ATA_____SAMSUNG_HD502IJ_________________________S13TJDWQ620722______ performance has deteriorated. I/O latency increased from average value of 22658 microseconds to 684718 microseconds.

2012-05-16T08:03:08.708Z: [scsiCorrelator] 65891868893us: [vob.scsi.device.io.latency.improved] Device t10.ATA_____SAMSUNG_HD502IJ_________________________S13TJDWQ620722______ performance has improved. I/O latency reduced from 684718 microseconds to 174335 microseconds.
[/CODE]

More:

My Windows 7 VM is basically completely unresponsive from an I/O standpoint. System is very very slow while copying a VMDK between Datastores. Yet the HDD Indicator light on the Server itself is only flashing once in a while, indicating that minimal I/O is happening on the disk itself.

Sorry mate i am still a bit of a noobie with this stuff but what are you running to see this? I can check and see if my system is doing the same thing? On the ESXi installation i tried that and imported the VM's and saw the same issue.

Right so the issue is happening on an ESXi Level, not a VM Level. So further diagnostics on the VMs are pointless.

I've setup a NFS Share on my File Server and attempted to copy over an ISO from one of my DataStores, the speed was averaging about 5% of the 1Gbps connection, 20 minutes for a 3GB file. Everything else can max out this Servers connection easily. I'll attempt to boot a VM over the network later and measure performance again, this will 100% prove is this is a networking issue or an I/O issue as I'll be able to monitor I/O directly on the 2008R2 server.

I'm going to try moving the 'Management Network' to a different LAN and see if that provides me with better performance. Then this weekend I'll install ESXi again on the 250GB Disk that's included with the server and test the same.

Sorry mate i am still a bit of a noobie with this stuff but what are you running to see this? I can check and see if my system is doing the same thing? On the ESXi installation i tried that and imported the VM's and saw the same issue.

SSHing into the server, and running cat on the files in /var/log

Have you considered that maybe there is just an incompatability with your hard drive and ESXi? Have you tried some other type of drive to rule that out? Obviously it is giving you some kind of I/O latency error. What drive is Budman using in his as he seems to have no problems and everything else is identical?

This is such a weird problem. I don't understand how yours can be not working with identical hardware to budmans. I think honestly that your drives are faulty in some way but I have no way to back that up it's just a gut feeling. I would seriously try some new drives with a fresh installation of ESXi and two copies of Windows. It's time to start ruling stuff out as tweaking settings isn't fixing it.

Late entrant to the thread but have been following it on and off.

This may be completely useless, but your mentioning NFS on this page got me thinking - What if you get rid of NFS and use something else? IIRC, the NFS client is v3 for VSphere 5 which means all the traffic for the ESX host uses just 1 single TCP connection.

Tried another disk, and deffo a problem with ESX, Same Disk, same hardware with native windows 7 i get about 80 MBPS, with ESXi i get about 10MBPS :-(

ESXi 5, uploading to a datastore via WAN 100Mbps link from home at around 5-6MB/s. Not seeing the issues reported here, DL380G6 servers mostly.

Its a given that moving files to the datastore is going to be slow -- this is all over the forums on vmware, etc.

But thats not really the issue we are troubleshooting, the issue comes down to this.

between 2 vms, on the same host, same vswitch using the vmxnet3 drivers -- he is seeing LOW iperf speeds in the range of 300mbps, on the other hand on the exact same hardware N40L

Same test, 2 windows vms on the same host, same vswitch using vmxnet3 drivers I see like 1.+ Gbps

this should have nothing to do with disks, etc.

Boom - All non-essential VMs moved to NFS and essential VMs moved to the 500GB Samsung Drive.

I'll zero-fill the SSD and 250GB HDD to ensure disk health and reinstall ESXi onto the 250GB Drive this weekend.

If that fails, I'll get a saucepan, put it on my head, and whack it with a wooden spoon until I no longer care about the performance issues.

Guys!!! It's a CPU issue!

Windows 7 Virtual Machine - iperf -s -w 256k

Galactica Physical Machine - iperf -c 10.0.1.11 -w 256k

Speed = Approx 400Mbits/sec

CPU Use = 100% on Windows 7 VM during Transfer.

pfSesne VM - same situation!!!!

So - why is Budman's performance better in this instance? What would cause a VM to use 100% CPU time for a basic network transfer?

Do you have your VM's limited to what they can use?

This could be the reason your seeing the limit.

Let me run another iperf and check what my vms cpu does and what it maxes out at, etc.

Ok -- here is client, server and host while doing iperf and getting 1.3+Gbps

client

post-14624-0-66086900-1337275902.jpg

server

post-14624-0-49541500-1337275757.jpg

Host

post-14624-0-68526400-1337275770.jpg

And remember we went over this before, my vms only have 1 cpu assigned to them. and here you can see not restricting their usage on the host

post-14624-0-45999700-1337276022.jpg

What does your cpu look like -- are you actually hitting 100% and just staying there.. That would explain why your seeing a limit on the speed.

Hey,

My VMs only have one core accessible to them. Done a complete reinstall today onto the Physical Hard Drive. So my setup is essentially identical to yours now! Setup the 8GB SSD as a Cache drive.

Can you SSH into your pfSense VM and run top and iperf server on it and let me know what CPU% you get during the transfer. If you get similar issues over pfSense to what I get on Winodws that further helps diagnose this.

Additionally how exactly do you have CPU setup in VM settings?

Strange - the Domain Controller VM doesn't get maxed out like the Windows 7 VM does when performing Network Tasks. Did you make any registry tweaks to Networking? I believe you mentioned something about disabling the IPv6 and Tunneling adapters at one point? the Server 2008 R2 is able to get approx 690Mbits/sec compared with 400ish from the Windows 7 VM.

Edit: Server 2003 R2 was able to achieve 772Mbits/sec

here is my win7 client to pfsense running iperf server

post-14624-0-42024700-1337278877.jpg

But I have never seen great performance to and from windows with pfsense, but as you see my windows client is only see about 50% cpu and 500Mbps -- which is much better than yours????

My pfsense is using the e1000 drivers because I was having an issue with access to outside vpn from my laptop when pfsense used the vmxnet3 drivers.

Have not had time to dig into why.. But I think it might be related to the way pfsense labels the interfaces while using this drivers vmx3f0 and vmx3f1 -- which looks to cause a problem with the vlan stuff, and might have be related??? All I know for sure is an ipsec\L2TP vpn to an outside server from a client on my network doesn't work with the vmxnet3 drivers.

Maybe I have tweaked my windows install different than you? I was playing with adapter settings the other day and netsh global options, RSS, Chimney offload, etc I was able to get 1.7Gbps the other day while playing. Now only seeing like 1.3 -- I might of reset something or made another tweak without check performance?

Your limits seem real close to that vmware thread I saw about the guy seeing a 300Mbps limit on his vswitches.

Maybe we should use a specific linux live cd or something as our benchmark for vswitch performance.. Could just fire up 2 exact vms that way and compare performance.

I believe I am running a different version of pfsense than you for one thing. I am running 2.1 32bit, with latest gitsync - its running 8.3 freebsd, etc.

networking settings are different in 2k8 vs w7 for sure -- even just out of the gate.

But yeah I have disabled all the ipv6 stuff on my windows vms. But I do run it on my workstation, and my linux vm and pfsense. But sure its been correctly setup ;) ie the teredo and 6to4 **** disabled because not using that!

as I mentioned in previous post - maybe picking a linux live distro to boot would be a better apples to apples test.

YEAAAAAAHHHHH!!!

post-180771-0-31327900-1337279754.png

Enabled VMCI and Communication between VMs just went through the roof!

That combined with the Server 2003 R2 general networking performance means this is somehow now only effecting the Windows 7 VM. And that I can more than live with.

So, for the VMs dealing with downloading I'm going to allocate 2 CPUs so one can handle the Downloading and Networking while the other can handle FileSystem and File Operations (PAR+RAR).

Now I've just gotta figure out how to get Server 2003 R2 to actually see the two CPUs without doing a full reinstall.

as I mentioned in previous post - maybe picking a linux live distro to boot would be a better apples to apples test.

Sure - Arch might be the best bet because we can turn basically everything off on a tiny install. Want me to build a VMDK and send you the package?

nice

C:\Windows\System32>iperf -c storage.local.lan -w 256k

------------------------------------------------------------

Client connecting to storage.local.lan, TCP port 5001

TCP window size: 256 KByte

------------------------------------------------------------

[268] local 192.168.1.216 port 49182 connected with 192.168.1.8 port 5001

[ ID] Interval Transfer Bandwidth

[268] 0.0-10.0 sec 1.95 GBytes 1.67 Gbits/sec

Now I need to put my tweaks back - I had backed them all out, atleast the ones I could remember and was seeing like 1.15Gbps -- now its freaking rockin!! Lets see what happens when I put some of the tweaks back.. Lets see if can get that 1.8 or 2 mark ;)

ok check this out -- still playing with settings

C:\Windows\System32>iperf -c storage.local.lan -w 256k

------------------------------------------------------------

Client connecting to storage.local.lan, TCP port 5001

TCP window size: 256 KByte

------------------------------------------------------------

[268] local 192.168.1.216 port 49307 connected with 192.168.1.8 port 5001

[ ID] Interval Transfer Bandwidth

[268] 0.0-10.0 sec 2.17 GBytes 1.86 Gbits/sec

But seems disable interrupt moderation on the virtual nic sure doesn't hurt ;)

This topic is now closed to further replies.
  • Posts

    • Google is opening the world's first AI museum in Los Angeles by Ivan Jenic Image via: Google Ever since AI image generators went mainstream, the debate over whether AI-generated art is real art hasn't let up. Those who don’t consider AI to be art say that if a machine does the creating and anyone can prompt it, there’s no skill involved, and therefore no art is produced. The counter-argument is equally persistent, as defenders of AI-generated artworks often claim that AI is just a tool, and that every major technological breakthrough, like the camera or the computer, was met with the same skepticism before eventually being accepted as a legitimate creative medium. Google’s position in this debate is clear. Which is no surprise, as the company is investing billions in AI infrastructure. And now, in efforts to encourage people to use its AI even more, Google is opening Dataland on June 20, which it's calling the world's first AI arts museum. Located inside The Grand LA, a Frank Gehry-designed building in Los Angeles, the museum spans 25,000 square feet. The museum is built around a collaboration with media artist Refik Anadol, who has worked with Google since 2016. The inaugural exhibition is called Machine Dreams: Rainforest, and is powered by an AI model trained on “an extensive dataset of the natural world.” It generates 1.2 billion pixels of visuals in real time and reacts to visitors dynamically. The space also generates soundscapes, real-time emotion sensing, and algorithmically produced scents. Image via: Refik Anadol Studio / Google Google says that the museum is powered by its Gemini models, which run on Google Cloud. So, everything is generated inside one of Google’s AI data centers and is streamed to the museum. Alongside the museum opening, Google Arts & Culture is funding an AI Artist Residency, giving four artists $25,000 grants each, along with mentorship from Refik Anadol Studio and access to Google's machine learning tools. Their work will be shown at Dataland and on the Google Arts & Culture website later this year. Google’s AI museum will undoubtedly initiate a fired-up debate on social media, and we can’t wait to see the first reactions. Via: Smithsonian Magazine
    • Calling GTA 6 overhyped crap doesn’t make you edgy, it just makes you sound like someone who hasn’t enjoyed anything since the PS2 era.
    • I’m not arguing whether Rockstar likes money. Obviously, they do, they’re a business. I’m saying this isn’t new. They’ve always launched console first. This is just how Rockstar operates.
    • I'm not sure how old the school is, but they've been doing this since GTA 3. Back in those days we'd be lucky for game companies to release on the PC at all. And with the current state of Sony (or Microsoft) their gaming wing won't be getting a penny from me.
    • We now know when and how the Universe may truly end by Sayan Sen Image by Marek Pavlík via Pexels| Not representative A study by physicist Henry Tye of Cornell University suggests that the universe may not expand forever. Instead, it could eventually stop expanding, begin contracting and end in a "Big Crunch" roughly 20 billion years from now. The research, published in the Journal of Cosmology and Astroparticle Physics, was conducted by Tye, Horace White Professor of Physics Emeritus at Cornell University. Using recent observations from major dark-energy surveys, Tye and his collaborators developed a cosmological model that predicts the universe could have a total lifespan of about 33 billion years. Since the universe is currently estimated to be 13.8 billion years old, the model places it near the midpoint of its existence. According to Cornell University's summary of the research, the study centers on the cosmological constant, a term introduced by Albert Einstein in his theory of general relativity. In modern cosmology, the cosmological constant is commonly used to describe the simplest form of dark energy, the unknown phenomenon believed to be driving the accelerating expansion of the universe. "For the last 20 years, people believed that the cosmological constant is positive, and the universe will expand forever," Tye said in a Cornell University news release. "The new data seem to indicate that the cosmological constant is negative, and that the universe will end in a big crunch." The study draws on data from the Dark Energy Survey (DES) and the Dark Energy Spectroscopic Instrument (DESI), two major projects designed to investigate the nature of dark energy. According to Tye, recent observations suggest that dark energy may not behave exactly like a simple cosmological constant. To account for those observations, Tye and his collaborators proposed a model involving an extremely light hypothetical particle that evolves over time. In their calculations, this produces a negative cosmological constant and leads to a future collapse of the universe. The model predicts that cosmic expansion would continue for approximately another 11 billion years before reaching a maximum size, after which the universe would begin contracting and eventually collapse. Scientists have long debated how the universe might end. As explained in an article published in The Conversation by Stephen DiKerby of Michigan State University, several possibilities have been proposed. If dark energy remains constant and positive, the universe could continue expanding indefinitely, gradually becoming colder, darker and more diffuse in a scenario often called the "heat death" of the universe. Other theoretical possibilities include a Big Rip, in which cosmic expansion accelerates so dramatically that galaxies, stars and even atoms are torn apart, or a Big Crunch, in which expansion reverses and the universe collapses back into an extremely dense state. DiKerby notes that the Big Crunch idea itself is not new. What distinguishes Tye's work is that it attempts to use current observational data to estimate when such a collapse might occur and how it could unfold. Much of the universe's long-term evolution remains uncertain. According to current astrophysical understanding, stars will continue to form and die for billions of years. The Sun, for example, is about halfway through its expected lifespan. Galaxies are also expected to continue merging; the Milky Way and Andromeda galaxies are projected to collide several billion years from now. At the same time, the nature of dark energy remains one of the biggest unanswered questions in cosmology. While observations indicate that the universe's expansion is accelerating, scientists still do not know what is causing that acceleration. Future observations may therefore alter current predictions about the cosmos's ultimate fate. Tye emphasized that additional evidence will be needed before firm conclusions can be drawn. DESI continues to collect data, while upcoming observations from missions and observatories including Euclid, SPHEREx and the Vera C. Rubin Observatory are expected to provide more precise measurements of dark energy. "People have said before that if the cosmological constant is negative, then the universe will collapse eventually. That's not new," Tye said. "However, here the model tells you when the universe collapses and how it collapses." For now, the study presents one possible future for the cosmos rather than a settled prediction. Whether the universe ultimately ends in a Big Crunch, expands forever, or follows another path entirely remains an open question that future observations will help answer. Source: Cornell University, The Conversation This article was generated with some help from AI and reviewed by an editor. Under Section 107 of the Copyright Act 1976, this material is used for the purpose of news reporting. Fair use is a use permitted by copyright statute that might otherwise be infringing.
  • Recent Achievements

    • First Post
      AndreaB earned a badge
      First Post
    • Week One Done
      Huge Trailer earned a badge
      Week One Done
    • Week One Done
      Classifyskilleducation earned a badge
      Week One Done
    • One Month Later
      eurospharma62 earned a badge
      One Month Later
    • Week One Done
      With What earned a badge
      Week One Done
  • Popular Contributors

    1. 1
      +primortal
      570
    2. 2
      +Edouard
      175
    3. 3
      PsYcHoKiLLa
      73
    4. 4
      Michael Scrip
      68
    5. 5
      neufuse
      64
  • Tell a friend

    Love Neowin? Tell a friend!