Recommended Posts

Going through the logs, found this:

2012-05-15T13:45:52.954Z: <<throttled>> Storage I/O Control: connection with vobd failed, error code: -1 errno: 2[/CODE]

Not sure what it is, but it's mentioned CONSTANTLY!

Additionally:

[CODE]
2012-05-16T07:37:08.952Z: [scsiCorrelator] 64332112349us: [vob.scsi.device.io.latency.high] Device t10.ATA_____SAMSUNG_HD502IJ_________________________S13TJDWQ620722______ performance has deteriorated. I/O latency increased from average value of 22658 microseconds to 684718 microseconds.

2012-05-16T07:37:08.952Z: [scsiCorrelator] 64336113058us: [esx.problem.scsi.device.io.latency.high] Device t10.ATA_____SAMSUNG_HD502IJ_________________________S13TJDWQ620722______ performance has deteriorated. I/O latency increased from average value of 22658 microseconds to 684718 microseconds.

2012-05-16T08:03:08.708Z: [scsiCorrelator] 65891868893us: [vob.scsi.device.io.latency.improved] Device t10.ATA_____SAMSUNG_HD502IJ_________________________S13TJDWQ620722______ performance has improved. I/O latency reduced from 684718 microseconds to 174335 microseconds.
[/CODE]

More:

My Windows 7 VM is basically completely unresponsive from an I/O standpoint. System is very very slow while copying a VMDK between Datastores. Yet the HDD Indicator light on the Server itself is only flashing once in a while, indicating that minimal I/O is happening on the disk itself.

Sorry mate i am still a bit of a noobie with this stuff but what are you running to see this? I can check and see if my system is doing the same thing? On the ESXi installation i tried that and imported the VM's and saw the same issue.

Right so the issue is happening on an ESXi Level, not a VM Level. So further diagnostics on the VMs are pointless.

I've setup a NFS Share on my File Server and attempted to copy over an ISO from one of my DataStores, the speed was averaging about 5% of the 1Gbps connection, 20 minutes for a 3GB file. Everything else can max out this Servers connection easily. I'll attempt to boot a VM over the network later and measure performance again, this will 100% prove is this is a networking issue or an I/O issue as I'll be able to monitor I/O directly on the 2008R2 server.

I'm going to try moving the 'Management Network' to a different LAN and see if that provides me with better performance. Then this weekend I'll install ESXi again on the 250GB Disk that's included with the server and test the same.

Sorry mate i am still a bit of a noobie with this stuff but what are you running to see this? I can check and see if my system is doing the same thing? On the ESXi installation i tried that and imported the VM's and saw the same issue.

SSHing into the server, and running cat on the files in /var/log

Have you considered that maybe there is just an incompatability with your hard drive and ESXi? Have you tried some other type of drive to rule that out? Obviously it is giving you some kind of I/O latency error. What drive is Budman using in his as he seems to have no problems and everything else is identical?

This is such a weird problem. I don't understand how yours can be not working with identical hardware to budmans. I think honestly that your drives are faulty in some way but I have no way to back that up it's just a gut feeling. I would seriously try some new drives with a fresh installation of ESXi and two copies of Windows. It's time to start ruling stuff out as tweaking settings isn't fixing it.

Late entrant to the thread but have been following it on and off.

This may be completely useless, but your mentioning NFS on this page got me thinking - What if you get rid of NFS and use something else? IIRC, the NFS client is v3 for VSphere 5 which means all the traffic for the ESX host uses just 1 single TCP connection.

Tried another disk, and deffo a problem with ESX, Same Disk, same hardware with native windows 7 i get about 80 MBPS, with ESXi i get about 10MBPS :-(

ESXi 5, uploading to a datastore via WAN 100Mbps link from home at around 5-6MB/s. Not seeing the issues reported here, DL380G6 servers mostly.

Its a given that moving files to the datastore is going to be slow -- this is all over the forums on vmware, etc.

But thats not really the issue we are troubleshooting, the issue comes down to this.

between 2 vms, on the same host, same vswitch using the vmxnet3 drivers -- he is seeing LOW iperf speeds in the range of 300mbps, on the other hand on the exact same hardware N40L

Same test, 2 windows vms on the same host, same vswitch using vmxnet3 drivers I see like 1.+ Gbps

this should have nothing to do with disks, etc.

Boom - All non-essential VMs moved to NFS and essential VMs moved to the 500GB Samsung Drive.

I'll zero-fill the SSD and 250GB HDD to ensure disk health and reinstall ESXi onto the 250GB Drive this weekend.

If that fails, I'll get a saucepan, put it on my head, and whack it with a wooden spoon until I no longer care about the performance issues.

Guys!!! It's a CPU issue!

Windows 7 Virtual Machine - iperf -s -w 256k

Galactica Physical Machine - iperf -c 10.0.1.11 -w 256k

Speed = Approx 400Mbits/sec

CPU Use = 100% on Windows 7 VM during Transfer.

pfSesne VM - same situation!!!!

So - why is Budman's performance better in this instance? What would cause a VM to use 100% CPU time for a basic network transfer?

Do you have your VM's limited to what they can use?

This could be the reason your seeing the limit.

Let me run another iperf and check what my vms cpu does and what it maxes out at, etc.

Ok -- here is client, server and host while doing iperf and getting 1.3+Gbps

client

post-14624-0-66086900-1337275902.jpg

server

post-14624-0-49541500-1337275757.jpg

Host

post-14624-0-68526400-1337275770.jpg

And remember we went over this before, my vms only have 1 cpu assigned to them. and here you can see not restricting their usage on the host

post-14624-0-45999700-1337276022.jpg

What does your cpu look like -- are you actually hitting 100% and just staying there.. That would explain why your seeing a limit on the speed.

Hey,

My VMs only have one core accessible to them. Done a complete reinstall today onto the Physical Hard Drive. So my setup is essentially identical to yours now! Setup the 8GB SSD as a Cache drive.

Can you SSH into your pfSense VM and run top and iperf server on it and let me know what CPU% you get during the transfer. If you get similar issues over pfSense to what I get on Winodws that further helps diagnose this.

Additionally how exactly do you have CPU setup in VM settings?

Strange - the Domain Controller VM doesn't get maxed out like the Windows 7 VM does when performing Network Tasks. Did you make any registry tweaks to Networking? I believe you mentioned something about disabling the IPv6 and Tunneling adapters at one point? the Server 2008 R2 is able to get approx 690Mbits/sec compared with 400ish from the Windows 7 VM.

Edit: Server 2003 R2 was able to achieve 772Mbits/sec

here is my win7 client to pfsense running iperf server

post-14624-0-42024700-1337278877.jpg

But I have never seen great performance to and from windows with pfsense, but as you see my windows client is only see about 50% cpu and 500Mbps -- which is much better than yours????

My pfsense is using the e1000 drivers because I was having an issue with access to outside vpn from my laptop when pfsense used the vmxnet3 drivers.

Have not had time to dig into why.. But I think it might be related to the way pfsense labels the interfaces while using this drivers vmx3f0 and vmx3f1 -- which looks to cause a problem with the vlan stuff, and might have be related??? All I know for sure is an ipsec\L2TP vpn to an outside server from a client on my network doesn't work with the vmxnet3 drivers.

Maybe I have tweaked my windows install different than you? I was playing with adapter settings the other day and netsh global options, RSS, Chimney offload, etc I was able to get 1.7Gbps the other day while playing. Now only seeing like 1.3 -- I might of reset something or made another tweak without check performance?

Your limits seem real close to that vmware thread I saw about the guy seeing a 300Mbps limit on his vswitches.

Maybe we should use a specific linux live cd or something as our benchmark for vswitch performance.. Could just fire up 2 exact vms that way and compare performance.

I believe I am running a different version of pfsense than you for one thing. I am running 2.1 32bit, with latest gitsync - its running 8.3 freebsd, etc.

networking settings are different in 2k8 vs w7 for sure -- even just out of the gate.

But yeah I have disabled all the ipv6 stuff on my windows vms. But I do run it on my workstation, and my linux vm and pfsense. But sure its been correctly setup ;) ie the teredo and 6to4 **** disabled because not using that!

as I mentioned in previous post - maybe picking a linux live distro to boot would be a better apples to apples test.

YEAAAAAAHHHHH!!!

post-180771-0-31327900-1337279754.png

Enabled VMCI and Communication between VMs just went through the roof!

That combined with the Server 2003 R2 general networking performance means this is somehow now only effecting the Windows 7 VM. And that I can more than live with.

So, for the VMs dealing with downloading I'm going to allocate 2 CPUs so one can handle the Downloading and Networking while the other can handle FileSystem and File Operations (PAR+RAR).

Now I've just gotta figure out how to get Server 2003 R2 to actually see the two CPUs without doing a full reinstall.

as I mentioned in previous post - maybe picking a linux live distro to boot would be a better apples to apples test.

Sure - Arch might be the best bet because we can turn basically everything off on a tiny install. Want me to build a VMDK and send you the package?

nice

C:\Windows\System32>iperf -c storage.local.lan -w 256k

------------------------------------------------------------

Client connecting to storage.local.lan, TCP port 5001

TCP window size: 256 KByte

------------------------------------------------------------

[268] local 192.168.1.216 port 49182 connected with 192.168.1.8 port 5001

[ ID] Interval Transfer Bandwidth

[268] 0.0-10.0 sec 1.95 GBytes 1.67 Gbits/sec

Now I need to put my tweaks back - I had backed them all out, atleast the ones I could remember and was seeing like 1.15Gbps -- now its freaking rockin!! Lets see what happens when I put some of the tweaks back.. Lets see if can get that 1.8 or 2 mark ;)

ok check this out -- still playing with settings

C:\Windows\System32>iperf -c storage.local.lan -w 256k

------------------------------------------------------------

Client connecting to storage.local.lan, TCP port 5001

TCP window size: 256 KByte

------------------------------------------------------------

[268] local 192.168.1.216 port 49307 connected with 192.168.1.8 port 5001

[ ID] Interval Transfer Bandwidth

[268] 0.0-10.0 sec 2.17 GBytes 1.86 Gbits/sec

But seems disable interrupt moderation on the virtual nic sure doesn't hurt ;)

This topic is now closed to further replies.
  • Posts

    • Go for a Echo Dot or Pop instead. These Echo shows just advertise to you.
    • NetSpeedTray 1.3.3 by Razvan Serea NetSpeedTray is a lightweight, open-source Windows network monitor that shows live upload and download speeds directly on the Taskbar. Designed for efficiency, it quietly sits in the system tray, conserving CPU and battery with dynamic updates. It blends seamlessly with Windows 10/11, adapts to light/dark themes, and auto-positions to avoid overlaps. Features include accurate interface detection, customizable display, optional mini-graph, color coding, granular font and unit control, detailed per-interface history graphs, safe data management, and easy CSV export—bringing the network monitoring Windows forgot. NetSpeedTray key features: Lightweight & Efficient Runs quietly in your system tray without consuming resources. Features a "Dynamic Update Rate" that lowers refresh frequency when the network is idle to save CPU and battery life. Native Look & Feel Blends seamlessly with Windows 10/11 UI. Smart detection for light and dark taskbar themes ensures text is always visible. Intelligent & Adaptive Positioning Automatically finds empty space next to your system tray and shifts to make room for new icons, preventing overlaps. Seamless OS Integration Behaves like a native Windows component. Hides instantly with auto-hiding taskbar Hides when a fullscreen app is active Smart Network Monitoring Accurate by Default: Auto mode identifies your main internet connection and ignores noise from VPNs or virtual adapters. Easy Interface Selection: Switch effortlessly between Auto, All, or Selected network interfaces via intuitive radio buttons. Total Visual Customization Free Move Mode: Unlock and place the widget anywhere on your screen. Optional Mini-Graph: Real-time graph of recent network activity with adjustable opacity. Color Coding: Customize colors and speed thresholds to quickly see network status. Granular Display Control Text & Font: Adjust font family, size, weight, and alignment. Units: Automatic (B/s, KB/s, MB/s) or fixed Mbps display. Precision: Set decimal places and always show them for uniform appearance. Detailed & Intelligent History Graph Smart Scale: Logarithmic scale shows low-level traffic and large spikes clearly. Per-Interface Filtering: View speed history for specific adapters (Wi-Fi, Ethernet, VPN). Safe & Efficient Data Management: Adjustable retention, automatic cleanup, optimized database. Easy Data Export: Export raw data to .csv or save high-quality graphs for reports. NetSpeedTray v1.3.3: The Updater Fix A stabilization release that repairs a critical regression in v1.3.2: the app shipped without OpenSSL, which silently broke every HTTPS request — including the built-in update checker (the "Could not check for updates" error many of you hit). This release restores it, hardens the build so it can't happen again, and fixes a startup crash plus four other reported bugs. Changes: Fixed update checking — Resolved a critical issue that prevented the app from checking for updates ("Could not check for updates"). Fixed startup crash with Auto-Cycling — The app no longer crashes on launch after enabling Cycle display mode. Fixed incorrect network speeds on 10GbE adapters — Multi-gigabit network cards now display speeds correctly instead of being stuck at 0. Improved color coding — Default color is shown when idle, and color/threshold changes now apply immediately without restarting. Fullscreen visibility fix — The widget now correctly stays visible over fullscreen apps when Keep Visible is enabled. Improved AMD Ryzen temperature detection — More reliable CPU temperature monitoring for Ryzen processors. Cleaner upgrades — Installer now removes outdated application files during upgrades, preventing DLL/version conflicts while preserving user settings. Improved stability — Fixed potential DLL loading issues by excluding critical OpenSSL and NumPy components from UPX compression. Better settings window — Scrollbars removed and layout improved for a cleaner experience. Localization improvements — Updated translations and completed missing UI text across all supported languages. More reliable releases — Added regression tests covering recent critical fixes, bringing the test suite to 196 passing tests. [full release notes] Download: NetSpeedTray 1.3.3 | 87.9 MB (Open Source) Download: NetSpeedTray Portable | 101.0 MB View: NetSpeedTray Home Page | Screenshot Get alerted to all of our Software updates on Twitter at @NeowinSoftware
    • Why Delta Chat is the best decentralized messenger you have probably never tried by Paul Hill There is no shortage of messaging apps out there; we have WhatsApp, Messenger, and Telegram, just to name a few. While Meta has taken steps to incorporate encryption into Messenger and WhatsApp, they still leave a lot to be desired. If you are in the market for a messaging app that promotes security, privacy, and optional anonymity, you'll want to read what I have to say about Delta Chat. For those not familiar with Delta Chat, rather than relying on centralized servers as you do with Facebook Messenger, it relies on email. Essentially, it is a chat interface that feels like a messaging app, but secretly in the background, it is firing off emails. In the past, you used to have to sign in with your email account. When you sent messages to people, it would just be sending encrypted messages to their inbox, which their Delta Chat client would decrypt. When I first learned about Delta Chat, it required users to sign in with an email account, but I was pleasantly surprised upon trying it in 2026 that this is no longer a requirement, or the preferred method was to use the app. Recently, I’ve tried UAD-ng on my old Nokia 3.4 to disable most of the Google apps because the bootloader is locked, and this is the next best option. While finding replacement apps in F-Droid, I came across Delta Chat again, and it has undergone quite a big change since I last used it, with its new chatmail relays, which no longer require you to sign in to your own email account, providing anonymity, and they offer greater security. Android and Desktop Delta Chat apps. Not only does it run on my de-googled phone, but it also works on desktop computers and iOS, making it truly ubiquitous. For me, Delta Chat is a wonderful alternative messenger because it gives you more control. It supports switching between different profiles, which you can set up super quickly; you don’t register a username, you don’t register a password. The only thing you do have is a random string email address on a chatmail relay (which you don’t have to memorize). To maintain access to your profile, you just need to add a second device to your account via QR code or make a backup of your account, which you can restore later. Fail to do these, your account is gone - as it should be if you don’t want to leave accounts that could get hacked later on. My decision to block Google stuff on my Nokia was done for practical reasons; the device sucked when it launched, and it sucks even more now. The nice thing about F-Droid and the apps within is that they’re usually lightweight, free of bloat, and work well on that device. What was inconvenient for me was that it was hard to send messages from that device, say if I wanted to copy a code over to my main phone or send family members a link from that device. That’s when I decided to look at the available chat apps and saw Delta Chat. Another nice thing about Delta Chat is its notifications. Some messaging apps rely on Google’s ecosystem for notification transport on Android; however, with Delta Chat, it can use Google’s solutions if you have Play Services or MicroG installed. Otherwise, it is able to keep a background connection to the chatmail relay server so that you can get notified when you receive a message. As free software, the code of Delta Chat is open for all who want to take it and build upon it. In the future, if the developers of Delta Chat make a catastrophically bad decision and take the app in an undesirable direction, users can take the code and fork the project. This contrasts with closed-source apps from corporations that can take their products in any direction they like. By relying on free software instead of closed-source programs, you actually control your computing. I’ve spoken at length about how running this type of software is like owning your own home rather than renting it. The same applies here; if you use Delta Chat, you don’t need to worry about it going away in the future. Whether it is Telegram, WhatsApp, or Messenger, you are required to register a username and password to use these services. A major flaw in this design is that anyone can try various passwords and potentially break into your account with your complete chat history intact. Sure, there is encryption in Messenger, where you need a second PIN and two-factor authentication in Telegram, but breaches happen all the time. Unlike before, when you used to sign in to your email account to send and receive messages, the primary way to do it now is to create an account on a chatmail relay. The resulting email address is a random string followed by the name of the relay you pick. This means you can start and begin adding contacts Without a username and password, you either need to ensure you have a backup or at least one device running your Delta Chat profile. The primary way to log in on another device is to go to the settings and add a second device. Then, you’ll just scan a QR code with your new device, and it’ll log in to your account and sync all your chat history and contacts. To end users, Delta Chat just looks like any instant messenger; however, it is really sending your messages as encrypted emails to your contact. This is pretty cool from a censorship perspective, as it makes the service more difficult to block. Previously, the main way to use the app was by logging in with email, but nowadays, it’s recommended that you use chatmail relays. Chatmail relays temporarily hold messages in case your device is offline. They are cheap, simple servers that don’t store data as group states. Other information, like your name and avatar, only exists on your device and the devices of those you share your contact information with. The relays are also decentralized and operated by various groups and individuals. It is even possible to set up your own chatmail relay, but most people will want to use one hosted elsewhere. To keep your messages secure, Delta Chat uses a secure subset of the OpenPGP standard that gives you automatic end-to-end encryption. It also uses Secure-Join to exchange encryption setup information through QR-code scanning or invite links. Autocrypt is also used to automatically establish end-to-end encryption between contacts and all members of group chat, but sometime this year Autocrypt v2 will be rolled out, bringing post-quantum resistant encryption and forward secrecy. The Delta Chat FAQ is an interesting read that explains many more details about the app. Credit: Pexels Delta Chat is unique among messaging apps because it is built on email, a technology that’s decades old and isn’t going anywhere soon. What’s more is that email is not centralized either, so it’s far more difficult for any authoritarian regime to disrupt the Delta Chat app. I haven’t spoken too much about features yet, so I will do that now. Delta Chat allows you to do one-on-one chats, group chats, and create channels. It also supports file sharing and making audio and video calls when chatting one-to-one, but it’s not available for group chats right now. At the time of writing, the calling functionality is disabled and can be enabled in Settings > Advanced > Debug Calls. I have used the video calling feature, and the quality is excellent. It works over WebRTC, another open standard. The app also lets you send voice notes, enables disappearing messages, and has its own app ecosystem. I did try playing chess one time there, but it was a bit spotty; though, we did manage to complete the game with a victory for me. To add people to Delta Chat, you can either give them your Delta Chat link or your QR code to scan. These are the only ways to add users, so you won't have any spam bots bothering you. If the people you want to chat with don't have the app yet, just send them your link, and it will take them to a webpage where they can install the app and then add you. It's really quick for them to install it and get started, which is nice. Credit: Microsoft. The Majorana 2 quantum chip unveiled in 2026. I do not think quantum computers are too far out now, and I do hope that Delta Chat is able to push out Autocrypt v2 sooner, rather than later, so bad actors do not attempt to collect encrypted communications and then decrypt them in the future using quantum computers. By getting people’s messages post-quantum-safe now, users won’t have to worry when quantum computers start cracking legacy encryption. Overall, I would recommend this app to people who are already past WhatsApp and Messenger and have perhaps begun using apps like Telegram or Session. It shares a lot of characteristics with these apps and goes a lot further than Telegram in terms of security. By being based on email, it is also resistant to censorship, and the lack of a username and password makes you anonymous (if you want to be) and safe from brute force password cracking attempts. Let me know in the comments if you’ve tried Delta Chat recently. Do you think it's a good bulwark against governments that are tightening their grip on the internet?
  • Recent Achievements

    • One Year In
      bernmeister earned a badge
      One Year In
    • Week One Done
      Scoobystu earned a badge
      Week One Done
    • Week One Done
      tuben earned a badge
      Week One Done
    • First Post
      OffsetAbs earned a badge
      First Post
    • Reacting Well
      OffsetAbs earned a badge
      Reacting Well
  • Popular Contributors

    1. 1
      +primortal
      474
    2. 2
      +Edouard
      220
    3. 3
      PsYcHoKiLLa
      156
    4. 4
      Steven P.
      73
    5. 5
      FloatingFatMan
      71
  • Tell a friend

    Love Neowin? Tell a friend!