• 0

Sending large message over TCP


Question

Hello, I have a little client-server application using tcp sockets. On both sides I use a buffer size of 4096 and I run all my tests with the client and server on the same machine, using address 127.0.0.1. Everything is going well except that now I suddenly need to send much larger messages. A message could now easily be 12MB in size. Now, what happens is that the transmission is incomplete : on the receiving end, I only get a chunk of the message instead of the whole thing.

I thought about simply increasing the buffer sizes on both ends to something larger than my maximum message size, but I'm not sure that's a reliable method, even if it works, and I'm not sure it'll work (if it's supported by TCP, if I won't run out of memory, etc.).

I also thought about splitting the message and actually my sending code automatically does that already, it will keep sending until all the data has been sent. However, on the receiving end, how do I know where a message ends, do I need to implement a protocol for that? I thought TCP already took care of that.

Anyway I'm a bit lost there, thanks for any tips.

Link to comment
https://www.neowin.net/forum/topic/809920-sending-large-message-over-tcp/
Share on other sites

9 answers to this question

Recommended Posts

  • 0

The main way to doing this (which you may have realized) is to send header data with each packet sent. About 5 years back when I got into remote desktop applications (using VB6 and winsock) I'd send something like 0|DATA. Each time something was sent, an integer was prefixed with the character | as the splitter. Alternatively, if you aren't sure what characters may be sent through, you can instead use char(0) (invisible char if displayed but detectable w/ programming). I'd have constants/enumerator defining exactly what each pre-fixed integer means (something like const FIRST_DOWNLOAD_PACKET = 0; const NEXT_DOWNLOAD_PACKET = 1; const LAST_DOWNLOAD_PACKET = 2;) and so forth. Whenever the client application would receive a packet, I'd split the packet into 2 - header info and data. Header information told me what to do with the information while the data contained the actual information.

You could create a class to handle something like this too (something like a packet class) for re-using in other apps using sockets.

  • 0
  dlegend said:
The main way to doing this (which you may have realized) is to send header data with each packet sent. About 5 years back when I got into remote desktop applications (using VB6 and winsock) I'd send something like 0|DATA. Each time something was sent, an integer was prefixed with the character | as the splitter. Alternatively, if you aren't sure what characters may be sent through, you can instead use char(0) (invisible char if displayed but detectable w/ programming). I'd have constants/enumerator defining exactly what each pre-fixed integer means (something like const FIRST_DOWNLOAD_PACKET = 0; const NEXT_DOWNLOAD_PACKET = 1; const LAST_DOWNLOAD_PACKET = 2;) and so forth. Whenever the client application would receive a packet, I'd split the packet into 2 - header info and data. Header information told me what to do with the information while the data contained the actual information.

You could create a class to handle something like this too (something like a packet class) for re-using in other apps using sockets.

Yes, that's pretty much what I did, although more simple. My problem was basically how to send variable length messages; the receiving end has no idea where a message begins and where it ends. Sending fixed-sized messages is fine though, I can just buffer the input and split every [LENGTH] bytes. (where LENGTH is, of course, a constant)

So how I solved it is that I each time I send a message, I prefix it with a fixed-size header containing the length of the message. Another way I could have done it is to begin and end each message with a special character, but I was worried that could too easily break.

And yup this is all encapsulated in a nice, single-purpose, reusable class. :p

  • 0

the tcp protocol will ensure the data arrives in the correct order if you just break it up into chunks and send them off one after the other

the most stable/reliable method i've come across is where you send a packet and don't send anymore until the client responds with some form of acknowledgement

server sends 4KB

client responses "ok"

server sends 4KB

client responses "ok"

~ repeat

problem with that is the speed is now completely dependant on latency and you can't take advantage of burst techniques however if you're sending to a low latency network you shouldn't have many issues

always good to have various approaches for various network types to get the most out of them :D

  • 0
  DDStriker said:
the tcp protocol will ensure the data arrives in the correct order if you just break it up into chunks and send them off one after the other

the most stable/reliable method i've come across is where you send a packet and don't send anymore until the client responds with some form of acknowledgement

server sends 4KB

client responses "ok"

server sends 4KB

client responses "ok"

~ repeat

TCP handles acknowledgment and throttling for you automatically. If you're wanting to send data to a remote endpoint as fast as possible, you might as well send it in chunks of 4096 bytes and let TCP handle those. If throttling occurs, eventually send() will not return for awhile on the local side until the congestion has been alleviated.

  • 0

Since you're already splitting your messages at this point, you may want to reduce their size. 4096 byte packets are definitely going to be fragmented (unless this is a specialized closed network of some sort, this would even fragment on the old token ring), and that could result in latency due to retransmission. 1456 would be a typical size that would not fragment on most contemporary ethernet networks (1500 MTU, 20 byte IP header, 24 byte TCP header).

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • Space Marine 2 gains a horde mode and DLSS 4 with new Siege update by Pulasthi Ariyasinghe Warhammer 40,000: Space Marine 2 just received its next major update. Dubbed the Siege update, it is finally bringing the long-promised horde mode for the third-person action game, as well as new tech for PC gamers. The Siege mode takes place on a fortified Imperial stronghold on Kadaku that is enduring endless attacks by Tyranids and Thousand Sons. There are three sections in total that players will defend in order before circling back to the original position, fighting away thousands of enemies in five waves before swapping locations. To give players a small break, there's a one-minute gap between each wave to resupply and purchase perks from the points they collect while in battle. Players can call in AI-controlled specialized soldiers, Space Marines, or even a Dreadnought to help out if the waves get a little too tough. To cap it all off, every fifth wave a boss encounter kicks off. There are 15 waves to go through in a single session, but parties can also kick off an endless mode at the end of these to see how far they can get. "Siege Mode expands on our PVE operations, emphasizing endless enemy waves that challenge even the strongest teams," says Focus Entertainment, describing the new mode. "When overwhelmed, players may call reinforcements for assistance. Engage in intense battles across three distinctive sectors, each requiring strategic adaptation. Side objectives emerge during combat, providing essential resources used to summon elite Space Marines or a formidable Dreadnought, drastically shifting battlefield dynamics. Over on PC, the studio has added two new options. One is RTAO (Ray-Traced Ambient Occlusion), though this is currently listed as an experimental feature. Next, Nvidia 5000 series graphics card owners can now enable DLSS 4 for better upscaling tech as well as multi-frame generation. The complete patch notes for Patch 8, which includes details on the balance changes, can be found here. The Warhammer 40,000: Space Marine 2 Siege update is now available on PC, Xbox Series X|S, and PlayStation 5.
    • Microsoft updates Media Creation Tool with a newer Windows 11 release by Taras Buria If you want to download a Windows 11 image or create an install media, the Media Creation Tool app is one of the best options—simple, fast, and, more importantly, official. While there are other useful alternatives, such as Rufus, Media Creation Tool remains the default choice for those who simply want a clean Windows 11 installation on a supported computer. Thanks to the latest MCT update, you will have to deal with fewer updates after installing Windows 11. Microsoft has quietly updated the Media Creation Tool app so that it downloads the latest Windows 11 release, version 24H2, with build number 26100.4349. This is the most recent public Windows 11 update (June 2025 Patch Tuesday), which eliminates the need to download large updates for your PC after installing the operating system. You can download Media Creation Tool from the official Windows 11 website. If you are familiar with it, this simple app can generate a Windows 11 image with the language you need or create a bootable USB media. Note that it only works on Windows, so Linux or macOS users have to download Windows images directly from the official website and then use apps that are available on their platforms. Another thing worth noting is Media Creation Tool is not the right choice if you want to install Windows 11 on an unsupported PC. In such a case, you can use MCT to download an ISO and then create a modified install media with apps like Rufus or Ventoy as described in our dedicated guide. If you want a more exotic Windows 11 Edition, try the Oofhours Media Tool, which allows you to select Windows 10 or 11, architecture (x64, x86, or ARM64), language, and edition. In addition to standard Home and Pro, the application can download non-mainstream SKUs, such as Enterprise, Education, or Professional for Workstations. Via: Deskmodder
    • Putting premium on your higher end laptops really makes it feel like your lower end laptops are worse.
    • Funny enough just removed insta account and whatsapp is next. Less power needed now
  • Recent Achievements

    • Conversation Starter
      Kavin25 earned a badge
      Conversation Starter
    • One Month Later
      Leonard grant earned a badge
      One Month Later
    • Week One Done
      pcdoctorsnet earned a badge
      Week One Done
    • Rising Star
      Phillip0web went up a rank
      Rising Star
    • One Month Later
      Epaminombas earned a badge
      One Month Later
  • Popular Contributors

    1. 1
      +primortal
      538
    2. 2
      ATLien_0
      207
    3. 3
      +FloatingFatMan
      174
    4. 4
      Michael Scrip
      148
    5. 5
      snowy owl
      119
  • Tell a friend

    Love Neowin? Tell a friend!