• 0

Sending large message over TCP


Question

Hello, I have a little client-server application using tcp sockets. On both sides I use a buffer size of 4096 and I run all my tests with the client and server on the same machine, using address 127.0.0.1. Everything is going well except that now I suddenly need to send much larger messages. A message could now easily be 12MB in size. Now, what happens is that the transmission is incomplete : on the receiving end, I only get a chunk of the message instead of the whole thing.

I thought about simply increasing the buffer sizes on both ends to something larger than my maximum message size, but I'm not sure that's a reliable method, even if it works, and I'm not sure it'll work (if it's supported by TCP, if I won't run out of memory, etc.).

I also thought about splitting the message and actually my sending code automatically does that already, it will keep sending until all the data has been sent. However, on the receiving end, how do I know where a message ends, do I need to implement a protocol for that? I thought TCP already took care of that.

Anyway I'm a bit lost there, thanks for any tips.

Link to comment
https://www.neowin.net/forum/topic/809920-sending-large-message-over-tcp/
Share on other sites

9 answers to this question

Recommended Posts

  • 0

The main way to doing this (which you may have realized) is to send header data with each packet sent. About 5 years back when I got into remote desktop applications (using VB6 and winsock) I'd send something like 0|DATA. Each time something was sent, an integer was prefixed with the character | as the splitter. Alternatively, if you aren't sure what characters may be sent through, you can instead use char(0) (invisible char if displayed but detectable w/ programming). I'd have constants/enumerator defining exactly what each pre-fixed integer means (something like const FIRST_DOWNLOAD_PACKET = 0; const NEXT_DOWNLOAD_PACKET = 1; const LAST_DOWNLOAD_PACKET = 2;) and so forth. Whenever the client application would receive a packet, I'd split the packet into 2 - header info and data. Header information told me what to do with the information while the data contained the actual information.

You could create a class to handle something like this too (something like a packet class) for re-using in other apps using sockets.

  • 0
  dlegend said:
The main way to doing this (which you may have realized) is to send header data with each packet sent. About 5 years back when I got into remote desktop applications (using VB6 and winsock) I'd send something like 0|DATA. Each time something was sent, an integer was prefixed with the character | as the splitter. Alternatively, if you aren't sure what characters may be sent through, you can instead use char(0) (invisible char if displayed but detectable w/ programming). I'd have constants/enumerator defining exactly what each pre-fixed integer means (something like const FIRST_DOWNLOAD_PACKET = 0; const NEXT_DOWNLOAD_PACKET = 1; const LAST_DOWNLOAD_PACKET = 2;) and so forth. Whenever the client application would receive a packet, I'd split the packet into 2 - header info and data. Header information told me what to do with the information while the data contained the actual information.

You could create a class to handle something like this too (something like a packet class) for re-using in other apps using sockets.

Yes, that's pretty much what I did, although more simple. My problem was basically how to send variable length messages; the receiving end has no idea where a message begins and where it ends. Sending fixed-sized messages is fine though, I can just buffer the input and split every [LENGTH] bytes. (where LENGTH is, of course, a constant)

So how I solved it is that I each time I send a message, I prefix it with a fixed-size header containing the length of the message. Another way I could have done it is to begin and end each message with a special character, but I was worried that could too easily break.

And yup this is all encapsulated in a nice, single-purpose, reusable class. :p

  • 0

the tcp protocol will ensure the data arrives in the correct order if you just break it up into chunks and send them off one after the other

the most stable/reliable method i've come across is where you send a packet and don't send anymore until the client responds with some form of acknowledgement

server sends 4KB

client responses "ok"

server sends 4KB

client responses "ok"

~ repeat

problem with that is the speed is now completely dependant on latency and you can't take advantage of burst techniques however if you're sending to a low latency network you shouldn't have many issues

always good to have various approaches for various network types to get the most out of them :D

  • 0
  DDStriker said:
the tcp protocol will ensure the data arrives in the correct order if you just break it up into chunks and send them off one after the other

the most stable/reliable method i've come across is where you send a packet and don't send anymore until the client responds with some form of acknowledgement

server sends 4KB

client responses "ok"

server sends 4KB

client responses "ok"

~ repeat

TCP handles acknowledgment and throttling for you automatically. If you're wanting to send data to a remote endpoint as fast as possible, you might as well send it in chunks of 4096 bytes and let TCP handle those. If throttling occurs, eventually send() will not return for awhile on the local side until the congestion has been alleviated.

  • 0

Since you're already splitting your messages at this point, you may want to reduce their size. 4096 byte packets are definitely going to be fragmented (unless this is a specialized closed network of some sort, this would even fragment on the old token ring), and that could result in latency due to retransmission. 1456 would be a typical size that would not fragment on most contemporary ethernet networks (1500 MTU, 20 byte IP header, 24 byte TCP header).

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • I'd concur too, as an informed enthusiast I'd find it a good deal, but your average Joe just wants cheap as possible and it's hard to beat Amazon or Ebay builds for that.
    • Now you know how many would-be Windows 11 users felt. I’m not justifying any of it, but MS and Apple both do it.
    • An Air India plane with 242 people on board crashed Thursday near a major international airport in the western Indian city of Ahmedabad, the airline and the country's government said. https://www.nbcnews.com/world/asia/air-india-plane-crashes-indias-ahmedabad-airport-rcna212545
    • NTLite 2025.06.10473 by Razvan Serea NTLite is a Windows configuration tool that allows you to modify your existing Windows install or an image yet to be deployed, remove Windows components, configure and integrate, speed up the Windows deployment process. Reduce Windows footprint on your RAM and storage drive memory. Remove components of your choice, guarded by compatibility safety mechanisms, which speed up finding that sweet spot. Windows Unattended feature support, providing many commonly used options on a single page for easy setup. Easily integrate a single or multiple drivers, update or language packages. Package integration features smart sorting, enabling you to seamlessly add packages for integration and the tool will apply them in the appropriate order, keeping hotfix compatibility in check. One of the important new features of NTLite (compared to its predecessors) is the ability to modify an already installed the operating system, by removing unnecessary components. Supports Windows 11, 10, 8.1 and 7, x86 and x64, live and image. Server editions of the same versions, excluding support for component removals and feature configuration. ARM64 image support in the alpha stage. Does not support Checked/Debug, Embedded, IoT editions, nor Vista or XP. NTLite 2025.06.10473 changelog: Upgrade UI: Text size now set as host, more readable on some systems Post-setup: Loading progress improvement for the 99% delay Components: ‘Snipping tool’ compatibility option UI-Translation: Thanks for Romanian (Coman) Fix Updates: Parallel WinRE integration getting stuck Updates: Download overwrite previous Defender updates Settings: Win10 News widget disabling Download: NTLite 64-bit | 21.5 MB (Free, paid upgrade available) Download: NTLite 32-bit | 19.3 MB Link: NTLite Home Page | NTLite Features | Screenshot Get alerted to all of our Software updates on Twitter at @NeowinSoftware
  • Recent Achievements

    • Week One Done
      elsafaacompany earned a badge
      Week One Done
    • Week One Done
      Yianis earned a badge
      Week One Done
    • Veteran
      Travesty went up a rank
      Veteran
    • One Month Later
      somar86 earned a badge
      One Month Later
    • Week One Done
      somar86 earned a badge
      Week One Done
  • Popular Contributors

    1. 1
      +primortal
      506
    2. 2
      ATLien_0
      258
    3. 3
      +FloatingFatMan
      183
    4. 4
      +Edouard
      180
    5. 5
      snowy owl
      131
  • Tell a friend

    Love Neowin? Tell a friend!