• 0

Sending large message over TCP


Question

Hello, I have a little client-server application using tcp sockets. On both sides I use a buffer size of 4096 and I run all my tests with the client and server on the same machine, using address 127.0.0.1. Everything is going well except that now I suddenly need to send much larger messages. A message could now easily be 12MB in size. Now, what happens is that the transmission is incomplete : on the receiving end, I only get a chunk of the message instead of the whole thing.

I thought about simply increasing the buffer sizes on both ends to something larger than my maximum message size, but I'm not sure that's a reliable method, even if it works, and I'm not sure it'll work (if it's supported by TCP, if I won't run out of memory, etc.).

I also thought about splitting the message and actually my sending code automatically does that already, it will keep sending until all the data has been sent. However, on the receiving end, how do I know where a message ends, do I need to implement a protocol for that? I thought TCP already took care of that.

Anyway I'm a bit lost there, thanks for any tips.

Link to comment
https://www.neowin.net/forum/topic/809920-sending-large-message-over-tcp/
Share on other sites

9 answers to this question

Recommended Posts

  • 0

The main way to doing this (which you may have realized) is to send header data with each packet sent. About 5 years back when I got into remote desktop applications (using VB6 and winsock) I'd send something like 0|DATA. Each time something was sent, an integer was prefixed with the character | as the splitter. Alternatively, if you aren't sure what characters may be sent through, you can instead use char(0) (invisible char if displayed but detectable w/ programming). I'd have constants/enumerator defining exactly what each pre-fixed integer means (something like const FIRST_DOWNLOAD_PACKET = 0; const NEXT_DOWNLOAD_PACKET = 1; const LAST_DOWNLOAD_PACKET = 2;) and so forth. Whenever the client application would receive a packet, I'd split the packet into 2 - header info and data. Header information told me what to do with the information while the data contained the actual information.

You could create a class to handle something like this too (something like a packet class) for re-using in other apps using sockets.

  • 0
  dlegend said:
The main way to doing this (which you may have realized) is to send header data with each packet sent. About 5 years back when I got into remote desktop applications (using VB6 and winsock) I'd send something like 0|DATA. Each time something was sent, an integer was prefixed with the character | as the splitter. Alternatively, if you aren't sure what characters may be sent through, you can instead use char(0) (invisible char if displayed but detectable w/ programming). I'd have constants/enumerator defining exactly what each pre-fixed integer means (something like const FIRST_DOWNLOAD_PACKET = 0; const NEXT_DOWNLOAD_PACKET = 1; const LAST_DOWNLOAD_PACKET = 2;) and so forth. Whenever the client application would receive a packet, I'd split the packet into 2 - header info and data. Header information told me what to do with the information while the data contained the actual information.

You could create a class to handle something like this too (something like a packet class) for re-using in other apps using sockets.

Yes, that's pretty much what I did, although more simple. My problem was basically how to send variable length messages; the receiving end has no idea where a message begins and where it ends. Sending fixed-sized messages is fine though, I can just buffer the input and split every [LENGTH] bytes. (where LENGTH is, of course, a constant)

So how I solved it is that I each time I send a message, I prefix it with a fixed-size header containing the length of the message. Another way I could have done it is to begin and end each message with a special character, but I was worried that could too easily break.

And yup this is all encapsulated in a nice, single-purpose, reusable class. :p

  • 0

the tcp protocol will ensure the data arrives in the correct order if you just break it up into chunks and send them off one after the other

the most stable/reliable method i've come across is where you send a packet and don't send anymore until the client responds with some form of acknowledgement

server sends 4KB

client responses "ok"

server sends 4KB

client responses "ok"

~ repeat

problem with that is the speed is now completely dependant on latency and you can't take advantage of burst techniques however if you're sending to a low latency network you shouldn't have many issues

always good to have various approaches for various network types to get the most out of them :D

  • 0
  DDStriker said:
the tcp protocol will ensure the data arrives in the correct order if you just break it up into chunks and send them off one after the other

the most stable/reliable method i've come across is where you send a packet and don't send anymore until the client responds with some form of acknowledgement

server sends 4KB

client responses "ok"

server sends 4KB

client responses "ok"

~ repeat

TCP handles acknowledgment and throttling for you automatically. If you're wanting to send data to a remote endpoint as fast as possible, you might as well send it in chunks of 4096 bytes and let TCP handle those. If throttling occurs, eventually send() will not return for awhile on the local side until the congestion has been alleviated.

  • 0

Since you're already splitting your messages at this point, you may want to reduce their size. 4096 byte packets are definitely going to be fragmented (unless this is a specialized closed network of some sort, this would even fragment on the old token ring), and that could result in latency due to retransmission. 1456 would be a typical size that would not fragment on most contemporary ethernet networks (1500 MTU, 20 byte IP header, 24 byte TCP header).

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • LibreOffice narrows gap with Microsoft Office in 25.8 Beta 1 by David Uzondu The Document Foundation has released LibreOffice 25.8 Beta 1 for public testing on Linux, macOS, and Windows. This is the second pre-release for the 25.8 cycle and the foundation says that the final, stable version of LibreOffice 25.8 is expected to land at the end of August 2025. Starting off with Writer, LibreOffice's Word, the developers have finally addressed some long-standing annoyances, including a new command to easily insert a paragraph break right before a table. This beta also introduces a useful privacy feature in its Auto-Redact tool, letting you strip all images from a document with a single option. To use it, go to Tools and select the Auto-Redact option: The application has improved its ability to handle different languages for punctuation, preventing mix-ups in multilingual documents. Other notable improvements have also been made. A new hyphenation rule lets you choose to prevent a word from splitting at the end of a page, moving the whole line to the next page instead. Microsoft Word has had this feature for years now. The Navigator now displays a handy tooltip with word and character counts for headings and their sub-outlines. Scrolling behavior when selecting text has been improved, making it less erratic. A new command with a keyboard shortcut was added for converting fields into plain text. Calc gets a lot of new functions that bring it closer to its competitors like Excel, including TEXTSPLIT, VSTACK, and WRAPROWS. Impress now properly supports embedded fonts in PPTX files, which should reduce headaches when sharing presentations with PowerPoint users. Alongside these additions, the project is also cleaning house; support for Windows 7, 8, and 8.1 has been completely dropped. There are also smaller UI tweaks across the suite, like allowing a single click to enter rotation mode for objects in Writer and Calc. macOS users get better integration, with proper support for native full screen mode and new window management features from the Sequoia update. In terms of performance, the team has optimized everything from loading huge DOC files and XLSX spreadsheets with tons of conditional formatting to simply switching between sheets in Calc. These improvements should be noticeable, especially when working with complex documents. A new application-wide "Viewer mode" has also been implemented, which opens all files in a read-only state for quick, safe viewing. On a related note, The Document Foundation has joined efforts by the likes of KDE to encourage Windows 10 users to switch to Linux. Also, you might have heard that Denmark, in a bid to lessen its reliance on Microsoft, has decided to make a full switch to LibreOffice, with plans to begin phasing out Office 365 in certain ministries as early as next month. If you're interested in this release, you can read the full release notes and download the binaries for your platform: Windows, macOS (Intel | Apple Silicon), or Linux (DEB | RPM). You can also get the latest stable version from our software stories page.
    • Until it can be used 100% offline (ie: PST file support or equiv) not even considering it. I'll jump to Thunderbird first which has gotten a LOT better since the last time I looked at it.
  • Recent Achievements

    • Explorer
      Case_f went up a rank
      Explorer
    • Conversation Starter
      Jamie Smith earned a badge
      Conversation Starter
    • First Post
      NeoToad777 earned a badge
      First Post
    • Week One Done
      JoeV earned a badge
      Week One Done
    • One Month Later
      VAT Services in UAE earned a badge
      One Month Later
  • Popular Contributors

    1. 1
      +primortal
      545
    2. 2
      ATLien_0
      227
    3. 3
      +FloatingFatMan
      159
    4. 4
      Michael Scrip
      113
    5. 5
      +Edouard
      105
  • Tell a friend

    Love Neowin? Tell a friend!