• 0

Sending large message over TCP


Question

Hello, I have a little client-server application using tcp sockets. On both sides I use a buffer size of 4096 and I run all my tests with the client and server on the same machine, using address 127.0.0.1. Everything is going well except that now I suddenly need to send much larger messages. A message could now easily be 12MB in size. Now, what happens is that the transmission is incomplete : on the receiving end, I only get a chunk of the message instead of the whole thing.

I thought about simply increasing the buffer sizes on both ends to something larger than my maximum message size, but I'm not sure that's a reliable method, even if it works, and I'm not sure it'll work (if it's supported by TCP, if I won't run out of memory, etc.).

I also thought about splitting the message and actually my sending code automatically does that already, it will keep sending until all the data has been sent. However, on the receiving end, how do I know where a message ends, do I need to implement a protocol for that? I thought TCP already took care of that.

Anyway I'm a bit lost there, thanks for any tips.

Link to comment
https://www.neowin.net/forum/topic/809920-sending-large-message-over-tcp/
Share on other sites

9 answers to this question

Recommended Posts

  • 0

The main way to doing this (which you may have realized) is to send header data with each packet sent. About 5 years back when I got into remote desktop applications (using VB6 and winsock) I'd send something like 0|DATA. Each time something was sent, an integer was prefixed with the character | as the splitter. Alternatively, if you aren't sure what characters may be sent through, you can instead use char(0) (invisible char if displayed but detectable w/ programming). I'd have constants/enumerator defining exactly what each pre-fixed integer means (something like const FIRST_DOWNLOAD_PACKET = 0; const NEXT_DOWNLOAD_PACKET = 1; const LAST_DOWNLOAD_PACKET = 2;) and so forth. Whenever the client application would receive a packet, I'd split the packet into 2 - header info and data. Header information told me what to do with the information while the data contained the actual information.

You could create a class to handle something like this too (something like a packet class) for re-using in other apps using sockets.

  • 0
  dlegend said:
The main way to doing this (which you may have realized) is to send header data with each packet sent. About 5 years back when I got into remote desktop applications (using VB6 and winsock) I'd send something like 0|DATA. Each time something was sent, an integer was prefixed with the character | as the splitter. Alternatively, if you aren't sure what characters may be sent through, you can instead use char(0) (invisible char if displayed but detectable w/ programming). I'd have constants/enumerator defining exactly what each pre-fixed integer means (something like const FIRST_DOWNLOAD_PACKET = 0; const NEXT_DOWNLOAD_PACKET = 1; const LAST_DOWNLOAD_PACKET = 2;) and so forth. Whenever the client application would receive a packet, I'd split the packet into 2 - header info and data. Header information told me what to do with the information while the data contained the actual information.

You could create a class to handle something like this too (something like a packet class) for re-using in other apps using sockets.

Yes, that's pretty much what I did, although more simple. My problem was basically how to send variable length messages; the receiving end has no idea where a message begins and where it ends. Sending fixed-sized messages is fine though, I can just buffer the input and split every [LENGTH] bytes. (where LENGTH is, of course, a constant)

So how I solved it is that I each time I send a message, I prefix it with a fixed-size header containing the length of the message. Another way I could have done it is to begin and end each message with a special character, but I was worried that could too easily break.

And yup this is all encapsulated in a nice, single-purpose, reusable class. :p

  • 0

the tcp protocol will ensure the data arrives in the correct order if you just break it up into chunks and send them off one after the other

the most stable/reliable method i've come across is where you send a packet and don't send anymore until the client responds with some form of acknowledgement

server sends 4KB

client responses "ok"

server sends 4KB

client responses "ok"

~ repeat

problem with that is the speed is now completely dependant on latency and you can't take advantage of burst techniques however if you're sending to a low latency network you shouldn't have many issues

always good to have various approaches for various network types to get the most out of them :D

  • 0
  DDStriker said:
the tcp protocol will ensure the data arrives in the correct order if you just break it up into chunks and send them off one after the other

the most stable/reliable method i've come across is where you send a packet and don't send anymore until the client responds with some form of acknowledgement

server sends 4KB

client responses "ok"

server sends 4KB

client responses "ok"

~ repeat

TCP handles acknowledgment and throttling for you automatically. If you're wanting to send data to a remote endpoint as fast as possible, you might as well send it in chunks of 4096 bytes and let TCP handle those. If throttling occurs, eventually send() will not return for awhile on the local side until the congestion has been alleviated.

  • 0

Since you're already splitting your messages at this point, you may want to reduce their size. 4096 byte packets are definitely going to be fragmented (unless this is a specialized closed network of some sort, this would even fragment on the old token ring), and that could result in latency due to retransmission. 1456 would be a typical size that would not fragment on most contemporary ethernet networks (1500 MTU, 20 byte IP header, 24 byte TCP header).

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • 2025 finally the Year of Linux? LibreOffice explains "real costs" of Windows 11 by Sayan Sen A big change is coming to Windows PCs as Microsoft will soon end support for systems and devices running on Windows 10. As such, the company, alongside its partners like AMD, Asus, and Dell, have begun urging users to embrace the "mandatory Windows 11 upgrade." The problem is that not every PC out there will be able to do so, at least not officially, as Microsoft had declared higher requirements for Windows 11 and thus many systems would be left out. Redmond's official stance for such situations is that users get a new computer by dumping their older system. There is another option users have: switching to Linux. Back in January, earlier this year, ESET recommended that users do that if they can not upgrade from Windows 10 to 11 or perhaps when they do not want to. Last month, KDE launched a new campaign dubbed "Endof10", which encourages users to make the jump. The project page explains several of the benefits of Linux over an unsupported Windows 10 system, like security and privacy, among others. And it also published another post earlier this month welcoming such "Windows 10 exiles". Now, The Document Foundation, maker of LibreOffice, has also joined in to support the Endof10 initiative. The foundation writes: "You don’t have to follow Microsoft’s upgrade path. There is a better option that puts control back in the hands of users, institutions, and public bodies: Linux and LibreOffice. Together, these two programmes offer a powerful, privacy-friendly and future-proof alternative to the Windows + Microsoft 365 ecosystem." It further adds the "real costs" of upgrading to Windows 11 as it writes: "The move to Windows 11 isn’t just about security updates. It increases dependence on Microsoft through aggressive cloud integration, forcing users to adopt Microsoft accounts and services. It also leads to higher costs due to subscription and licensing models, and reduces control over how your computer works and how your data is managed. Furthermore, new hardware requirements will render millions of perfectly good PCs obsolete. .... The end of Windows 10 does not mark the end of choice, but the beginning of a new era. If you are tired of mandatory updates, invasive changes, and being bound by the commercial choices of a single supplier, it is time for a change. Linux and LibreOffice are ready — 2025 is the right year to choose digital freedom!" To help users with the migration from Windows to Linux, The Document Foundation has laid out some key steps on how to proceed: Start by testing Linux and LibreOffice on a second partition of your PC (for individuals) or in less critical departments (for companies). Check the compatibility of your software configuration with Linux and LibreOffice; most office tasks can easily be transferred or adapted with minimal effort. Build documentation to learn how Linux and LibreOffice work and organise training if necessary. Find a consultant who can help with the migration process, such as someone certified by the Linux Professional Institute or The Document Foundation (for LibreOffice). The foundation stresses how "important" it is to "start immediately" with the transition. You can find the full details about the announcement here in the official blog post.
    • Why is it that some people think we have a tinfoil hat mentality, as you put it, just because we don't want AI on our devices? As for MS or any other company looking at everything, it seems to be the thing these days that companies want to know all about us. Can't even go shopping these days without being asked if you have some sort of card that tell them what you are buying and who you are.
    • Until the employer looks right though the AI created letter and notice it is AI created. You will be surprised how many can tell, there is something about an AI created letter that stands out to some people.
    • Snapchat brings new features, including auto-saving for public Stories by David Uzondu Snapchat, which recently expanded to watchOS after a decade, is rolling out new tools for its creators. The company says people are watching more content than ever, so these updates are meant to help creators get more out of that attention. The new features focus on a few key areas: making video editing less of a hassle, showing who is actually watching, and saving posts that used to vanish forever. Video editing on the platform has always been a huge problem. The tools are fine for quick clips, but they get messy if you try to do anything more complex. A new Timeline Editor for creators in the US is coming "soon" to fix this. It shows your video clips in a simple timeline, letting you cut and move parts around without wanting to pull your hair out. You can still add all the usual Lenses and music on top of these edits. For even quicker edits, a "Create a Video" template is now available globally for iOS users. The idea is simple: you look through your saved Memories, pick a handful of photos or videos, and then choose a song from the app's library. After you select a template, Snapchat mashes it all together into a single compilation video for you. But making videos is only part of the job. To help people see if anyone is paying attention, new insights are also available. These stats give a much better idea of how an audience behaves. Instead of just one giant view number, creators can now track "Returning Viewers." This counts how many people watched their content at least 12 times in the last month. Other metrics include the "Spotlight Average View Rate," showing what percentage of a video people finished. You can also see "Views by Traffic Sources," which tells you if people found you from the Discover page, search, or their own feed. Finally, there is a new "Auto-Save Stories to Public Profiles" feature. For years, the whole point of a Snapchat Story was that it was temporary. This new option, which is rolling out to creators globally, automatically saves public Stories to a creator's profile. Snapchat notes that the new feature "helps creators build a lasting collection of their best moments, giving fans a deeper look into their creative journey and making it easier to keep content alive, accessible, and meaningful over time."
    • I did not think it could be removed, all it does is remove the icon, co-pilot is still there as far as I know. Also don't forget the other apps it is in. What is needed is a simple thing to click to remove co-pilot and any AI rubbish from the computer, just like I can on my Mac.
  • Recent Achievements

    • One Month Later
      POR2GAL4EVER earned a badge
      One Month Later
    • One Year In
      Orpheus13 earned a badge
      One Year In
    • One Month Later
      Orpheus13 earned a badge
      One Month Later
    • Week One Done
      Orpheus13 earned a badge
      Week One Done
    • Week One Done
      serfegyed earned a badge
      Week One Done
  • Popular Contributors

    1. 1
      +primortal
      562
    2. 2
      ATLien_0
      256
    3. 3
      +Edouard
      163
    4. 4
      +FloatingFatMan
      156
    5. 5
      Michael Scrip
      109
  • Tell a friend

    Love Neowin? Tell a friend!