• 0

Sending large message over TCP


Question

Hello, I have a little client-server application using tcp sockets. On both sides I use a buffer size of 4096 and I run all my tests with the client and server on the same machine, using address 127.0.0.1. Everything is going well except that now I suddenly need to send much larger messages. A message could now easily be 12MB in size. Now, what happens is that the transmission is incomplete : on the receiving end, I only get a chunk of the message instead of the whole thing.

I thought about simply increasing the buffer sizes on both ends to something larger than my maximum message size, but I'm not sure that's a reliable method, even if it works, and I'm not sure it'll work (if it's supported by TCP, if I won't run out of memory, etc.).

I also thought about splitting the message and actually my sending code automatically does that already, it will keep sending until all the data has been sent. However, on the receiving end, how do I know where a message ends, do I need to implement a protocol for that? I thought TCP already took care of that.

Anyway I'm a bit lost there, thanks for any tips.

Link to comment
https://www.neowin.net/forum/topic/809920-sending-large-message-over-tcp/
Share on other sites

9 answers to this question

Recommended Posts

  • 0

The main way to doing this (which you may have realized) is to send header data with each packet sent. About 5 years back when I got into remote desktop applications (using VB6 and winsock) I'd send something like 0|DATA. Each time something was sent, an integer was prefixed with the character | as the splitter. Alternatively, if you aren't sure what characters may be sent through, you can instead use char(0) (invisible char if displayed but detectable w/ programming). I'd have constants/enumerator defining exactly what each pre-fixed integer means (something like const FIRST_DOWNLOAD_PACKET = 0; const NEXT_DOWNLOAD_PACKET = 1; const LAST_DOWNLOAD_PACKET = 2;) and so forth. Whenever the client application would receive a packet, I'd split the packet into 2 - header info and data. Header information told me what to do with the information while the data contained the actual information.

You could create a class to handle something like this too (something like a packet class) for re-using in other apps using sockets.

  • 0
  dlegend said:
The main way to doing this (which you may have realized) is to send header data with each packet sent. About 5 years back when I got into remote desktop applications (using VB6 and winsock) I'd send something like 0|DATA. Each time something was sent, an integer was prefixed with the character | as the splitter. Alternatively, if you aren't sure what characters may be sent through, you can instead use char(0) (invisible char if displayed but detectable w/ programming). I'd have constants/enumerator defining exactly what each pre-fixed integer means (something like const FIRST_DOWNLOAD_PACKET = 0; const NEXT_DOWNLOAD_PACKET = 1; const LAST_DOWNLOAD_PACKET = 2;) and so forth. Whenever the client application would receive a packet, I'd split the packet into 2 - header info and data. Header information told me what to do with the information while the data contained the actual information.

You could create a class to handle something like this too (something like a packet class) for re-using in other apps using sockets.

Yes, that's pretty much what I did, although more simple. My problem was basically how to send variable length messages; the receiving end has no idea where a message begins and where it ends. Sending fixed-sized messages is fine though, I can just buffer the input and split every [LENGTH] bytes. (where LENGTH is, of course, a constant)

So how I solved it is that I each time I send a message, I prefix it with a fixed-size header containing the length of the message. Another way I could have done it is to begin and end each message with a special character, but I was worried that could too easily break.

And yup this is all encapsulated in a nice, single-purpose, reusable class. :p

  • 0

the tcp protocol will ensure the data arrives in the correct order if you just break it up into chunks and send them off one after the other

the most stable/reliable method i've come across is where you send a packet and don't send anymore until the client responds with some form of acknowledgement

server sends 4KB

client responses "ok"

server sends 4KB

client responses "ok"

~ repeat

problem with that is the speed is now completely dependant on latency and you can't take advantage of burst techniques however if you're sending to a low latency network you shouldn't have many issues

always good to have various approaches for various network types to get the most out of them :D

  • 0
  DDStriker said:
the tcp protocol will ensure the data arrives in the correct order if you just break it up into chunks and send them off one after the other

the most stable/reliable method i've come across is where you send a packet and don't send anymore until the client responds with some form of acknowledgement

server sends 4KB

client responses "ok"

server sends 4KB

client responses "ok"

~ repeat

TCP handles acknowledgment and throttling for you automatically. If you're wanting to send data to a remote endpoint as fast as possible, you might as well send it in chunks of 4096 bytes and let TCP handle those. If throttling occurs, eventually send() will not return for awhile on the local side until the congestion has been alleviated.

  • 0

Since you're already splitting your messages at this point, you may want to reduce their size. 4096 byte packets are definitely going to be fragmented (unless this is a specialized closed network of some sort, this would even fragment on the old token ring), and that could result in latency due to retransmission. 1456 would be a typical size that would not fragment on most contemporary ethernet networks (1500 MTU, 20 byte IP header, 24 byte TCP header).

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • Steam games now detail what accessibility features they offer on store pages by Pulasthi Ariyasinghe Valve has pushed a new update for its Steam platform, and it's fully focused on accessibility. The latest change makes it both easier for developers to show what accessibility options they offer within their games and for players to find options they require to enjoy titles. "This update comes after gathering valuable feedback from developers as well as players with disabilities," says Valve in the announcement blog post. "Over 5,000 applications have added details about their accessibility support, with more developers updating their games each day." Here are the options currently offered by Valve: Gameplay: Adjustable Difficulty: Players can adjust gameplay difficulty. Save Anytime: Players can save gameplay using both automatic and manual saves. Saving can be performed at any point in the game. Audio: Custom Volume Controls: Players can adjust the volume of the audio. Different types of audio can be muted independently from each other. Narrated Game Menus: Players can listen to game menus with narrated audio. Stereo Sound: Players can identify how far to the left or right sounds are coming from. Surround Sound: Players can identify how far in any direction sounds are coming from. Visual: Adjustable Text Size: Players can adjust text size. (In-game text, menu text, character dialog text, subtitle text) Subtitle Options: Players have options to customize the display of subtitles for all spoken content and essential audio information. Color Alternatives: Gameplay doesn't rely on colors to communicate important information, or players have an option to adjust colors used for distinguishing information. Camera Comfort: Players have an option to adjust or disable uncomfortable camera movement such as screen shaking, camera bob, or motion blur, or the game doesn't feature these effects. Input: Keyboard Only Option: Players can play the game with just a keyboard and no other additional input mechanisms such as a mouse or controller. Mouse Only Option: Players can play the game with just a mouse and no other additional input mechanisms such as a keyboard or controller. Touch Only Option: Players can play the game with just touch controls and no other additional input mechanisms such as a mouse, keyboard or controller. Playable without Timed Input: Players have an option to adjust gameplay to not require precisely timed button presses ("quick time events") or gameplay does not require such button presses. Text-to-speech: Text chat can be narrated out loud in real time. Speech-to-text communications: Voice chat can be read as a text transcript in real time. These can be found when searching in the filters as well as inside games' store pages in the features section on the right side, as seen in the screenshot above. While developers are not required to mention accessibility options of their games on store pages, Valve says it is highly recommended to make sure players that require it can find relevant games more easily. "We've worked to make it as easy as possible for developers to indicate these features are available by using feedback to standardize these options as much as possible," added the company.
    • They have a concept of a plan. How fitting.
    • As Total War turns 25, Creative Assembly is teasing a "new era" for the strategy franchise by Pulasthi Ariyasinghe Creative Assembly has now officially been making Total War games for 25 years. It was in June 2000 that the UK-based studio put forth Shogun: Total War for PC, with Electronic Arts publishing the ambitious project. With a quarter of a century now behind the series, the developer is now looking towards the "next era" of Total War. The studio will host its first-ever Total War showcase in December 2025, which is described as a "flagship video presentation unveiling what’s next for the franchise." Creative Assembly said that the showcase will have announcements for new games as well as content for existing titles, with developer commentary also happening regarding the future of the series. "We’re honoured and humbled to be one of the few video games that get the chance to celebrate such a huge milestone, and we want to thank colleagues past and present, as well as our fans for all the support," says Total War vice president Roger Collum. "2025 marks an exciting new era for the franchise as we reveal what’s next for Total War, and we can’t wait to celebrate throughout the year with everyone." Read an anniversary letter from Roger Collum regarding the studio's history, Total War beginnings, and more over here. Prior to the showcase, Creative Assembly will be hosting retrospective video interviews, developer livestreams, and more behind-the-scenes content regarding the series starting this August. There will also be multiplayer tournaments for fans, giveaways, Q&A sessions with developers, mod spotlights, and more events happening during this period. As the studio celebrates the occasion, a massive Total War historical games sale has just gone live on Steam. This has discounts of up to 75% across everything from the original Shogun: Total War to the latest entry, Total War: PHARAOH. The discounts will last until June 26.
    • ANY standard Windows PC, either desktop or laptop, can install the Copilot app, period, no NPU or Copilot+ certification required. I should know, I have it running on a 3770K (2011) Windows 11 PC and works perfectly fine. So, your laptop can definitely run it, too.
    • Just what the driving public does not need--still another very attractive and addictive distraction.
  • Recent Achievements

    • One Month Later
      POR2GAL4EVER earned a badge
      One Month Later
    • One Year In
      Orpheus13 earned a badge
      One Year In
    • One Month Later
      Orpheus13 earned a badge
      One Month Later
    • Week One Done
      Orpheus13 earned a badge
      Week One Done
    • Week One Done
      serfegyed earned a badge
      Week One Done
  • Popular Contributors

    1. 1
      +primortal
      538
    2. 2
      ATLien_0
      252
    3. 3
      +FloatingFatMan
      166
    4. 4
      +Edouard
      163
    5. 5
      Xenon
      112
  • Tell a friend

    Love Neowin? Tell a friend!