• 0

Sending large message over TCP


Question

Hello, I have a little client-server application using tcp sockets. On both sides I use a buffer size of 4096 and I run all my tests with the client and server on the same machine, using address 127.0.0.1. Everything is going well except that now I suddenly need to send much larger messages. A message could now easily be 12MB in size. Now, what happens is that the transmission is incomplete : on the receiving end, I only get a chunk of the message instead of the whole thing.

I thought about simply increasing the buffer sizes on both ends to something larger than my maximum message size, but I'm not sure that's a reliable method, even if it works, and I'm not sure it'll work (if it's supported by TCP, if I won't run out of memory, etc.).

I also thought about splitting the message and actually my sending code automatically does that already, it will keep sending until all the data has been sent. However, on the receiving end, how do I know where a message ends, do I need to implement a protocol for that? I thought TCP already took care of that.

Anyway I'm a bit lost there, thanks for any tips.

Link to comment
https://www.neowin.net/forum/topic/809920-sending-large-message-over-tcp/
Share on other sites

9 answers to this question

Recommended Posts

  • 0

The main way to doing this (which you may have realized) is to send header data with each packet sent. About 5 years back when I got into remote desktop applications (using VB6 and winsock) I'd send something like 0|DATA. Each time something was sent, an integer was prefixed with the character | as the splitter. Alternatively, if you aren't sure what characters may be sent through, you can instead use char(0) (invisible char if displayed but detectable w/ programming). I'd have constants/enumerator defining exactly what each pre-fixed integer means (something like const FIRST_DOWNLOAD_PACKET = 0; const NEXT_DOWNLOAD_PACKET = 1; const LAST_DOWNLOAD_PACKET = 2;) and so forth. Whenever the client application would receive a packet, I'd split the packet into 2 - header info and data. Header information told me what to do with the information while the data contained the actual information.

You could create a class to handle something like this too (something like a packet class) for re-using in other apps using sockets.

  • 0
  dlegend said:
The main way to doing this (which you may have realized) is to send header data with each packet sent. About 5 years back when I got into remote desktop applications (using VB6 and winsock) I'd send something like 0|DATA. Each time something was sent, an integer was prefixed with the character | as the splitter. Alternatively, if you aren't sure what characters may be sent through, you can instead use char(0) (invisible char if displayed but detectable w/ programming). I'd have constants/enumerator defining exactly what each pre-fixed integer means (something like const FIRST_DOWNLOAD_PACKET = 0; const NEXT_DOWNLOAD_PACKET = 1; const LAST_DOWNLOAD_PACKET = 2;) and so forth. Whenever the client application would receive a packet, I'd split the packet into 2 - header info and data. Header information told me what to do with the information while the data contained the actual information.

You could create a class to handle something like this too (something like a packet class) for re-using in other apps using sockets.

Yes, that's pretty much what I did, although more simple. My problem was basically how to send variable length messages; the receiving end has no idea where a message begins and where it ends. Sending fixed-sized messages is fine though, I can just buffer the input and split every [LENGTH] bytes. (where LENGTH is, of course, a constant)

So how I solved it is that I each time I send a message, I prefix it with a fixed-size header containing the length of the message. Another way I could have done it is to begin and end each message with a special character, but I was worried that could too easily break.

And yup this is all encapsulated in a nice, single-purpose, reusable class. :p

  • 0

the tcp protocol will ensure the data arrives in the correct order if you just break it up into chunks and send them off one after the other

the most stable/reliable method i've come across is where you send a packet and don't send anymore until the client responds with some form of acknowledgement

server sends 4KB

client responses "ok"

server sends 4KB

client responses "ok"

~ repeat

problem with that is the speed is now completely dependant on latency and you can't take advantage of burst techniques however if you're sending to a low latency network you shouldn't have many issues

always good to have various approaches for various network types to get the most out of them :D

  • 0
  DDStriker said:
the tcp protocol will ensure the data arrives in the correct order if you just break it up into chunks and send them off one after the other

the most stable/reliable method i've come across is where you send a packet and don't send anymore until the client responds with some form of acknowledgement

server sends 4KB

client responses "ok"

server sends 4KB

client responses "ok"

~ repeat

TCP handles acknowledgment and throttling for you automatically. If you're wanting to send data to a remote endpoint as fast as possible, you might as well send it in chunks of 4096 bytes and let TCP handle those. If throttling occurs, eventually send() will not return for awhile on the local side until the congestion has been alleviated.

  • 0

Since you're already splitting your messages at this point, you may want to reduce their size. 4096 byte packets are definitely going to be fragmented (unless this is a specialized closed network of some sort, this would even fragment on the old token ring), and that could result in latency due to retransmission. 1456 would be a typical size that would not fragment on most contemporary ethernet networks (1500 MTU, 20 byte IP header, 24 byte TCP header).

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • Taras Buria, do you think you could write more about Windows Vista? It is always such a pleasure — quite a pleasure — to see it mentioned in the title of a news article. What about an editorial?
    • Alienware 34 240Hz QD-OLED Monitor: A premium ultrawide for serious gamers - save $100! by Paul Hill Are you a serious gamer looking to save money on a monitor upgrade? If so, check out this deal on the Alienware 34 240Hz QD-OLED Gaming Monitor (AW3425DW), which is discounted by 13% from $799.99 to just $699.99. This ultrawide monitor features a 3,440 x 1,440 pixel display and the screen is curved for added immersion. Two features that will stand out to competitive gamers are the 240Hz refresh rate and 0.03ms GtG response time which will minimize any lagging from input to display, making you a more lethal player. Why QD-OLED matters for gamers and creators This Alienware uses QD-OLED (Quantum Dot Organic Light Emitting Diode) technology which combines OLED’s self-emitting LEDs with the color-enhancing capabilities of quantum dots. This results in higher brightness, a wider color gamut, deeper blacks, and wide viewing angles. With the 1800R curve built into this display, whatever you’re immersed in will wrap around, making your games more absorbing, potentially reducing other distractions. There is also VESA DisplayHDR TrueBlack 400 certification and 1,000 nits peak HDR brightness on this display. Gamers using this monitor will be able to get the best quality picture from this monitor to improve the experience and their competitiveness. It’s also good for creators who want to edit images and videos as they will see the content they're editing in the best ways possible, so they can be totally sure it’s ready for publication. Design, connectivity, and user experience The Alienware 34 uses an updated Interstellar Indigo design which is more compact and uses a flat stand. This frees up desk space compared to older Alienware designs and still allows for height, tilt, and swivel adjustments and VESA mount compatibility. Regarding ports, this monitor features 2x HDMI 2.1, 1x DisplayPort 1.4, 1x USB 5Gbps Type-B upstream, 1x USB 5Gbps Type-A downstream, 1x USB-C 5Gbps downstream with 15W charging. The USB-C port is for data and charging, not video input and the lack of KVM switch could be a drawback for some. If you decide to pick up this monitor, according to PCWorld, it does not include in-built speakers so you will need to connect your own. This is not a big issue because gamers looking for a premium playing experience will not want to rely on subpar in-built speakers anyway. Is the AW3425DW the right fit for you? Given its curved display, quality QD-LED display technology, and its fast response time, this monitor is a great pick for serious gamers looking for top-tier performance and immersion. Content creators who value color accuracy will also do well in picking up this Alienware monitor. Alienware 34: $699.99 (Amazon US) / MSRP $799.99 This Amazon deal is US-specific and not available in other regions unless specified. If you don't like it or want to look at more options, check out the Amazon US deals page here. Get Prime (SNAP), Prime Video, Audible Plus or Kindle / Music Unlimited. Free for 30 days. As an Amazon Associate, we earn from qualifying purchases.
  • Recent Achievements

    • Rising Star
      Phillip0web went up a rank
      Rising Star
    • One Month Later
      Epaminombas earned a badge
      One Month Later
    • One Year In
      Bert Fershner earned a badge
      One Year In
    • Reacting Well
      ChrisOdinUK earned a badge
      Reacting Well
    • One Year In
      Steviant earned a badge
      One Year In
  • Popular Contributors

    1. 1
      +primortal
      545
    2. 2
      ATLien_0
      205
    3. 3
      +FloatingFatMan
      170
    4. 4
      Michael Scrip
      150
    5. 5
      Som
      131
  • Tell a friend

    Love Neowin? Tell a friend!