• 0

Sending large message over TCP


Question

Hello, I have a little client-server application using tcp sockets. On both sides I use a buffer size of 4096 and I run all my tests with the client and server on the same machine, using address 127.0.0.1. Everything is going well except that now I suddenly need to send much larger messages. A message could now easily be 12MB in size. Now, what happens is that the transmission is incomplete : on the receiving end, I only get a chunk of the message instead of the whole thing.

I thought about simply increasing the buffer sizes on both ends to something larger than my maximum message size, but I'm not sure that's a reliable method, even if it works, and I'm not sure it'll work (if it's supported by TCP, if I won't run out of memory, etc.).

I also thought about splitting the message and actually my sending code automatically does that already, it will keep sending until all the data has been sent. However, on the receiving end, how do I know where a message ends, do I need to implement a protocol for that? I thought TCP already took care of that.

Anyway I'm a bit lost there, thanks for any tips.

Link to comment
https://www.neowin.net/forum/topic/809920-sending-large-message-over-tcp/
Share on other sites

9 answers to this question

Recommended Posts

  • 0

The main way to doing this (which you may have realized) is to send header data with each packet sent. About 5 years back when I got into remote desktop applications (using VB6 and winsock) I'd send something like 0|DATA. Each time something was sent, an integer was prefixed with the character | as the splitter. Alternatively, if you aren't sure what characters may be sent through, you can instead use char(0) (invisible char if displayed but detectable w/ programming). I'd have constants/enumerator defining exactly what each pre-fixed integer means (something like const FIRST_DOWNLOAD_PACKET = 0; const NEXT_DOWNLOAD_PACKET = 1; const LAST_DOWNLOAD_PACKET = 2;) and so forth. Whenever the client application would receive a packet, I'd split the packet into 2 - header info and data. Header information told me what to do with the information while the data contained the actual information.

You could create a class to handle something like this too (something like a packet class) for re-using in other apps using sockets.

  • 0
  dlegend said:
The main way to doing this (which you may have realized) is to send header data with each packet sent. About 5 years back when I got into remote desktop applications (using VB6 and winsock) I'd send something like 0|DATA. Each time something was sent, an integer was prefixed with the character | as the splitter. Alternatively, if you aren't sure what characters may be sent through, you can instead use char(0) (invisible char if displayed but detectable w/ programming). I'd have constants/enumerator defining exactly what each pre-fixed integer means (something like const FIRST_DOWNLOAD_PACKET = 0; const NEXT_DOWNLOAD_PACKET = 1; const LAST_DOWNLOAD_PACKET = 2;) and so forth. Whenever the client application would receive a packet, I'd split the packet into 2 - header info and data. Header information told me what to do with the information while the data contained the actual information.

You could create a class to handle something like this too (something like a packet class) for re-using in other apps using sockets.

Yes, that's pretty much what I did, although more simple. My problem was basically how to send variable length messages; the receiving end has no idea where a message begins and where it ends. Sending fixed-sized messages is fine though, I can just buffer the input and split every [LENGTH] bytes. (where LENGTH is, of course, a constant)

So how I solved it is that I each time I send a message, I prefix it with a fixed-size header containing the length of the message. Another way I could have done it is to begin and end each message with a special character, but I was worried that could too easily break.

And yup this is all encapsulated in a nice, single-purpose, reusable class. :p

  • 0

the tcp protocol will ensure the data arrives in the correct order if you just break it up into chunks and send them off one after the other

the most stable/reliable method i've come across is where you send a packet and don't send anymore until the client responds with some form of acknowledgement

server sends 4KB

client responses "ok"

server sends 4KB

client responses "ok"

~ repeat

problem with that is the speed is now completely dependant on latency and you can't take advantage of burst techniques however if you're sending to a low latency network you shouldn't have many issues

always good to have various approaches for various network types to get the most out of them :D

  • 0
  DDStriker said:
the tcp protocol will ensure the data arrives in the correct order if you just break it up into chunks and send them off one after the other

the most stable/reliable method i've come across is where you send a packet and don't send anymore until the client responds with some form of acknowledgement

server sends 4KB

client responses "ok"

server sends 4KB

client responses "ok"

~ repeat

TCP handles acknowledgment and throttling for you automatically. If you're wanting to send data to a remote endpoint as fast as possible, you might as well send it in chunks of 4096 bytes and let TCP handle those. If throttling occurs, eventually send() will not return for awhile on the local side until the congestion has been alleviated.

  • 0

Since you're already splitting your messages at this point, you may want to reduce their size. 4096 byte packets are definitely going to be fragmented (unless this is a specialized closed network of some sort, this would even fragment on the old token ring), and that could result in latency due to retransmission. 1456 would be a typical size that would not fragment on most contemporary ethernet networks (1500 MTU, 20 byte IP header, 24 byte TCP header).

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • Garmin Dash Cam X310: 4K dash cam on sale at lowest price for $334.99 by Paul Hill Dash cam footage can be vital for drivers who get into accidents as it can be used to help with insurance claims if you’re involved in an accident, outside of that, it can be used for filming your trips or recording freak events such as asteroids burning up in the atmosphere. If you’re still without one, or looking to upgrade, take a moment to read about the Garmin Dash Cam X310 which has been reduced by 16% from $399.99 to $334.99. To briefly touch on the features before we get started, the X310 is a compact 4K Ultra HD touchscreen dash cam with a 140-degree field of view, ensuring it captures everything. There’s also a built-in Clarity Polarizer to reduce glare from the windshield and it features automatic recording. The main attraction of the Garmin Dash Cam X310 is undoubtedly the 4K Ultra HD video which will capture fine details, and the 140-degree field of view that ensures it doesn’t miss anything important. Coupled with this, the Clarity Polarizer helps to reduce glare from the windshield so that your video footage is better. It also includes HDR which improves the night vision. Aside from having a touchscreen, the X310 comes with voice control allowing you to save video and audio hands-free, meaning you can focus on driving. It supports multiple languages - English, German, French, Spanish, Italian, and Swedish. Other features include built-in GPS for recording the location, date, and time of incidents; parking guard and live view for security while parked (required a paid Vault subscription); and its compact design which allows it to sit “virtually unnoticed” on the windshield. Please note that the product description says some jurisdictions may restrict the use of dashcams, so check your local laws before buying. If you are looking for a high-resolution dash cam with voice control and advanced parking surveillance (paid subscription), the X310 could be for you. The main downside of this product is that it’s still at a premium price point, despite it being at its lowest price on Amazon. If you don’t mind this, it could definitely be a great choice for you. Garmin Dash Cam X310: $334.99 (Amazon US) / MSRP $399.99 This Amazon deal is US-specific and not available in other regions unless specified. If you don't like it or want to look at more options, check out the Amazon US deals page here. Get Prime (SNAP), Prime Video, Audible Plus or Kindle / Music Unlimited. Free for 30 days. As an Amazon Associate, we earn from qualifying purchases.
    • I have a Motorola B12, and it's been working pretty solidly for a year now. 
    • yes, which is especially funny considering you no longer have an option to turn them off, so how could this matter to anyone? It would be pretty evil if they waited until after Windows 10 EOL to renew the certs, but based on these dates, it looks like that wouldn't be possible, so no worries that I can see.
  • Recent Achievements

    • Week One Done
      Marites earned a badge
      Week One Done
    • One Year In
      runge100 earned a badge
      One Year In
    • One Month Later
      runge100 earned a badge
      One Month Later
    • One Month Later
      jfam earned a badge
      One Month Later
    • First Post
      TheRingmaster earned a badge
      First Post
  • Popular Contributors

    1. 1
      +primortal
      566
    2. 2
      +FloatingFatMan
      178
    3. 3
      ATLien_0
      173
    4. 4
      Michael Scrip
      129
    5. 5
      Xenon
      119
  • Tell a friend

    Love Neowin? Tell a friend!