• 0

Sending large message over TCP


Question

Hello, I have a little client-server application using tcp sockets. On both sides I use a buffer size of 4096 and I run all my tests with the client and server on the same machine, using address 127.0.0.1. Everything is going well except that now I suddenly need to send much larger messages. A message could now easily be 12MB in size. Now, what happens is that the transmission is incomplete : on the receiving end, I only get a chunk of the message instead of the whole thing.

I thought about simply increasing the buffer sizes on both ends to something larger than my maximum message size, but I'm not sure that's a reliable method, even if it works, and I'm not sure it'll work (if it's supported by TCP, if I won't run out of memory, etc.).

I also thought about splitting the message and actually my sending code automatically does that already, it will keep sending until all the data has been sent. However, on the receiving end, how do I know where a message ends, do I need to implement a protocol for that? I thought TCP already took care of that.

Anyway I'm a bit lost there, thanks for any tips.

Link to comment
https://www.neowin.net/forum/topic/809920-sending-large-message-over-tcp/
Share on other sites

9 answers to this question

Recommended Posts

  • 0

The main way to doing this (which you may have realized) is to send header data with each packet sent. About 5 years back when I got into remote desktop applications (using VB6 and winsock) I'd send something like 0|DATA. Each time something was sent, an integer was prefixed with the character | as the splitter. Alternatively, if you aren't sure what characters may be sent through, you can instead use char(0) (invisible char if displayed but detectable w/ programming). I'd have constants/enumerator defining exactly what each pre-fixed integer means (something like const FIRST_DOWNLOAD_PACKET = 0; const NEXT_DOWNLOAD_PACKET = 1; const LAST_DOWNLOAD_PACKET = 2;) and so forth. Whenever the client application would receive a packet, I'd split the packet into 2 - header info and data. Header information told me what to do with the information while the data contained the actual information.

You could create a class to handle something like this too (something like a packet class) for re-using in other apps using sockets.

  • 0
  dlegend said:
The main way to doing this (which you may have realized) is to send header data with each packet sent. About 5 years back when I got into remote desktop applications (using VB6 and winsock) I'd send something like 0|DATA. Each time something was sent, an integer was prefixed with the character | as the splitter. Alternatively, if you aren't sure what characters may be sent through, you can instead use char(0) (invisible char if displayed but detectable w/ programming). I'd have constants/enumerator defining exactly what each pre-fixed integer means (something like const FIRST_DOWNLOAD_PACKET = 0; const NEXT_DOWNLOAD_PACKET = 1; const LAST_DOWNLOAD_PACKET = 2;) and so forth. Whenever the client application would receive a packet, I'd split the packet into 2 - header info and data. Header information told me what to do with the information while the data contained the actual information.

You could create a class to handle something like this too (something like a packet class) for re-using in other apps using sockets.

Yes, that's pretty much what I did, although more simple. My problem was basically how to send variable length messages; the receiving end has no idea where a message begins and where it ends. Sending fixed-sized messages is fine though, I can just buffer the input and split every [LENGTH] bytes. (where LENGTH is, of course, a constant)

So how I solved it is that I each time I send a message, I prefix it with a fixed-size header containing the length of the message. Another way I could have done it is to begin and end each message with a special character, but I was worried that could too easily break.

And yup this is all encapsulated in a nice, single-purpose, reusable class. :p

  • 0

the tcp protocol will ensure the data arrives in the correct order if you just break it up into chunks and send them off one after the other

the most stable/reliable method i've come across is where you send a packet and don't send anymore until the client responds with some form of acknowledgement

server sends 4KB

client responses "ok"

server sends 4KB

client responses "ok"

~ repeat

problem with that is the speed is now completely dependant on latency and you can't take advantage of burst techniques however if you're sending to a low latency network you shouldn't have many issues

always good to have various approaches for various network types to get the most out of them :D

  • 0
  DDStriker said:
the tcp protocol will ensure the data arrives in the correct order if you just break it up into chunks and send them off one after the other

the most stable/reliable method i've come across is where you send a packet and don't send anymore until the client responds with some form of acknowledgement

server sends 4KB

client responses "ok"

server sends 4KB

client responses "ok"

~ repeat

TCP handles acknowledgment and throttling for you automatically. If you're wanting to send data to a remote endpoint as fast as possible, you might as well send it in chunks of 4096 bytes and let TCP handle those. If throttling occurs, eventually send() will not return for awhile on the local side until the congestion has been alleviated.

  • 0

Since you're already splitting your messages at this point, you may want to reduce their size. 4096 byte packets are definitely going to be fragmented (unless this is a specialized closed network of some sort, this would even fragment on the old token ring), and that could result in latency due to retransmission. 1456 would be a typical size that would not fragment on most contemporary ethernet networks (1500 MTU, 20 byte IP header, 24 byte TCP header).

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • Isn't the CPU used to calculate the parity for the RAID? If so, the combination of SSDs and 10GBe might make the CPU more important
    • yeah GSMA began working to enable end to end encryption between android and iphone last year and apparently a new standard was developed. apple has said that they would implement this in "future software updates" but i haven't heard anything since march, the time this was all reported on. shortly after, i read on forbes that the FBI suggests not sending texts between iphone and android because they're unencrypted. i use signal to chat with my wife but i'd rather just use messages tbh (she has an iphone), i'm not really a 3rd party guy haha
    • Well, I did not like the trailer for the project he went to work on also, but why do you think he should waste time with this… did you love the season 2? Maybe I am missing out after the crap I saw in first season ep1-3? I love the first last of us game… while not the BEST it was one of the games that I will remember for the EXPERIENCE it game me… last of us 2 was not on the same level at all and the show🤔 complete miss in my experience of the first few level
    • They're likely moving all resources to other things. Clearly Windows is not important to them.
    • Image Uploader 1.4.3 Build 5352 by Razvan Serea Image Uploader is a free and open-source program for Windows that that allows you to effortlessly upload images, screenshots, and various files to a wide array of hosting services. With its capability to capture selected screen areas, it promptly uploads content to image hosting services, while also offering the convenience of automatically copying the URL to your clipboard. Key Features of Image Uploader: Upload to Multiple Hosting Services Image Uploader supports uploading images and files to over 30 popular hosting services. Additionally, it can upload directly to your own FTP, SFTP, or WebDAV server. After upload, the tool automatically generates sharing codes in HTML, BBCode, and Markdown, with support for custom output templates tailored to your needs. Video Frame Grabbing and Screenshot Tools You can extract multiple frames from video files in a wide range of formats including AVI, MP4, MKV, WMV, and more. It supports both system-installed codecs and built-in ones. The extracted frames can be uploaded individually or compiled into a single mosaic image. It also includes screenshot capabilities for the full screen or selected regions, along with a simple image editor for annotations, highlights, and blurring. Advanced Integration and Usability Image Uploader supports drag-and-drop, clipboard monitoring, and can be accessed via Windows Explorer’s context menu. It also features URL shortening, multi-account support, reuploading, and the ability to upload images embedded in text while retaining formatting. The app is available in several languages, including English, Russian, Turkish, Korean, Arabic, and more. Image Uploader 1.4.3 Build 5352 changelog: New Features Screen Recording: Added two powerful capture methods: DirectX (Desktop Duplication API) FFmpeg-based recording Expanded Hosting Services: Added support for new file hosting platforms: TeleBox (linkbox.to) take-me-to.space ranoz.gg webshare.cz lobfile.com imgpx.com freeimghost.net radikal.cloud anonpic.org fotozavr.ru imgtr.ee thumbsnap.com 8upload.com filemail.com Others Video Uploads: Added Flickr.com support for video uploads Localization: New French translation added Context Menu: Added "File Information" option to video file context menus DPI Support: Improved support for: Screen DPI changes Mixed-DPI multi-monitor setups Improvements Disabled application window animations during screenshot/screen recording initiation Updated API and documentation Improved overall stability Bug Fixes Fixed network client error that could cause application crashes Resolved unauthorized startup registration issue Fixed upload functionality for pixeldrain.com Restored tray icon balloon notifications visibility Various minor bug fixes Download: Image Uploader 64-bit | Portable 64-bit | ~16.0 MB | (Open Source) Download: Image Uploader 32-bit | Portable 32-bit | ~15.0 MB Download: Image Uploader ARM64 | Portable ARM64 | ~11.0 MB Links: Image Uploader Home Page | Screenshot | GitHub Get alerted to all of our Software updates on Twitter at @NeowinSoftware
  • Recent Achievements

    • Reacting Well
      SteveJaye earned a badge
      Reacting Well
    • One Month Later
      MadMung0 earned a badge
      One Month Later
    • One Month Later
      Uranus_enjoyer earned a badge
      One Month Later
    • Week One Done
      Philsl earned a badge
      Week One Done
    • Week One Done
      Jaclidio hoy earned a badge
      Week One Done
  • Popular Contributors

    1. 1
      +primortal
      430
    2. 2
      ATLien_0
      156
    3. 3
      +FloatingFatMan
      149
    4. 4
      Nick H.
      64
    5. 5
      +thexfile
      62
  • Tell a friend

    Love Neowin? Tell a friend!