• 0

[C++] Recursive S/FTP Sync - Guidance Needed


Question

I've got a project I'm working on, and I need some advice on how to code something. Let me explain the goal.

 

I have 3 servers that my team uploads files too every day. I have a backup server that I run. It's tedious as all hell to download from each one, every day. I don't know what is new, and what isn't. The 3 Servers run CentOS, while my server runs Windows. Here's what I need it to do.

 

Every 24 hours, I want it to check for new files, if there are new files, it initiates a download to my back up server. There is a specific time frame that the server will be able to download. I have a relatively fast internet, however the server is only running a dual core with 15TB of Storage in a Pooled Environment. The download can run from 12AM PST to 3PM PST, a 15 hour window. At most that would be uploaded/downloaded would be 5GB or less, depending on whats being updated. My server at home will be running sftp, so there isn't that problem. I could make the files download in ftps, and only accept connections from my 3 servers.

 

I've never worked on anything in Linux, it's always been Windows. I'm open to suggestions for other software for syncing, but I'd like to build this on my own.

6 answers to this question

Recommended Posts

  • 0
  On 19/03/2016 at 00:05, Circaflex said:

@BudMan, @Jaredor @sc302 would probably know the best and most efficient way. Tagging them as I don't know if they browse this subforum often.

Expand  

I try and not stick my hand in that particular honey pot too often. I know their more than willing to lend their expertise, but I consult them way too much, haha.

 

I know there are excellent FTP Libraries. I'm wondering if the same libraries will work in Linux too.

  • 0

If you're talking about transferring files via SSH's FTP, then it wouldn't be too difficult. In fact you could probably do it with a simple script (sftp is supplied with OpenSSH) rather than a fully fledged C++ program. Either way, you'd likely want to run a scheduled Cron job to perform the task.

 

There are plenty of libraries to plug into if you want to do it in code.

  • 0
  On 20/03/2016 at 01:51, simplezz said:

If you're talking about transferring files via SSH's FTP, then it wouldn't be too difficult. In fact you could probably do it with a simple script (sftp is supplied with OpenSSH) rather than a fully fledged C++ program. Either way, you'd likely want to run a scheduled Cron job to perform the task.

 

There are plenty of libraries to plug into if you want to do it in code.

Expand  

I was looking at libssh2, actually. I don't have a good understanding of ssh, so I thought this might be a good opportunity to learn about it. I also don't know much about Linux and how things are coded there. Which interests me quite a bit. I came up with a plan on how it should work, coding it will be a bit more difficult. I'm not sure if you can code a Linux Application in Visual Studio and then compile and export it to Windows. I was looking into communication, i.e.

 

Server 1 - Begins the Transfers to Server X, Server 2 - Waits until Server 1 completes, and Server 3 - Waits until Server 2 is complete. So on and so forth. I think that might be the hardest part.

  • 0
  On 20/03/2016 at 02:00, BinaryData said:

I was looking at libssh2, actually. I don't have a good understanding of ssh, so I thought this might be a good opportunity to learn about it. I also don't know much about Linux and how things are coded there. Which interests me quite a bit. I came up with a plan on how it should work, coding it will be a bit more difficult. I'm not sure if you can code a Linux Application in Visual Studio and then compile and export it to Windows. I was looking into communication, i.e.

 

Server 1 - Begins the Transfers to Server X, Server 2 - Waits until Server 1 completes, and Server 3 - Waits until Server 2 is complete. So on and so forth. I think that might be the hardest part.

Expand  

libssh is a good start. curl is another possibility, though I hear it's a pita to work against when dealing with authentication etc.

 

I suppose it depends upon whether you want to do the whole end-to-end transfer yourself using a client/server architecture, or partially. That is to say, write a client on Linux that does the transfer to your main server that's running a standard sftp server. There's a lot of prebuilt software already out there.

 

  • 0

I'm hoping to not have to code it, but it would be a good project. My colleague pointed out "What about data validation?" I haven't even thought of that portion until he mentioned it. After he mentioned that, I thought about it. I would generate the original crc after the files transferred and then send it, if it matches, we're good. If it doesn't match, request that file again.

 

I'll look into those prebuilt software, I was trying to find a possible way to automate it on CentOS without killing the system resources.

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • Both a game adaptation and it's on Amazon I'll set my hopes low
    • You shut your mouth you young fart, that was just a few years ago, lol.
    • Good effort, but unfortunately not very useful in the real world, since it doesn't support UEFI - which is a major drawback. More than that, most PCs these days have Secure Boot enabled by default, so unless you're tech savvy, the average joe is not going to disable Secure Boot - which means automated Windows > Linux migration programs like this will not work. And if you're tech savvy enough to know how to disable Secure Boot, then you're tech savvy enough to just flash a USB drive and install Linux the usual way - after all, the actual installation process is just point-and-click. So I don't see a use case here to be honest, especially considering the lack of UEFI support - especially when you consider that more and more Linux distros are completely dropping BIOS support (eg Fedora dropped it back in 2022). So this program is a bit too late to the party.
    • I agree with open rights group that the age check companies should be regulated but action does need to be taken imo, they may be called irresponsible parents but the fact is that many don't know how to set up blocks. Why should kids suffer because their parents don't know something? I think the UK has always been a bit like this. We don't say healthcare is your responsibility, we tax everyone and make people register with the NHS. We also don't expect everyone to brush their teeth to maintain dental health, we just put fluoride in their water assuming people aren't brushing enough, because some probably aren't or aren't doing it properly. In general, this way of doing things works.
  • Recent Achievements

    • One Month Later
      Falcon.ai earned a badge
      One Month Later
    • Week One Done
      Falcon.ai earned a badge
      Week One Done
    • Dedicated
      EYEREX earned a badge
      Dedicated
    • First Post
      Electronic Person earned a badge
      First Post
    • Week One Done
      CyberCeps666 earned a badge
      Week One Done
  • Popular Contributors

    1. 1
      +primortal
      629
    2. 2
      ATLien_0
      237
    3. 3
      Xenon
      165
    4. 4
      neufuse
      142
    5. 5
      +FloatingFatMan
      123
  • Tell a friend

    Love Neowin? Tell a friend!