• Sign in to Neowin Faster!

    Create an account on Neowin to contribute and support the site.

  • 0
Sign in to follow this  

[C++] Recursive S/FTP Sync - Guidance Needed

Question

BinaryData    777

I've got a project I'm working on, and I need some advice on how to code something. Let me explain the goal.

 

I have 3 servers that my team uploads files too every day. I have a backup server that I run. It's tedious as all hell to download from each one, every day. I don't know what is new, and what isn't. The 3 Servers run CentOS, while my server runs Windows. Here's what I need it to do.

 

Every 24 hours, I want it to check for new files, if there are new files, it initiates a download to my back up server. There is a specific time frame that the server will be able to download. I have a relatively fast internet, however the server is only running a dual core with 15TB of Storage in a Pooled Environment. The download can run from 12AM PST to 3PM PST, a 15 hour window. At most that would be uploaded/downloaded would be 5GB or less, depending on whats being updated. My server at home will be running sftp, so there isn't that problem. I could make the files download in ftps, and only accept connections from my 3 servers.

 

I've never worked on anything in Linux, it's always been Windows. I'm open to suggestions for other software for syncing, but I'd like to build this on my own.

Share this post


Link to post
Share on other sites

6 answers to this question

Recommended Posts

  • 0
Circaflex    3,564

@BudMan, @Jaredor @sc302 would probably know the best and most efficient way. Tagging them as I don't know if they browse this subforum often.

Share this post


Link to post
Share on other sites
  • 0
BinaryData    777
17 hours ago, Circaflex said:

@BudMan, @Jaredor @sc302 would probably know the best and most efficient way. Tagging them as I don't know if they browse this subforum often.

I try and not stick my hand in that particular honey pot too often. I know their more than willing to lend their expertise, but I consult them way too much, haha.

 

I know there are excellent FTP Libraries. I'm wondering if the same libraries will work in Linux too.

Share this post


Link to post
Share on other sites
  • 0
simplezz    863

If you're talking about transferring files via SSH's FTP, then it wouldn't be too difficult. In fact you could probably do it with a simple script (sftp is supplied with OpenSSH) rather than a fully fledged C++ program. Either way, you'd likely want to run a scheduled Cron job to perform the task.

 

There are plenty of libraries to plug into if you want to do it in code.

Share this post


Link to post
Share on other sites
  • 0
BinaryData    777
5 minutes ago, simplezz said:

If you're talking about transferring files via SSH's FTP, then it wouldn't be too difficult. In fact you could probably do it with a simple script (sftp is supplied with OpenSSH) rather than a fully fledged C++ program. Either way, you'd likely want to run a scheduled Cron job to perform the task.

 

There are plenty of libraries to plug into if you want to do it in code.

I was looking at libssh2, actually. I don't have a good understanding of ssh, so I thought this might be a good opportunity to learn about it. I also don't know much about Linux and how things are coded there. Which interests me quite a bit. I came up with a plan on how it should work, coding it will be a bit more difficult. I'm not sure if you can code a Linux Application in Visual Studio and then compile and export it to Windows. I was looking into communication, i.e.

 

Server 1 - Begins the Transfers to Server X, Server 2 - Waits until Server 1 completes, and Server 3 - Waits until Server 2 is complete. So on and so forth. I think that might be the hardest part.

Share this post


Link to post
Share on other sites
  • 0
simplezz    863
1 minute ago, BinaryData said:

I was looking at libssh2, actually. I don't have a good understanding of ssh, so I thought this might be a good opportunity to learn about it. I also don't know much about Linux and how things are coded there. Which interests me quite a bit. I came up with a plan on how it should work, coding it will be a bit more difficult. I'm not sure if you can code a Linux Application in Visual Studio and then compile and export it to Windows. I was looking into communication, i.e.

 

Server 1 - Begins the Transfers to Server X, Server 2 - Waits until Server 1 completes, and Server 3 - Waits until Server 2 is complete. So on and so forth. I think that might be the hardest part.

libssh is a good start. curl is another possibility, though I hear it's a pita to work against when dealing with authentication etc.

 

I suppose it depends upon whether you want to do the whole end-to-end transfer yourself using a client/server architecture, or partially. That is to say, write a client on Linux that does the transfer to your main server that's running a standard sftp server. There's a lot of prebuilt software already out there.

 

Share this post


Link to post
Share on other sites
  • 0
BinaryData    777

I'm hoping to not have to code it, but it would be a good project. My colleague pointed out "What about data validation?" I haven't even thought of that portion until he mentioned it. After he mentioned that, I thought about it. I would generate the original crc after the files transferred and then send it, if it matches, we're good. If it doesn't match, request that file again.

 

I'll look into those prebuilt software, I was trying to find a possible way to automate it on CentOS without killing the system resources.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.