Need help making an "offline" copy of a website I'm a member of

[JayZJay]

Hi, everyone.

I'm a member of a private group, one that costs many thousands of dollars to join. The person that started the group and runs it, I'll call "Bob", isn't very technical at all. He hired someone to create a website for members. Each member has their own username and password. As far as I know, for at least the last few years Microsoft SharePoint Server has been used. I do not know any more specifics about what the website is running on.

"Bob" has always encouraged people to download things from the website here and there since content is always changing and in case anything ever happened to the server. Well, this website has many areas (called "Sites"), thousands of forum/blog type postings, and hundreds or thousands of links, files, and folders scattered all over the place. Yes, I could manually make local folders and download things, but that would be a nightmare and quite difficult.

So, all along over the last year or so, I've thought about just making an "offline" copy and figured I would use WinHTTrack, which I have known about for many years. I went to do it about a week ago, with the intention of backing up the site to a new Western Digital Caviar Black 2TB drive. I think the website is only maybe 100GB. Anyway, I am getting "Access Denied" and "Unauthorized" errors (if I look at the logs), which don't make any sense when I know I am using the correct username and password that I use to login. I am even copying and pasting from my password manager.

Here is the log file generated when telling WinHTTrack to copy http://www.WEBSITE.com :

HTTrack3.46+htsswf+htsjava launched on Wed, 17 Oct 2012 14:12:40 at http://USERNAME:PASS...www.WEBSITE.com +*.png +*.gif +*.jpg +*.css +*.js -

ad.doubleclick.net/* -mime:application/foobar

(winhttrack -WC2%Pns2u1%s%uN0%I0p7DaK0c2R3H0%kf2A25000%c1%f#f -F "Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)" -%F "<!-- Mirrored from %s%s by HTTrack Website

Copier/3.x [XR&CO'2010], %s -->" -%l "en, en, *" http://USERNAME:PASS...www.WEBSITE.com -O1 I:\WEBSITE_BACKUP_FOLDER +*.png +*.gif +*.jpg

+*.css +*.js -ad.doubleclick.net/* -mime:application/foobar )

Information, Warnings and Errors reported for this mirror:

note: the hts-log.txt file, and hts-cache folder, may contain sensitive information,

such as username/password authentication for websites mirrored in this project

do not share these files/folders if you want these information to remain private

14:12:42 Error: "Access denied" (401) at link USERNAME:[email protected]/ (from primary/primary)

14:12:42 Info: No data seems to have been transfered during this session! : restoring previous one!

Here is the log file generated when telling WinHTTrack to copy http://www.WEBSITE.com/default.aspx :

HTTrack3.46+htsswf+htsjava launched on Wed, 17 Oct 2012 14:16:53 at http://USERNAME:PASS...om/default.aspx +*.png +*.gif +*.jpg +*.css +*.js

-ad.doubleclick.net/* -mime:application/foobar

(winhttrack -WC2%Pns2u1%s%uN0%I0p7DaK0c2R3H0%kf2A25000%c1%f#f -F "Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)" -%F "<!-- Mirrored from %s%s by HTTrack Website

Copier/3.x [XR&CO'2010], %s -->" -%l "en, en, *" http://USERNAME:PASS...om/default.aspx -O1 I:\WEBSITE_BACKUP_FOLDER +*.png

+*.gif +*.jpg +*.css +*.js -ad.doubleclick.net/* -mime:application/foobar )

Information, Warnings and Errors reported for this mirror:

note: the hts-log.txt file, and hts-cache folder, may contain sensitive information,

such as username/password authentication for websites mirrored in this project

do not share these files/folders if you want these information to remain private

14:16:56 Error: "Unauthorized" (401) at link USERNAME:[email protected]/default.aspx (from primary/primary)

14:16:56 Info: No data seems to have been transfered during this session! : restoring previous one!

Attached you will find an a screen-shot of the error I get via a pop-up window (for WEBSITE.com or WEBSITE.com/default.aspx) as well as screen-shots showing the various tabs of the Preferences.

So, I was wondering if there is a problem in general with using WinHTTrack with SharePoint websites? Does anyone have any experience making offline copies of SharePoint websites? Any suggestions? Other software that someone has used for SharePoint websites?

Thanks in advance for any feedback and/or help...

-JayZJay

[moloko]

Why not just do it at a server level backup. I never had any issues with the standard options in httrack in getting websites. Or just FTP into the server and DL the entire website from there. If it uses PHP I believe you still will need a local server to run the website.

[JayZJay]

The server is not physically here, nor do I as only a member have any type of admin access to do a server level backup. As far as I know or can tell, there is not any type of ftp access available to anyone. If there is, I do not know the URL and "Bob", being a control freak, is not going to provide that information to any of us members.

[Midge]

try ignoring robots.txt

[Kraftman]

What type of authentication does Sharepoint have? Forms-based or basic access HTTP authentication?

[+BudMan MVC]

well from this

"14:12:42 Error: "Access denied" (401) at link USERNAME:[email protected]/ (from primary/primary)"

Looks to me that you can not use that form of url to auth to the site? If sharepoint its prob using ntml auth, and they prob have basic auth off so you can not do that sort of url to auth.

your never going to download anything if site requires auth to access. I would verify that you can use that url your using just in your browser address bar to access the site. If not then do a bit of google for how to use ntml auth with httrack, the old school was was to use a proxy with httrack that would do the ntml auth for you and then all the connections from httrack would use the local proxy you were running on the same machine. Have not play with httrack in years and years - maybe they support direct ntml auth now?

[n_K]

Login using your browser and get the cookies. Now use these cookies to download what you want to download inside of the downloader program (Sharepoint uses ASP sessions from what I remember)

[Xoligy]

"Bob" runs a website where he encourages people to download, yes sounds very legit...

[JayZJay]

"Bob" runs a website where he encourages people to download, yes sounds very legit...

Try not assuming. It is a research group. We study alternative health, banking, law, trusts, estates, and other topics. The only thing on the site is PDFs, Word docs, discussion on various topics, private presentations recorded from group meetings, webinars, etc. Those are the things on the site and there is a ton of it going back years. "Bob" encourages us to download and keep copies of stuff because 1) the site constantly changes, 2) there are times when you are in a law library or other place without Internet access and cannot access the site, and 3) "Bob" isn't very technical and he loses things, deletes stuff, etc -- so having our own backup copies is certainly a good idea just in case a problem arises.

[DrJohnSmitherson]

Try not assuming. It is a research group. We study alternative health, banking, law, trusts, estates, and other topics. The only thing on the site is PDFs, Word docs, discussion on various topics, private presentations recorded from group meetings, webinars, etc. Those are the things on the site and there is a ton of it going back years. "Bob" encourages us to download and keep copies of stuff because 1) the site constantly changes, 2) there are times when you are in a law library or other place without Internet access and cannot access the site, and 3) "Bob" isn't very technical and he loses things, deletes stuff, etc -- so having our own backup copies is certainly a good idea just in case a problem arises.

i dont know how to help, but this sounds really cool. Why can't we know who bob is, or this website?

[Xoligy]

Try not assuming. It is a research group. We study alternative health, banking, law, trusts, estates, and other topics. The only thing on the site is PDFs, Word docs, discussion on various topics, private presentations recorded from group meetings, webinars, etc. Those are the things on the site and there is a ton of it going back years. "Bob" encourages us to download and keep copies of stuff because 1) the site constantly changes, 2) there are times when you are in a law library or other place without Internet access and cannot access the site, and 3) "Bob" isn't very technical and he loses things, deletes stuff, etc -- so having our own backup copies is certainly a good idea just in case a problem arises.

All i have gotten from this whole thread is bob is a complete and utter idiot that runs a website that people seem to pay alot of money to join. If you are paying him money to join this secret website and he has no clue on what he's doing then he should hire someone to take care of the tech side of things including backups to another hard drive/server (after all you say your paying alot of money why can he not have more than one server?). If there are so many files like you claim then the best thing would be to get limited ftp access to the downloads directory (can access and download that directory and thats it).

In all honesty you would be better talking to this bob and telling him to hire someone for back up reasons, as for the offline content maybe he does not want that because the information could end up being on the internet for free instead of people paying him to gain access. As you said things change and people leave things laying about which soon gets onto the internet,.

Anyway im out of this thread, i personally think what you are doing is wrong you are a member of a site and nothing more.