Jump to content



Photo

Need with with Imgur program

java imgur download

  • Please log in to reply
19 replies to this topic

#1 thatguyandrew1992

thatguyandrew1992

    Neowinian Senior

  • Tech Issues Solved: 4
  • Joined: 22-January 09

Posted 04 October 2012 - 19:10

Hey guys,
I wanted to work on a new project just for fun, so I thought a program that downloaded imgur galleries would be cool. I am working on it, but the program will run through duplicates after 51 images usually.

I've looked at all i can and can't find the problem. My guess is that it's the server resetting me. I wanted a challenge so i tried to get past it. I tried creating a new url connection each time, I also created a seperate class that did the downloading and creating a new object of that class everytime but it didn't work.

Thoughts?

Source
http://shayconcepts....der/imgurDL.zip


#2 Aethec

Aethec

    Neowinian Senior

  • Joined: 02-May 10

Posted 04 October 2012 - 20:30

You are parsing the HTML page. Don't do that. Use imgur's API instead - from the documentation, you can just append ".xml" to any gallery page and it returns the gallery as an XML file.

In general, don't parse HTML pages unless there is no other way. It's not easy and it can change any day, making your program useless.

#3 Andre S.

Andre S.

    Asik

  • Tech Issues Solved: 11
  • Joined: 26-October 05

Posted 05 October 2012 - 01:51

In general, don't parse HTML pages unless there is no other way.

Especially not with regex.

#4 OP thatguyandrew1992

thatguyandrew1992

    Neowinian Senior

  • Tech Issues Solved: 4
  • Joined: 22-January 09

Posted 05 October 2012 - 04:14

You are parsing the HTML page. Don't do that. Use imgur's API instead - from the documentation, you can just append ".xml" to any gallery page and it returns the gallery as an XML file.

In general, don't parse HTML pages unless there is no other way. It's not easy and it can change any day, making your program useless.


Thanks! I'll take a look.

EDIT: it seems it only returns 56 images though :/

Especially not with regex.

I learned about regex and it has worked really well for me! I'll take a look at your link and find a better way to parse HTML. Thanks!

#5 Andre S.

Andre S.

    Asik

  • Tech Issues Solved: 11
  • Joined: 26-October 05

Posted 05 October 2012 - 06:05

I learned about regex and it has worked really well for me! I'll take a look at your link and find a better way to parse HTML. Thanks!

If you really need to parse HTML, you should use a proper HTML library; for Java there is HtmlUnit and Jsoup at least. Better would be to not parse any HTML and use imgur's API as Aethec suggested.

#6 OP thatguyandrew1992

thatguyandrew1992

    Neowinian Senior

  • Tech Issues Solved: 4
  • Joined: 22-January 09

Posted 05 October 2012 - 14:00

If you really need to parse HTML, you should use a proper HTML library; for Java there is HtmlUnit and Jsoup at least. Better would be to not parse any HTML and use imgur's API as Aethec suggested.

But when using the API, I append the .xml but it doesn't show ALL the images in the gallery

#7 Aethec

Aethec

    Neowinian Senior

  • Joined: 02-May 10

Posted 05 October 2012 - 14:05

To get other pages, append /page/XXX to an existing gallery URL. XXX is the page number. Don't forget the .xml, too.

#8 OP thatguyandrew1992

thatguyandrew1992

    Neowinian Senior

  • Tech Issues Solved: 4
  • Joined: 22-January 09

Posted 05 October 2012 - 19:59

To get other pages, append /page/XXX to an existing gallery URL. XXX is the page number. Don't forget the .xml, too.

Thank you so much!! How did you find this? The API doc?

#9 ZakO

ZakO

    Neowinian

  • Tech Issues Solved: 2
  • Joined: 21-September 07
  • Location: Finland

Posted 05 October 2012 - 20:02

Thank you so much!! How did you find this? The API doc?

http://api.imgur.com/gallery

#10 OP thatguyandrew1992

thatguyandrew1992

    Neowinian Senior

  • Tech Issues Solved: 4
  • Joined: 22-January 09

Posted 05 October 2012 - 21:02

http://api.imgur.com/gallery

Thanks!

#11 OP thatguyandrew1992

thatguyandrew1992

    Neowinian Senior

  • Tech Issues Solved: 4
  • Joined: 22-January 09

Posted 08 October 2012 - 22:47

So Im using an XML parser and the /page/#.xml and it's working perfectly! Superb help! Thanks guys!

Now if a server disconnects me, how can I reestablish a connection, as if I had restarted my program.

#12 OP thatguyandrew1992

thatguyandrew1992

    Neowinian Senior

  • Tech Issues Solved: 4
  • Joined: 22-January 09

Posted 09 October 2012 - 01:54

Sometimes I get the java.net.ConnectException: Connection time out...

#13 ArialBlue

ArialBlue

    var lulz;

  • Joined: 24-June 10
  • Location: Democratic People's Republic of Korea
  • OS: Windows Master Race

Posted 09 October 2012 - 02:00

try {
// le magic
} catch (Exception e)
{
// re-establish connection
}

#14 Andre S.

Andre S.

    Asik

  • Tech Issues Solved: 11
  • Joined: 26-October 05

Posted 09 October 2012 - 03:34

try {
// le magic
} catch (Exception e)
{
// re-establish connection
}

catch (ConnectException e) ... always catch the specific exception you want, and never the base class of all exceptions.

#15 OP thatguyandrew1992

thatguyandrew1992

    Neowinian Senior

  • Tech Issues Solved: 4
  • Joined: 22-January 09

Posted 09 October 2012 - 16:37

If I catch that ConnectException, how do I reetasblish the connection?