Comments on: Writing a simple web crawler in Perl http://www.stratos.me/2009/05/writing-a-simple-web-crawler-in-perl/ Just writing what hits my mind! Thu, 13 Jan 2011 22:34:15 +0000 hourly 1 http://wordpress.org/?v=3.3 By: Alexandr Ciornii http://www.stratos.me/2009/05/writing-a-simple-web-crawler-in-perl/comment-page-1/#comment-6983 Alexandr Ciornii Sat, 17 Apr 2010 07:37:11 +0000 http://www.stratos.me/?p=1156#comment-6983 peer: If your program dies on http error, try Try::Tiny module to catch exceptions. peer: If your program dies on http error, try Try::Tiny module to catch exceptions.

]]> By: peer http://www.stratos.me/2009/05/writing-a-simple-web-crawler-in-perl/comment-page-1/#comment-6000 peer Mon, 15 Mar 2010 13:10:56 +0000 http://www.stratos.me/?p=1156#comment-6000 Hi, i'm writing a crawler in perl but i get http errors like 404 and this breaks my $mech->get() loop. Did you find a way to avoid http errors to exit your script ? regards, peer Hi,
i’m writing a crawler in perl but i get http errors like 404 and this breaks my $mech->get() loop. Did you find a way to avoid http errors to exit your script ?
regards,
peer

]]> By: davidak http://www.stratos.me/2009/05/writing-a-simple-web-crawler-in-perl/comment-page-1/#comment-2441 davidak Mon, 07 Dec 2009 15:05:11 +0000 http://www.stratos.me/?p=1156#comment-2441 Maybe you can modify this for a SQLite Database and a Download with the complete script? Maybe you can modify this for a SQLite Database and a Download with the complete script?

]]> By: desiNerd http://www.stratos.me/2009/05/writing-a-simple-web-crawler-in-perl/comment-page-1/#comment-2292 desiNerd Tue, 28 Jul 2009 09:58:07 +0000 http://www.stratos.me/?p=1156#comment-2292 @stratosg: thanks a lot...I've been looking for some simple/sample web spider to get started in Perl/Python...after spending some time what I found is that Perl is best suited for this kinda of job b'coz of the availability of almost anything though CPAN, which you might not get off-the-shelf in case of python. Now I'm clear its THE Perl thats best suited for writing a custom web spider. Your code is really nice and simple. Thanks a lot. @stratosg: thanks a lot…I’ve been looking for some simple/sample web spider to get started in Perl/Python…after spending some time what I found is that Perl is best suited for this kinda of job b’coz of the availability of almost anything though CPAN, which you might not get off-the-shelf in case of python. Now I’m clear its THE Perl thats best suited for writing a custom web spider. Your code is really nice and simple. Thanks a lot.

]]> By: stratosg http://www.stratos.me/2009/05/writing-a-simple-web-crawler-in-perl/comment-page-1/#comment-2216 stratosg Sat, 11 Jul 2009 16:21:28 +0000 http://www.stratos.me/?p=1156#comment-2216 @Abishek: Thanks for stopping by! @Abishek: Thanks for stopping by!

]]> By: Abhishek Sharma http://www.stratos.me/2009/05/writing-a-simple-web-crawler-in-perl/comment-page-1/#comment-2211 Abhishek Sharma Sat, 11 Jul 2009 13:10:20 +0000 http://www.stratos.me/?p=1156#comment-2211 @Dzieci: I agree with u... @stratosg: thanks 4 xplainin it so beautifully.... :) @Dzieci: I agree with u…
@stratosg: thanks 4 xplainin it so beautifully…. :)

]]> By: Gry Dla Dzieci http://www.stratos.me/2009/05/writing-a-simple-web-crawler-in-perl/comment-page-1/#comment-2071 Gry Dla Dzieci Thu, 04 Jun 2009 07:37:33 +0000 http://www.stratos.me/?p=1156#comment-2071 Getting perl on windows takes more time than installing linux and learning to use it. Seriously. Getting perl on windows takes more time than installing linux and learning to use it. Seriously.

]]> By: Nihar http://www.stratos.me/2009/05/writing-a-simple-web-crawler-in-perl/comment-page-1/#comment-2057 Nihar Tue, 02 Jun 2009 16:30:49 +0000 http://www.stratos.me/?p=1156#comment-2057 Great... Now i know why i get lot of spam. I took some part of bot code from donace and improved the bot to drop entrecards automatically everyday :) Great… Now i know why i get lot of spam.

I took some part of bot code from donace and improved the bot to drop entrecards automatically everyday :)

]]> By: stratosg http://www.stratos.me/2009/05/writing-a-simple-web-crawler-in-perl/comment-page-1/#comment-2022 stratosg Mon, 25 May 2009 18:39:29 +0000 http://www.stratos.me/?p=1156#comment-2022 @Donace: It can actually work with anything. The only thing we would need to change is the regular expression, (the one that has =~ ) other than that we can extract any kind of info... @Donace: It can actually work with anything. The only thing we would need to change is the regular expression, (the one that has =~ ) other than that we can extract any kind of info…

]]> By: Donace http://www.stratos.me/2009/05/writing-a-simple-web-crawler-in-perl/comment-page-1/#comment-2021 Donace Mon, 25 May 2009 18:20:05 +0000 http://www.stratos.me/?p=1156#comment-2021 let say instead of email addresses we search the source code for the comment section and then 'rel=follow'. Would that work in theory with some jerry rigging? let say instead of email addresses we search the source code for the comment section and then ‘rel=follow’. Would that work in theory with some jerry rigging?

]]>