Thread: Question How to start on a web bot ?
View Single Post
Old 06-04-2008, 07:18 PM   #5
lut
Registered User
 
lut's Avatar
 
Last Online: 11-16-2009 11:51 AM
Join Date: Dec 2005
Location: Michigan
Posts: 1,633
Rep Power: 6
Rep Points: 31
lut is on a distinguished road
Feedback: (0)
Points: 54,476.63
Bank: 0.00
Total Points: 54,476.63
Re: How to start on a web bot ?

the fastest way would be either wamp or xampp they are both one install apache,php,mysql servers easy to use and easy to setup.

Wamp is probably the easiest of the two.

If you want to do this an not tie up your server with a infinite loop "while(1=1)" you should enter the urls to parse into a text file then you just have to check for the end of the file "while(!file.eof)".

A parser like this can get pretty hefty depending on how far you want to go into a site. A nice way to do something like this is create a new connection for each internal link you find on a page. run the parser on each connection then close it as soon as you reach the end of a page. Keep each internal link in an array for that site so you can check to see if you have visited each link and don't go over ones you have already done.

The other way is to only search certain pages for each site and list them in your text file.

If you don't want to use a text file you can store each url in a large array and use a for loop with sizeof(myarray)-1 to go through each link.

Enjoy

Feral
__________________
The things in life that are handed to you are not what holds any value.
Its the things that you have to work for that hold the greatest rewards.
lut is offline   Reply With Quote