Question How to start on a web bot ?

[Akutski]

Hey guys, I was looking for a way to create a web bot that basically copies a link from one site and copies it to another site. There are a lot of links so the bot has to go through the names systematically. It will also have to go to certain pages for each different links which I will hard code into the bot itself. Example, if this name is copied then place it in this page etc... As a noob to PhP I would like to know what these kind of bots are called so that I may search for tutorials for them on google. Or if anyone would be kind enough to link me with tutorials that may help me with this kind of coding, it would be greatly appreciated. Thanks.

[x1nixmzeng]

You mean you want to parse links from another site?

Assuming so, with a language like PHP you'd fetch the webpage (file_get_contents("url") or CURL - Google these) and run some Regex.

I'll bold all these terms you need to Google, but really, there is a lot of source out there.

[XTZGZoReX]

PHP can indeed easily do this task. I think it's the way to go. Although, you will need to find a way to keep it up (like a while 1 == 1 loop) or you will have to reload it everytime you want to parse new texts/urls.

[Akutski]

ok thanks for the bolded words, managed to find some really good sites about parcing. That was the word that really helped in my searches btw XD. Now just one last thing to get me going.... whats the easiest and fastest way to setup a PhP server to test out my scripts ? I know of apache but are there any simpler ones just to test my scripts on ?

[lut]

the fastest way would be either [Only registered and activated users can see links. ] or [Only registered and activated users can see links. ] they are both one install apache,php,mysql servers easy to use and easy to setup.

Wamp is probably the easiest of the two.

If you want to do this an not tie up your server with a infinite loop "while(1=1)" you should enter the urls to parse into a text file then you just have to check for the end of the file "while(!file.eof)".

A parser like this can get pretty hefty depending on how far you want to go into a site. A nice way to do something like this is create a new connection for each internal link you find on a page. run the parser on each connection then close it as soon as you reach the end of a page. Keep each internal link in an array for that site so you can check to see if you have visited each link and don't go over ones you have already done.

The other way is to only search certain pages for each site and list them in your text file.

If you don't want to use a text file you can store each url in a large array and use a for loop with sizeof(myarray)-1 to go through each link.

Enjoy

Feral

[XTZGZoReX]

Now that I think of it, he doesn't need to use a while loop.

He can just go through the sites, tell the bot to sleep for like 30 secs, then go through the sites again, and so on. This would be done with [Only registered and activated users can see links. ] and [Only registered and activated users can see links. ].

Update: What? I thought PHP had goto?

[lut]

The 'goto' for php isn't coming till version 6 and even then it's not exactly a goto but more of a break that can be pointed at an name anchor/target.


Either way a goto is just a make shift loop anyway, He could always use javascript to refresh the page after a site had been scanned but that would be pointless when a loop would work much better.

Lut

[XTZGZoReX]

Well, the point of a goto + a sleep would be that the application wouldn't be as CPU-intensive when sleep'ing as when using a while loop to constantly loop through things.

[lut]

yea but you can use a sleep in a while loop also.

[XTZGZoReX]

True enough.. True. I wonder why I didn't take of that.

Well, that is obviously the way to go; while() + sleep().