Crawling the Web - Made Easy with Python

Earlier today, I began development on Code Crawler. I was stuck between Java and Python. Both have great networking libraries that are extremely easy to use. Java, being my first language, is easier to me, but that's actually what attracted me to Python as the choice for this project - the learning potential.

Anyways, after only maybe an hour of development, I have to say that I am extremely impressed with Python. I had a crawler, reading a list of seed urls from a file, pulling content off the urls, and checking each resources content type to ensure it was allowed (to avoid images) in about 30 lines of code. Not too bad at all.

You can check up on the status of this project here.

Code Crawler

In the previous post, I introduced the project I would be working on - aside from codeXplode. I code named it Code Crawler.

It's (or will be) a web crawler design specifically for finding code. Whether it's php, C, Java, Python or any other programming language known to man, the crawler job will be to find it and let codeXplode know of it's existence on the web.

I'm doing this in an effort to create an easily searchable resource for us programmers. Whenever I can't quite come up with a way to do something or if I've just forgotten, I want to be able to just search for the bit of code that I need and modify it appropriately for my use.

Anyways, the codeXplode profile for the Code Crawler project is up now and can be found here.

First Project - Web Crawler

codeXplode is first, a place for hosting repositories. Without a good project to develop, it's hard to test all of codeXplode's functionality. So I've been thinking about what project I would like to work on. The project I've finally decided on is a web crawler.

First, though, what is a web crawler? It's a piece of software, usually used by search engines such as Google but are sometimes used as a standalone alternative to the big web based search engines.

Mine, though, will be aimed at pulling code. It's priority will be to find snippets of code, break them down, and index them in such a way that will allow for code to be easily searchable. This will ultimately be built into the codeXplode feature set at some point. It will allow for users to search for certain functions, instead of having to search for a specific file or project.

Hence, the project's name will be Code Crawler. I will post a link to the codeXplode page for the project once it is created.