wuulsoftware

wuul · Site Admin Joined: 25 Oct 2007 Posts: 69

I have started writing a new application - it is designed to monitor websites and RSS feeds for particular keywords.

I thought this would be useful, as I have been following the Gary McKinnon case (the hacker who is fighting extradition to the US on charges of hacking into the military networks) and an decision is due any day by the house of Lords - so I was constantly checking the BBC news site to see if there were any updates. Eventually I wondered why I didn't have this automated - a simple program that would scan the website regularly for the words "Gary McKinnon" and alert me when these were found.

There are a number of programs that do similar things, but I couldn't find any decent free ones that did exactly what I wanted so I started writing one.

It's a long way from being finished, but it does basically work now - details are on http://members.lycos.co.uk/wuul/webu/readme.html

Please feel free to try it and let me know how you get on - there are bound to be a few bugs at this stage and I have a lot of things I want to do with it, but any feedback would be appreciated.

Two points to note: at present, the program only saves all the config when it exits, so if you spend ages setting up various sites, feeds and keywords, you may want to restart it to ensure your config is saved. Also, please read carefully the notes on the format of the keywords, it won't work unless you set these up correctly.

The program can monitor RSS feeds as well as standard websites.

wuul · Site Admin Joined: 25 Oct 2007 Posts: 69

Simple keyword monitoring isn't always sufficient for informing you about news articles that you are interested in, so the latest version (0.5) of the program can learn over time about the kind of articles that you like, and will start to show you similar articles even if they don't contain any of your keywords. It works as follows: each time Webu shows you a web page it will offer you the change to rate the page on a scale of "totally irrelevant" to "very relevant". As you indicate your preferences for or against particular types of content, Webu will start to learn about the kind of stories that interest you and will start to show you other articles it thinks you may find interesting. The more you provide feedback, the more it will learn about what interests you.

To allow you to kickstart the learning process I have put a "Learn" button on the RSS/Atom form - when you click this it will show you all articles from the selected feed and ask you to rate them. This can take a while, but the more feedback you provide, the more it will learn. If it starts to show too many or too few articles you can adjust the "Show articles with relevance" factor - this defaults to 70, and means that an article must have a relevance factor of 70 (when webu shows you a page it will display the "relevance factor" on the titlebar). The higher this number, the fewer articles will be displayed.

To start you off I have added some feeds from the BBC website, so to start the program learning, select one of the feeds and click "Learn" (you can delete any feeds you don't like or add ones from other websites). The current version of the program doesn't save the configuration until it exits or you click the "Save" button, so make sure you save the configuration after running the learning process otherwise it won't save your preferences (I will fix this in a future release).

I would appreciate any feedback about how well the learning algorithm is working - there are some parameters that can be tuned to optimise the process, but as I have only written this recently I am not sure exactly what these should be. Please feel free to let me know if you think the program works well or could be improved.

wuul · Site Admin Joined: 25 Oct 2007 Posts: 69

I have uploaded version 0.6, this fixes a whole bunch of things I wasn't happy with in 0.5. The learning algorithm is improved, I have tidied up the GUI and it now automatically saves config at regular intervals and also after doing a learning session.

The program has started showing up on various download sites and many of them are showing 0.5, so if you have downloaded v0.5 please get the latest version from my web site.

W.