Monthly Archives: August 2007

XML Sitemaps for Pligg

Web, , , , , , , , , , ,

Update: There is be a new version of this module. Click here to get it.

I created a module that generates XML Sitemaps for Pligg ( the well known open source cms used for creating sites similar to digg.com ).

The module generates a sitemap index and sitemaps with all the stories in the database dynamically so nothing is stored on disk and you don't have to set a cron job to generate it.

The sitemaps are updated automatically when a new story is submitted. Because of the structure of the sitemap index and because it contains "lastmod" info, the search engines should only download the latest entries in the index so you shouldn't worry about the module putting too much load on your system.

There is also a "ping" function that will announce google, yahoo and ask.com every time a new story is submitted so that they know they have to download the sitemap. The ping function required a little patching to pligg source code to add some hooks ( only if you use 9.6, 9.7 should already have those hooks ). Here is the diff file in case you use pligg 0.9.6 : pligg submit hooks diff

The module was only tested on pligg 0.9.6, I haven't upgraded to 0.9.7 yet, so if you try this on 0.9.7 let me know how it works, any feedback is appreciated.

Download:

You can download Xml_Sitemaps module from here: xml_sitemaps-0.1.tar.gz and in case you want to verify it here is the md5 sum and the sha256 sum

the code is released under the same license as pligg, so feel free to use it, modify and share.

Installation:

This is pretty straight forward, you have to install this like any other pligg module, just put the .tar.gz file in the modules, un-archive it then go into pligg admin and activate it. If you use pligg 0.9.6 and you want to be able to ping the search engines don't forget to apply the submit hooks patch .

Configuration:

After installation you should be able to access the sitemap index like this : http://yourdomain.com/module.php?module=xml_sitemaps_show_sitemap or if you want the sitemap to look friendly ( btw ask.com will only accept a friendly sitemap ending in .xml ) , you just have to go into Admin->Configuration->XmlSitemaps and enable "Sitemap Friendly URL", and if you do that then you have to put the following lines in your .htaccess somewhere before the line "##### URL Method 2 ("Clean" URLs) Begin #####" :

  1. RewriteRule ^sitemapindex.xml module.php?module=xml_sitemaps_show_sitemap [L]

  2. RewriteRule ^sitemap-([a-zA-Z0-9]+).xml module.php?module=xml_sitemaps_show_sitemap&i=$1 [L]

Here is how the index looks on a site with sitemap friendly urls enabled: http://sapa.ro/sitemapindex.xml

There are other configuration options in there, you can set the maximum number of stories to put in a sitemap, and you can chose whether to ping any of the three search engines supported. You can also set your yahoo.com key in there if you want to ping yahoo.

That's it! Happy Sitemapping! and as always ... let me know how it works in the comments.

Blog action day for environment

blogging, , , , ,

Let's see what happens if a lot of bloggers write about the same thing in one day? Would it matter ? Would you make a difference. Blog action day, on October 15'th will be the day when thousands of blogs will write about the one thing.

One day per year, one topic, thousands of bloggers contributing to making a better world. This year's topic is "Environment".

I have signed up my blog and hopefully I'll post something related to Unix, Linux, Patches and Hacks to help create a better environment.

How to make linux feel like windows

nix, , , ,

Anyone missing windows ? Not me, but sometimes is good to go back to windows just to see how it was in the old days when nothing worked well and you would have a crash every few minutes or reinstall once a week.

Do that and you'll realize how privileged you are for using a Unix like os and be thankful to those that develop and contribute to it.

Don't want to reinstall it ? Well here's a little script inspired by the author of random lines ( a program that reads lines from a file or stdin, randomizes them and then outputs them ) that will make your linux system feel like windows:

  1. span style="color: #ff0000;">'{print $1}'

I recommend running the script as root and only if you understand completely what it does. You may not see immediate results, have confidence and let it run for some time like an hour or more. If you want to see results faster you can replace 3600 with 360 or even 36 or less.

You will need random lines ( rl ) installed if you want this script to run or you can modify it and replace every occurrence of "rl --count=" with "shuf -n " if you have shuf (should be there if you have coreutils and most distros have it )

Feel free to let me know how your "back to windows experience" affected you 🙂 ...

Sidebar decluttering

blogging, , ,

A few days ago I came across a post at Problogger.net where Darren Rowse suggested bloggers should de clutter the sidebar of their blogs.

I looked at my sidebar and I saw what he meant by clutter. Darren could have used my sidebar as an example of a cluttered sidebar.

The sidebar was full of widgets that were not really useful to me or to my visitors. Here is what I had and removed:

  • Most popular Posts: I removed this because I didn't think it was really useful for my visitors. A lot of them come from a search engine and look for something specific. I don't think they are interested in what's popular on my blog, but I might be wrong about it and I might add it back in the future, maybe on the footer.
  • Tag cloud: this one was getting too big. I occupied too much space and the most popular tags were already the words in categories to it was kind of redundant. I might add this on a page of it's own
  • Top commentators: I removed this because it looks bad to see a list of top commentators where #1 has only 3 comments.
  • Technorati fave this blog: removed because I already have a technorati button at the bottom of each post
  • Google Referrals banners: The banners increased the load times and were not making money. I may add them back as links like this :
    because I see some potential in this but I'll have to get more traffic before I start using it again
  • Text link adds affiliate banner: In two months of using this I got about 80 clicks on it and no conversions. But the text link ads is reported to be one of the best affiliate program. They give you $25 for signing up an active publisher or advertiser
  • Reviewme banner: removed because I received no offer for a review since I started using it a few months ago. I guess no one wants me to review a patch 🙂
  • WordPress meta: does anyone care about this ?
  • DoFollow: This doesn't really have anything to do with the sidebar but I removed this plugin because it was pretty much useless. I don't think anyone that commented on my posts did it because of the dofollow plugin. The only ones that commented because of dofollow were those that commented on the post about dofollow. The idea to reward your commentators is good but if there isn't anyone ( or very few ) to reward the plugin is useless. Also I think the Top commentators plugin might be better for this.

Some thoughts about what I should do, but I'm not sure about this yet:

  • move archives and maybe categories in the footer
  • add most popular posts in the footer
  • remove google ads, I this I can afford losing about $2/month 🙂
  • remove kontera
  • create a top navigation bar and move the links to pages on that bar

What do you think I should do about the blog ? Any suggestions are welcome in the comments.

No browser supporting socks5 authentication?

Web, , , , ,

If you're trying to use a socks server with Internet Explorer , Firefox, Opera or Safari everything will work just fine, except for authentication.

From my point of view this is a big problem. Who in the world would leave such a proxy server unprotected? Yeah of course you can always limit access to a proxy server based on ip address, but in some cases ( see NAT ) this is just not going to work.

Internet explorer supports only the socks4 protocol which doesn't even support full password authentication ( only username and it defaults to the current logged in username ) .

Firefox supports socks5 but no authentication mechanism so supporting socks5 is pretty much useless. I think I saw some ticket in bugzilla about this but no one managed to commit a fix yet.

Opera doesn't even support socks protocol but I thought I should mention all major browsers 🙂

Safari supports SOCKS5 and even allows you to set a username and password to access the SOCKS server but it does not use them.

I tried Konqueror, but I was unable to specify the Socks server, I guess this is because it was not compiled with a socks library.  Has anyone had any success with Konqueror and Socks ?

Solaris in qemu

nix, ,

For some time I wanted to try Solaris mainly because all the buzz around it, going open source then with the buzz around ZFS, but I never got the chance to do it until today when I had to make a perl script work on solaris 9 ( this is a outdated but that's what the client had installed ).

The perl script was using the Net-SSH-Perl module and the module had some requirements that had to be installed.

The problem is that the system where this script had to run had no C compiler so I had to install solaris9 in qemu, compile the requirements and send them all in one package.

Geting Solaris 9

So I went to solaris 9 download page and downloaded the DVD version ( split over 5 zip files suffixed with a-e ) , decompressed each file, then cat them all in one big file respecting the alphabetical order of the names and I got my big .iso file.

Qemu Scripts

When I use qemu ( and I use it a lot ) I have these 3-4 scripts that I use everywhere:

1 ) the start script : start.sh . I use this script to basically boot the guest system

  1. span style="color: #ff0000;">"c""$1"

I run this like "./start.sh" or "./start.sh d" if I want to boot from cd instead of the disk. DU_sol9.iso contains some drivers i will discuss later in this post. macaddr is optional but I set it so that my dhcp server knows what ip to provide for this system.

2) restart_dhcp.sh : this script restarts my dhcp server so that the server will start answering requests coming in from the tap devices created by qemu. this way the guest systems can get their ips from my dhcp server.

  1.  

Sleeps a bit before restarting the dhcp server because qemu does not create the interface right at the start and we have to make sure the interface ( tap11 ) exists before we restart the dhcp server.

3) qemu-ifup.sh and if I want two interfaces in the guest system then I will have qemu-if2up.sh. This script just sets the ips on the host side of the tunnel between the host and the guest system

  1.  

I can't remember exactly why I wanted to use ip tools, I think ifconfig $1 172.16.10.2 would work just as well.

Installing Solaris 9

Installing solaris went pretty smooth in qemu. I just modified the start.sh script to use the solaris9.iso for -cdrom and ran "./start.sh d" and a few F2 keys hit and a reboot later the system was up and running. Everything seemed to be fine but I had no network access to the guest os and the CDE ( Common Desktop environment ) locks from time to time .

I am unable to determine the problem with CDE but so I just killed the X server and then I ran /usr/openwin/bin/xdm which gave me an X session in a minimal window manager ( Twm ) but that was enough for me to install Sun Studio and get a C compiler.

There was no network access because solaris 9 does not have the drivers required for the network cards that qemu emulates ( NE2000 rtl 8139 and a few others ). I found this page providing a community network drivers for those card. the page also contains instructions about how to add the drivers and at the official sun documentation site you can find out more about configuring network interfaces in solaris

When trying to install Sun Studio, qemu just crashed at about %30 with "qemu: fatal: Trying to execute code outside RAM or ROM". Luckily the C compiler was in the first %30 of studio's files and I managed to uses to for compiling the requirements ( modules ) for my perl script, but when I was finished, the client thought he should mention that his server was SPARC not x86 ( what I tried ) 🙂 . Why didn't I think of that, I guess most servers out there running solars are sparc....

Next I'm going to install the SPARC version of solaris 9 in qemu and I'll let you know how it goes in another post...

Repair a MySQL table

databases, , ,

I'm running mysql 5.0.23 on a FreeBSD server. I have several databases there and a few phpbb forums.

I noticed the tables used for searching the forums ( phpbb_search_wordlist and phpbb_search_wordmatch ) crash quite a lot lately from various reasons but mainly because of hardware problems ( like lack of power 🙂 ). Nothing unusual here so far. When I notice this I go into mysql and do a repair like :

  1.  

But this time I got this answer: " Table is already up to date". So MySQL client tells me the table is fine but in phpbb when I try to search something I get this error message: "SQL Error : 145 Table './simscripts_phpbb/phpbb_search_wordlist' is marked as crashed and should be repaired". I try to read the table from the mysql client and I get a similar message.

Going through the MySQL documentation I find some extra parameters i can pass to repair table. And use_frm seems to be the one that fixes the problem. This parameter should recreate the indexes by looking at the .frm file ( the structure definition of the table )

  1.  

did the job and search in phpbb is back online.

Unfortunately it seem that the table was so badly damaged that no rows could be recovered so the repair did was to recreate the database structure. Good thing I had a backup!

OR maybe it was just because my table was created on an older mysql version and as the documentation says:

Caution

Do not use USE_FRM if your table was created by a different version of the MySQL server than the one you are currently running. Doing so risks the loss of all rows in the table.

I don't know if this is a bug only in the version I run, but I think MySQL should really look at indexes and if they need to be recreated it should just do it automatically or atleast tell you the table is not ok instead of lying like that.

Update:

If you have shell access to your server with root or mysql user permissions you can go in the mysql data directory ( usually /var/lib/mysql or /var/db/mysql on freebsd ) go into your broken database directory and use myisamchk to repair the table without the risk of losing all the rows in it :

  1.