PatchLog

A little more then a year ago I wrote a post presenting three different methods to transfer a mysql database. The third method suggested in that post was copying the mysql database files directly from one server ( or location ) to another. This involved locking the tables with a read lock or even shutting down the mysql before the actual copy.

For my work I usually have a main system and a development system and each system have their own database so there is a need from time to time to copy the main database over the dev database but because the database is very big ( a lot of tables and some with a large size ) and not every table is changed I like using rsync to transfer only the changes especially when I'm transferring to remote locations because it saves bandwidth and is faster.

In this case where I found the problem I actually use the same mysql server to hold both main and dev database but I still use rsync to transfer just because it still is faster then cp.

So here is what I do: I stop the mysql server , run rsync -av /var/lib/mysql/main_db/ /var/lib/mysql/dev_db/ , the differences are transferred, I start the mysql server look at the dev_db and Boom! some of the tables are corrupt. The main database was fully functional before shutting down mysql , no tables were corrupt or needed a repair, and still don't need starting up mysql.

Maybe something even more interesting is that it's very likely noone was using any of the databases before mysql was shut down.

It seems that after the transfer I just have to "repair table table_name" for some of the tables in dev_db and the repair statements returns some message saying that the number of rows has changed. Of course since I don't want to go over each db and see if it actually needs a repair I chose to just repair all of them and I wrote a script for that.So I just run the bellow script after each transfer, just to make sure everything is ok:

 
< ?php
$db_host="localhost";$db_user="root";$db_pass="";
$dbname="dev_db";
$db_link=mysql_connect($db_host,$db_user,$db_pass);
mysql_query("use $dbname");
$res=mysql_query("show tables");
if(!$res)die(mysql_error());
while($arr=mysql_fetch_array($res)){
	echo "repairing table: ${arr[0]}\n";
	$res2=mysql_query("repair table ${arr[0]}");
	while($arr2=mysql_fetch_array($res2)){
		print_r($arr2);
	}
}
?>
 

This script should also show you the messages returned by the repair statements. So you can see if there really was a problem. Make sure you set the correct db connection parameters and database name before you try it.

When observing this problem I was using rsync version 2.6.9 and mysql 5.0.44 on gentoo x86_64. The problem doesn't come up on every transfer and not on all tables. Could this be a problem with rsync or mysql?

I'm thinking that if this is a problem with rsync then... wow...that is a big problem. I was relying on rsync for transferring a lot of stuff ... what if it didn't transfer something and who knows what else it didn't transfer?.

If it's a mysql problem, maybe mysql doesn't update the row counts on the tables correctly before shutting down so the files were actually correctly transferred just not correctly stored by mysql. If the row count is the only problem here then it's not such a bit problem. I'm hopping this is the case ...

I wonder if this problem would show up when using something like cp for the transfer. If that would happen then it's clearly a mysql problem but I cannot test with cp at the moment as my db is very large and that means I would have to keep the tables locked too much which is just not an option on a system that was just "promoted" to production.

I'll come back with another post once I find out more about this problem but until then just make sure to check your tables after the transfer if you are using something like rsync to transfer the files directly.

This one is a quick release just like the previous one that fixes just one thing.

All previous versions had this problem that the urls were not urlencoded so those urls that contained special characters like those with an accent or diacritics were invalid and of course google would show an error on those sitemaps.

Version 0.9 makes escapes those urls so now those of you with such special characters in the urls can finally enjoy this module.

It seems like the modules is getting closer to version 1.0 . If you have any suggestion about some feature you would like in 1.0 or you found some other bug that needs fixed, don't hesitate to let me know about it.

download v 0.9 from the module's page

It seems my last version of the Xml Sitemaps module for pligg didn't really fixed the date format problem with the generated sitemaps.

Back when this module was created google had less strict rules about the date format in the lastmod section. My module generated a date and time string with this format YYYY-mm-ddTHH:MM:SS and it was ok but now it's only valid if it also contains the timezone offset in this format +/-HH:MM or if the string doesn't contain the time anymore.

So here is another update on this module that also contains the timezone offset so the sitemap is considered valid by google.

Download here

I have just upgraded to wordpress 2.5.1. My upgrade routine worked well without any problem.

The new release seems to bring a few bug fixes to  some annoying issues and a security fix as well as some performance enhancements.

Unfortunately the automatic plugin updater still fails at times and doesn't save the ftp password and the media uploader still renames .tar.gz files in a stupid way.

I know vim gurus would criticize me for using arrows in vim's insert mode but it's really hard to give them up.

I have this problem when I connect from my linux box to a MacOSX or FreeBSD box over ssh. I find it one of the most annoying things when using vim. When you are in insert mode and hit one of the arrows to move around, instead of the expected action vim will just print A, B, C or D on a new line. This makes vim practically useless.

So either you are very careful and always exit the insert mode before you move or fix the keys.

I think it's hard to always remember to get out of insert mode and it's one extra operation you have to do that I find useless not to mention you will probably have to enter insert again a few seconds after that.

So here's the fix for the arrow keys.  Edit vimrc either the global vimrc ( I'm using vim from macports so my vimrc is /opt/local/share/vim/vimrc ) or ~/.vimrc like this:

$ vim ~/.vimrc
set t_ku= (now type Ctrl-V and press cursor up)
set t_kd= (now type Ctrl-V and press cursor down)
set t_kr= (now type Ctrl-V and press cursor right)
set t_kr= (now type Ctrl-V and press cursor left)

This solution was stolen from vim tips wiki. I posted it here to avoid looking for it again if I need it. It's the second time I am hit by this problem and every time I had to search through a few pages with solutions that didn't work for me

This is a quick fix for a bug introduced in a previous version because I tried to make it compatible with php4 date().

This bug may have made your sitemap invalid because the lastmod date contained the timezone between the date and time.

This version also brings a new feature that could be useful for larger sites.

The cache

I noticed that on a site with over 20000 links it may take a lot of time to generate the sitemap index and sitemap files and will put some significant load on the server if google, yahoo or ask will try to access the sitemap every few minutes or hours ( depending on your site's posting/pinging frequency ), so I thought it would be nice to have some kind of cache.

The module will save generated sitemap index and sitemaps in pligg's cache directory ( this means it needs to be writable by the user running the webserver ) and if the cache has not expired yet the module will serve the sitemaps from the cache instead of trying to regenerate every time.

You can set the expire time ( TTL ) and the module will regenerate the sitemap if TTL seconds have passed since the last time it was modified.

You don't need to set any cron job to generate the sitemap files. It will only generate the sitemap if someone/something(google,yahoo,ask) tries to access it.

Another modification related to the caching system is that the site will only ping services if the cache has expired so make sure you set your cache's TTL accordingly.

Upgrade:

To upgrade to this version just download and unzip in your module's directory and then to pligg admin -> module management , disable and uninstall the module and then reinstall it so that you can see the new options ( "Use Cache" and "Cache TTL")

Download from module's own page

After spending about an hour trying to figure out why I can't mount disks from /etc/fstab in MacOSX 10.4 I finally figured it out.

MacOSX 10.4 ( and maybe lower ) doesn't use fstab anymore. Well not just fstab. The stupidest thing about this is that fstab is referenced in every man page related to mount but nowhere it says it does NOT work anymore. I guess apple didn't want to invest in updating the man pages, no one is reading them anyway, everyone is using the gui tool, etc...

MacOSX holds all this information about mounts, users, groups, passwords, services, protocols, hosts and other stuff ( that uses to be in plain text files on Unix ) in it's NetInfo database. The good news is that you can import the information from the Unix standard format plain text files into the NetInfo database and you have an easy way to do this over the command line.

To mount one or more external disks at boot you will simply import the fstab entries just by piping them to the niload command:

echo "echo "/dev/disk1s10 none hfs rw,auto"  > /etc/fstab
cat /etc/fstab | niload -m fstab /   	# load fstab in netinfo

At the next boot /dev/disk1s10 will be mounted to /Volumes/disk1s10's label.
The weird thing about this is that you can't just remove /etc/fstab after you import in NetInfo. If you do this will not work anymore. If you just put the info in fstab but not inport it with niload it will not work. You need to both have a valid /etc/fstab and import it in order for this to work.

Update: There is be a new version of this module. Click here to get it.

Last year I have released a pligg module that creates XML sitemaps for better SEO. The module was tested and working on pligg 0.9.7. Since that time pligg was updated two times and is now at 0.9.9.

In pligg 0.9.8 they removed the clear_cache function that the my module's install code was using to clear the configuration cache. Because of this it was impossible to install the module.

The quick fix was just to edit xml_sitemaps_install.php and remove a line (#5) that called clear_cache() and then you could install it without any problems. This workaround was already known and discussed in the comments
Someone in the pligg community released a new version of this module that is basically just my module with that clear_cache function call removed and another 1 bug that was also discussed in the comments and was easy to fix. They bumped the version number to 0.5 which in my opinion was really not necessary since it was such a small fix.
They have also added a .htaccess file to the archive, again not necessary since my initial post already had the details ( .htaccess code ) about how to set it up and the module was also showing that in the configuration section.
One other thing I didn't like about their release was that they didn't make the module available for download without having to register on their forum.

So here I release a new version ( 0.6 ) of this module with more bugs fixed and now the module will have a page of it's own so that it will be easier to track new versions .

What changed:

• use safe category names
• generate links according to URLMethod configuration
• modified use of date() function to work on php4

This module was tested on pligg 0.9.9.

Go to xml sitemap pligg module page for download

here's a quick tip about how you can change the hostname on your MacOSX and keep it changed after a reboot:

scutil --set HostName your.server.name

Of course you have to type that as root.

It seems there is also possible to set in it /etc/hostconfig but that file on it's own will not work. You have to load it in netinfo somehow but I don't know how to do that.

I think I like the /etc/hostconfig approach more then the scutil (even if it has to be loaded in netinfo ) because it's somehow closer to how most unix/linux systems do it.
Anyone knows how this can be done ?

This post will show you how to enable or disable permissions on a HFS+ volume and also how to check if a volume had the permissions enabled or not.

By default when you create a HFS+ filesystem it will have the permissions/owners disabled. I'm using a macmini for backing up files from a linux system and if I want the backup to be fully consistent with the source I need filesystem permissions.

Another problem is that using rsync between a permission based and a non permission based filesystem is useless because all files will always be out of sync.

Check permissions

vsdbutil  -c  /Volumes/YourVolume

it will "Permissions on '/Volumes/YourVolume' are enabled ( or disabled ) of just "No entry found for '/Volumes/YourVolume'" which means the permissions are disabled.

Enable Permissions

vsdbutil  -a  /Volumes/YourVolume

Disable Permissions

Since permissions are disabled by default I don't see many wanting to disable permissions but here's how you can do it anyway:

vsdbutil  -d  /Volumes/YourVolume

I guess this is really useful only when you want to distribute a dmg image which currently has permissions enabled.
Distributing images with permissions enabled doesn't make much sense because the user that will open the image may have a different id then the user id that owns the files in the image so he may not be able to access the files inside the image.