PatchLog
Virtualmin custom dynip
Virtualmin has this nice feature where you can enable dynamic host and virtualhost updates so when your ip changes virtualmin updates the ips in the virtualhosts definitions. Without this feature your sites would not work anymore if your ip changes and you would have to update the ips manually.
The problem with this feature is that it only supports the DynDNS service and not everyone is using this service to update their host when their ip changes.
I'm using my own dynamic dns service and the good news is that this service is using the same protocol as DynDNS so virtualmin needs little change in order to support it.
All I had to do was add a way of specifying the hostname and port of the dynip update server.
Here's the patch for virtualmin ( tested with 3.75 and 3.76 ) :
Virtualmin custom DynIP patch-1.0 (3.74 KB)Faster wordpress page admin
In a recent post about wordpress I explained how you can create a lot of pages really fast. If that went well and you inserted a few thousand pages in your wordpress blog the page admin became useless. Displaying the list of pages would take 3 minutes for 7000 pages on my test server.
It seems this is not a new problem and there is a bug created in 2007 about it. Although it seems like there was a patch to fix this, the problem still exists in the 2.8.4 version.
Why is this so slow ?
Short story: because wordpress is trying to display and sort pages hierarchically .
At first I thought the problem was caused by the sql queries that fetched all the pages ( even though it doesn't display all of them on a page ) but that was not the case.
After profiling the code with xdebug and Kcachegrind I found there were a few parts of the code that were taking the longest time to complete.
The main problem is that wordpress is trying to find the children for all the pages in an inefficient way. There is this function get_page_children in wp-includes/post.php which was taking about 2 thirds of the total time to complete ( ~ 2 minutes on my example ).
The Solution
I rewrote that function to make it a lot more efficient. In my case it reduced the time from 2 minute to 1-2 seconds but on other page hierarchy it might take more, the worst case being when every page is the parent of another page. The diff is here : Fast page_get_children-0.1 (1.05 KB)
The second problem is that wordpress updates the page cache every time you list pages. This was taking almost 1 minute to complete. I'm not sure if it's the right thing to just remove that call to update_page_cache in wp-includes/post.php get_pages , but doing that made the page admin load in about 15 seconds.
Now this might still be annoying but it's way better then 3 minutes. Hopefully at least the new get_page_children function will b included in the next wordpress release... maybe you can help promote this ticket by giving it a positive vote although I'm not sure if those votes actually have any influence.
Qmail Big concurrency
Wanna send messages faster with your qmail server? Everyone will tell you to increase the remote concurrency. Till you find out that it can only go as high as 255. If you want more then that you have to apply the big concurrency patch
The Problem
Applying the patch and setting concurrency ( conf-spawn ) bigger then 509 will break the compilation. I was hoping to get at least 1000 on that new quad core 
Why 509? It seems the number depends on the maximum size of the FD_SET array used for "monitoring" ( using select ) the file descriptors ( connection sockets or opened files ) . This limit is set in FD_SETSIZE constant to 1024. In case you're wondering ... the formula that gets you from 1024 to 509 is (FD_SETSIZE-5)/2 ( from chkspawn.c )
Trying to define FD_SETSIZE to a higher value in conf-cc like this -DFD_SETSIZE=4096 doesn't work because FD_SETSIZE is redefined in sys/select.h like this
#define FD_SETSIZE __FD_SETSIZE
__FD_SETSIZE is defined somewhere in /usr/include/bits/types.h ( actually typesizes.h ) to 1024. Defining -D__FD_SETSIZE doesn't work either...I even tried both and still no luck.
The Solution
After hours of digging through mail archives and sites I found this mailing list post that really helped:
Re: fd_setsize
If you just want to get it working just download my patch qmail Big Concurrency fix-1.0 (898 bytes) , apply it ( after you apply the big concurrency patch ) , set conf-spawn to something big ( but less then 65000 ) and then you should be able to compile qmail.
If you want to know how it's done, read bellow...
It seems like the solution is to include bits/types.h, undefine __FD_SETSIZE and then define it to a higher value. The author of that post says that this is not a good idea (from the portability point of view, but I don't care about that ) since programs should never directly include bits/types.h ( true ) but the alternative is to modify that system file, again not a good idea since it will be overwritten by a possible update.
My first idea was to just use that code from the mailing list post into the select.h2 , since this is the file used to generate select.h and select.h is included in spawn.c and ckhspawn.c but this didn't work because spawn.c was including "select.h" after "sys/types.h" so even if select.h would define __FD_SETSIZE it would be useless since FD_SETSIZE ( this is the one that really matters ) would have been already defined in sys/types.h .
The solution I found at the time was to just move "select.h" at the top of the file and remove "sys/types.h" since it was already included from select.h but now I realized I could have just as well undefined and defined FD_SETSIZE too inside select.h
And that's the story about how I got to run 1000 concurrent connection in qmail.
The real problem
Now that we can have so much concurrency we hit another wall. Qmail, as most other MTAs, doesn't have any way of controlling the remote concurrency per destination domain.
At 1000 simultaneous connections it's very likely that it would create a few tens or hundreds of connections simultaneously to the same domain.
When this happens that domain will just ban your ip. So how do we fix this one?
PS: I have an answer but I want to see what you have for a solution 
so hit the comments...
Qmail TLS + SMTP Plugins on FreeBSD
Problem
One of the worst problems in qmail is that it accepts messages for non existent users and then sends back a bounce to the sender. This is the perfect setup to be exploited by a spammer. I'm using the qmail-tls port configured with vpopmail on FreeBSD.
Solution
There might be some patches that make qmail use the vpopmail command line tools directly to check if a user exists before accepting messages but I either didn't find them or thought that using the SMTP Plugins patch is more elegant and gives me more flexibility (It might be slower but this was not a high traffic server so I didn't care).
The SMTP plugins patch adds hooks at all STMP stages and the plugins can use those hooks to accept or reject a message based on the data in the smtp statements sent by the sender.
I used the Qmail-SPP - Vpopmail check user Script written by Werner Maier.
The only problem was that I had to patch the source code for qmail-tls with the qmail-spp patch and there were quite a few rejects because of the other patches applied by freebsd ports system.
I managed to adapt the code to make this patch work and here is how I did it...
First I built the original freebsd port:
cd /usr/ports/mail/qmail-tls make config # pick up the options you want here ... I needed tls, smtpd-auth,vpopmail and a few others make make install
This will also install the files and then after we apply the spp plugin we just copy the modified file ( qmail-smtpd )
Next step if to download the qmail-spp plugin into /usr/ports/mail/qmail-tls/work and apply it.
cd work tar -xzpf qmail-spp-0.42.tar.gz cd qmail-1.03 patch -p1 < ../qmail-spp-0.42/qmail-spp-smtpauth-tls-20060105.diff
This will give you some rejects but don't worry because you'll fix them with my next patch : Qmail-tls-spp-FreeBSD patch-0.1 (4.73 KB)
patch < qmail-tls-spp-freebsd.patch make /usr/local/etc/rc.d/svscan stop cp qmail-smtpd /var/qmail/bin /usr/local/etc/rc.d/svscan start
At this point qmail-smtpd should have smtp plugins working and we can set up the vpopmail user verifier script.
cd /var/qmail mkdir plugins fetch http://www.maiers.de/qmail/vpopmail_check_recipient.sh chown root:qmail vpopmail_check_recipient.sh chmod 755 vpopmail_check_recipient.sh
Now edit vpopmail_check_recipient.sh and make sure the correct paths to vpopmail directory ( /home/vpopmail on freebsd ) are set in the script.
To test it, connect to smtp and try to send a message to an nonexistent account, you should see a reject message instead of the usual "ok".
Adding new php syntax
This is a quick patch I did to php's source code to implement some special syntax. Basically I wanted to be able to define an array like this:
$a=[ 1,2,3,4];
Get the patch here: New array syntax for php-5.2.6-5.2.9 (997 bytes)
To get this behavior download php's source code, extract it put my patch in the source directory and do this:
patch -p1 < new_array_zend_syntax.patch cd Zend ./buildconf ./configure make cd .. ./configure make
( This will only compile but not install - in case you want that just type make install at the end )
And here's a test script :
$a=[ 'key' => 'value',1,2,3]; print_r($a);
Save this script as test.php then run sapi/cli/php test.php
This was tested on php 5.2.6 and 5.2.9. I think this is probably useless for most people but it was fun to write so why not share it
Weekend’s piece of shell magic
A few days ago I wrote a post about setting up squid as an anonymous proxy using multiple ips.
That setup would basically make squid listen for connections on certain ips and will create outgoing connections ( for fetching the requested page ) from the same ip that received the connection. The setup, as described in my previous post involves creating one acl and specifying one tcp_outgoing_address for each ip you want to use. Now that's ok if you have just a few ips or even a full class (because you can create one acl to match the whole class ) , but what do you do if you have multiple classes and in each class non consecutive ips? And I'm talking about many such ips not just 10, but 100 or more...
The shell Magic
Well if you have already defined the ips on your interface(s) and you want to use all those ips in squid then you can use this simple shell script that will parse the output of ifconfig and output the acls and tcp_outgoing_address directives for each ip.
for i in $(ifconfig -a|grep "inet addr"|cut -f2 -d:|cut -f1 -d' '|grep -v 127.0.0.1) ; do an=$(echo $i | tr . _) ; echo "acl in_$an myip $i" ; echo "tcp_outgoing_address $i in_$an" ; done
Run this script on the server where you want to install squid and it will output the acls and directives needed for using all the ips ( except 127.0.0.1 ) . then just copy and paste them into squid.conf.
Now shell magic is nice but if you want to get your hands dirty you might want to look into patching squid to do the same thing. With the squid outoing ip patch you will not have to write any acl or tcp_ougoing_address for it, but of course you'll have to do the "get source - patch - compile" stuff .. which some may find harder and others more fun
XML Sitemaps for Pligg
Update: There is be a new version of this module. Click here to get it.
I created a module that generates XML Sitemaps for Pligg ( the well known open source cms used for creating sites similar to digg.com ).
The module generates a sitemap index and sitemaps with all the stories in the database dynamically so nothing is stored on disk and you don't have to set a cron job to generate it.
The sitemaps are updated automatically when a new story is submitted. Because of the structure of the sitemap index and because it contains "lastmod" info, the search engines should only download the latest entries in the index so you shouldn't worry about the module putting too much load on your system.
There is also a "ping" function that will announce google, yahoo and ask.com every time a new story is submitted so that they know they have to download the sitemap. The ping function required a little patching to pligg source code to add some hooks ( only if you use 9.6, 9.7 should already have those hooks ). Here is the diff file in case you use pligg 0.9.6 : pligg submit hooks diff
The module was only tested on pligg 0.9.6, I haven't upgraded to 0.9.7 yet, so if you try this on 0.9.7 let me know how it works, any feedback is appreciated.
Download:
You can download Xml_Sitemaps module from here: xml_sitemaps-0.1.tar.gz and in case you want to verify it here is the md5 sum and the sha256 sum
the code is released under the same license as pligg, so feel free to use it, modify and share.
Installation:
This is pretty straight forward, you have to install this like any other pligg module, just put the .tar.gz file in the modules, un-archive it then go into pligg admin and activate it. If you use pligg 0.9.6 and you want to be able to ping the search engines don't forget to apply the submit hooks patch .
Configuration:
After installation you should be able to access the sitemap index like this : http://yourdomain.com/module.php?module=xml_sitemaps_show_sitemap or if you want the sitemap to look friendly ( btw ask.com will only accept a friendly sitemap ending in .xml ) , you just have to go into Admin->Configuration->XmlSitemaps and enable "Sitemap Friendly URL", and if you do that then you have to put the following lines in your .htaccess somewhere before the line "##### URL Method 2 ("Clean" URLs) Begin #####" :
RewriteRule ^sitemapindex.xml module.php?module=xml_sitemaps_show_sitemap [L] RewriteRule ^sitemap-([a-zA-Z0-9]+).xml module.php?module=xml_sitemaps_show_sitemap&i=$1 [L]
Here is how the index looks on a site with sitemap friendly urls enabled: http://sapa.ro/sitemapindex.xml
There are other configuration options in there, you can set the maximum number of stories to put in a sitemap, and you can chose whether to ping any of the three search engines supported. You can also set your yahoo.com key in there if you want to ping yahoo.
That's it! Happy Sitemapping! and as always ... let me know how it works in the comments.
Subscribe
Social Homes
PatchLog Sponsors
