Web Architecture, Web Production, Web Marketing

I wanted to get a quick count of messages being sent to and from all my domains in plesk.   I wrote this script to get the mail domains from the plesk database.  Then I loop through them and grep a regex against the maillog file.  Since I’m actually using zgrep to look decompress yesterday’s maillog file at the same time I get the counts, it’s a little processor intensive.  It takes about 10 seconds to run through 30 domains in a 5 meg gzipped file.  If I were to gunzip the file first and just grep it, this would go much quicker.  But I’m fine with a 10 second run time and the bonus of not having to trash a decompressed maillog file when I’m done.

Here’s the bash script I’ve saved as mailcount.sh:

#!/bin/sh

MYSQLPASS=`cat /etc/psa/.psa.shadow`

for DOMAIN in `mysql -uadmin -p$MYSQLPASS -e "select distinct domains.name from mail inner join domains on mail.dom_id=domains.id" -B -N psa`

do
echo $DOMAIN `zgrep -c -E "(to|from)=.+@$DOMAIN" /usr/local/psa/var/log/maillog.processed.1.gz`
done

Here’s a version that copies and gunzip’s the maillog before doing any processing.

#!/bin/sh

MYSQLPASS=`cat /etc/psa/.psa.shadow`

# copy the log file before decompress
cp /usr/local/psa/var/log/maillog.processed.1.gz /usr/local/psa/var/log/maillog.processed.1-working.gz &&

# decompress the log working file
gunzip /usr/local/psa/var/log/maillog.processed.1-working.gz

for DOMAIN in `mysql -uadmin -p$MYSQLPASS -e "select distinct domains.name from mail inner join domains on mail.dom_id=domains.id" -B -N psa`
do
echo $DOMAIN `grep -c -E "(to|from)=.+@$DOMAIN" /usr/local/psa/var/log/maillog.processed.1-working`
done

# remove the working log file
rm -f /usr/local/psa/var/log/maillog.processed.1-working &&
echo "file removed"
exit;

I’m finally getting around to cleaning out dreamweaver LCK files from a large website. We have one developer that uses dreamweaver so these files are useless. So we’ve got these LCK files all over the place in this site. I’m just sick of looking at them.

Enter a shell one liner …

find . -name '*.LCK' -type f -print0 | xargs -0 /bin/rm -f

That command recursively looks in and under the directory I’m in for any files with the LCK extension and removes them.

Here’s a shell script you can save that will prompt you for an extension name to remove.

#!/bin/sh

echo "Enter Extension"
read filepattern
echo "Files matching *.$filepattern will be removed."
echo "Is this correct? y|n"
read confirmation

if  [ $confirmation = y ]; then
	find $PWD -name "*.$filepattern" -type f -print0 | xargs -0 /bin/rm -f
else
	echo "quitting"
	exit;
fi

Or if you don’t want any confirmation of the file extensions you’re about to delete.

#!/bin/sh

echo "Enter Extension"
read filepattern
find $PWD -name "*.$filepattern" -type f -print0 | xargs -0 /bin/rm -f

After installing Plesk 8 on a RedHat 4 box I’ve needed to install Plesk 8 on Redhat 5.  RedHat 5 uses mod_jrun22.  Well it’s Apache 2.2 that uses mod_jrun22, so any version of Linux running Apache 2.2 will use this connector.

Since Plesk doesn’t support CF8 yet, it still writes out its apache directives looking for mod_jrun20.  This needs to be modified to get per domain support for CF8 and Apache 2.2

Warning this will void your warranty!

I grepped the plesk install directory looking for mentions of jrun.  I found a binary file that matched: /usr/local/psa/admin/sbin/websrvmng

Looking at this file, on line 228 there’s a mention of mod_jrun20 in plain text.  This is the part of the websrvmng that writes out the http directives.  After making a backup of this file I took a chance at making an edit.  I simply changed mod_jrun20 to mod_jrun22 and saved the file.

Sure enough, plesk is happy.  Now when checking the ColdFusion box in a domain setup I see <IfModule mod_jrun22.c> in the httpd.include file for that domain rather than <IfModule mod_jrun20.c>.

Be careful here, I’m sure this change would not survive a plesk update to websrvmng.  After each plesk update double check this file to make sure it still points to mod_jrun22.

When backing up mysql databases via mysqldump you can either specify a database name to backup or use the –all-databases switch to dump all databases in one big file. I’d like to backup all my databases but I’d like them in separate files so they’re easier to manage and I can do a quick restore of a single db if needed.

I wrote the following bash script to help out.

#!/bin/sh
# all db separate files

for customerdb in `mysql -uadmin -pweb7A8u -e "show databases" -B -N`
do
        mysqldump -uadmin -pweb7A8u $customerdb > /var/www/vhosts/mysql/$customerdb.sql
done

Or on my plesk server, I’ve got a table in plesk’s psa database that lists out all the databases created within plesk. A modification to this script reads the table names in the psa database table and makes backups of all those.

#!/bin/sh

for customerdb in `mysql -uadmin -pweb7A8u -e "select name from data_bases" -B -N psa`
do
        mysqldump -uadmin -pweb7A8u --protocol=tcp --port=3307 $customerdb > /storage/mysqlbackup/$customerdb.sql
done

By default the Plesk control panel does not work with ColdFusion 8 yet. It sounds like official support will be available in Q1 of ‘08. I couldn’t wait that long. I’ve been able to get Plesk 8 to work with ColdFusion 8 to a degree.

I assume you’re familiar with how to install both separately so I’ll just be going over how to get both to play together.

In my world the Plesk control panel is just used simply used so that staff in our small software company can manage websites and not need to use a shell. No one else even has access to Plesk. Because of this, separation and security are not primary concerns. Also, I have no need to let anyone or myself manage CF DSN’s from within Plesk, being able to do this in CF Admin is fine.

So what I’ve done is enable the ColdFusion checkbox used when setting up a website so that CF code can be used on a particular website. Any ColdFusion management capabilities of Plesk are not enabled.

[root@yadda]# /usr/local/psa/bin/cfsetting -s /opt/coldfusion8
I/O warning : failed to load external entity "/opt/coldfusion8/runtime/servers/default/SERVER-INF/jrun.xml"
Config not parsed successfully.

Running that cfsetting command with the -s switch and the cf root from the commandline showed me an error plesk was running into. I found jrun.xml in a different place.

/opt/coldfusion8/runtime/servers/coldfusion/SERVER-INF/jrun.xml

Since I think jrun.xml should be pretty much the same from CF7 to CF8, I’m going to just try pointing Plesk to the right place with a simlink.

cd /opt/coldfusion8/runtime/servers/ && ln -s coldfusion default

Now I run the cfsetting command again.

/usr/local/psa/bin/cfsetting -s /opt/coldfusion8

Now I notice when starting up Apache that it’s trying to startup mod_jrun twice. In my ColdFusion install I selected to build a mod_jrun connector to hook into Apache during the install. Turns out that this conflicts with Plesk. It’s a simple fix. Just fire up your httpd.conf file and look for all instances of cfm or jrun. You should see these entries toward the bottom of the file. Just comment out the whole block of stuff by putting a hash sign in from of the lines. Or you could always delete this stuff if you want.

For each website that you enable ColdFusion support for you should see this code in the httpd.include file for the site.

JRunConfig Verbose false
JRunConfig Apialloc false
JRunConfig Ignoresuffixmap false
JRunConfig Serverstore /opt/coldfusion8/runtime/lib/wsconfig/psa/jrunserver.store
JRunConfig Bootstrap 127.0.0.1:51800
AddHandler jrun-handler .jsp .jws .cfm .cfml .cfc

If you need to do a little troubleshooting, just make sure that you don’t see code like this in any of the global config files for apache. As well make sure your jrunserver.store file exists. And make sure you’re loading the mod_jrun module somewhere. Plesk should drop this into zz010_psa_httpd.conf in the conf.d folder. Make sure that exists: LoadModule jrun_module /opt/coldfusion8/runtime/lib/wsconfig/psa/mod_jrun20.so. And lastly, double check your jrun.xml file you should see a mention of jrun broadcasting on port 51800 … you should see that in the jrun.servlet.jrpp.JRunProxyService block of the jrun.xml file. It’s possible that jrun is running on a different port, if that’s the case you’ll need to change it to 51800. Or somehow find out how to change the Plesk default.

If you have any troubles, post a comment here and I’ll see if I can help.

Subversion comes prebuilt with a system of “hooks”. Basically for most actions you would perform with a subversion repository you can also hook another program up to that action. For example you can tell subversion to also update a site locally on each commit.

The first step in this process is to make sure that your website as it exists on your dev server has been checked out from subversion and that permissions and ownership is set as you want it (that part’s up to you and outside the scope of this post). Let’s say I was working on the google site that was held in my repository in the “site” directory, I’d do something like:

svn checkout file:///home/google/repository/site/trunk/ webroot/

Now that a fresh checkout exists in the webroot, we’ll set things up to do an automatic subversion “update” on each commit.

From the subversion FAQ’s I found this handy chunk of C code.

#include
#include
#include
int main(void)
{
  execl("/usr/bin/svn", "svn", "update", "/path/to/site/webroot/",
        (const char *) NULL);
  return(EXIT_FAILURE);
}

The FAQ basically says that you take this chunk of code, add your own webroot path and compile it. If you’re like me and you compile C code once every year, or perhaps never, here’s the specifics on a Linux box. I use the gcc compiler and save the file as “update_site.c”

gcc update_site.c -o update_site

Not too complicated.

Now to preserve file permissions on your site properly, make sure that the owner of the website can execute this program from the svn hook but changing the mode of the compiled program to add the s switch.

chmod +s update_site

The last step is to tell subversion that it should run this update_site program after each commit. In your repository navigate to the hooks directory on the file system. Open up the file called post-commit.tmpl. There’s a couple example lines at the bottom of mine that execute python scripts for mailing and logging the commit, I just comment those out by putting a # in from of the lines and then on the last line of the file add the full path to the new update_site script. Like: /full/path/to/update_site

When you’re done, copy or move this post-commit.tmpl template to just post-commit and make sure that the webserver user can execute it.

With this process, you’re pretending that the webroot is the working directory for a subversion user. This means that the .svn directories normally in your working directory will now be present in the webroot as well. In case you have any passwords mixed in with you’re code, disable browsing of those directories in Apache. Thankfully this isn’t often an issue with ColdFusion datasources but it still doesn’t hurt to be safe.

Add the following to your apache httpd.conf file:

# Disallow browsing of Subversion working copy administrative dirs.

    Order deny,allow
    Deny from all

One of my responsibilities at Aslan is the Linux SysAdmin. I’ve decided to replace my job with several tiny robots and take on the title of Linux Architect instead. One of the main things to manage is a subversion repository for each project we work on. Rather than keep documentation around for me to manually follow a set of steps for each new repo, I’m using the Bourne shell to write a script that will backup the existing site, create a repository, and make a login and password to use when accessing the repository though a local svn client.

Part 2 will show how to have a site automatically publish itself to the dev server on each commit.

For now, here’s some code for Part 1:

#!/bin/sh
echo "creating svn repository and moving files ..."
tar zcvf htdocs.tgz htdocs &&
svnadmin create repository &&
mkdir site &&
mkdir site/trunk &&
mv htdocs/* site/trunk/ &&
rm -rf htdocs &&
ln -s site/trunk htdocs &&
svn import site file://$PWD/repository/site -m "initial import" &&


echo "Who owns this directory?"
read owner
echo using ... $owner
chown -R $owner:$owner * &&
chown -R apache repository &&


# make apache hook for subversion
APACHESVN=/etc/httpd/conf.d/subv-$owner.conf
touch $APACHESVN
echo "" >> $APACHESVN
echo "DAV svn" >> $APACHESVN
echo "SVNPathAuthz off" >> $APACHESVN
echo "SVNPath $PWD/repository" >> $APACHESVN
echo "SSLRequireSSL" >> $APACHESVN
echo "AuthType Basic" >> $APACHESVN
echo "AuthName \"$owner\"" >> $APACHESVN
echo "AuthUserFile $PWD/subv-auth" >> $APACHESVN
echo "Require valid-user" >> $APACHESVN
echo "" >> $APACHESVN


echo "Supply a username for accessing the svn repository"
read svnuser
echo "thanks, how about a password"
read svnpass
echo "I'm using $svnuser and $svnpass"



htpasswd -bc $PWD/svn-auth $svnuser $svnpass &&
/etc/init.d/httpd stop &&
/etc/init.d/httpd start &&
echo "all done"


The RoundCube webmail project does look pretty slick. The developers look to have a solid roadmap lined up and a good, conservative set of features coming up. I’d signed up for the dev list and am considering helping out in some capacity.

Update 6/12/07: I’ve been using Roundcube inhouse for the past couple of weeks now. It’s awesome. I think this is the best webmail client out there. Others have more features but not more functionality. Roundcube does exactly what I need it to do.