Upgrading Server to Ubuntu Hardy Heron LTS from Dapper Drake LTS

My company hosts several rails applications. For the ones in high demand - we use mongrel_cluster with nginx. The only problem is ... we use apache for everything else. So we proxy pass requests into nginx from apache. That seemed so redundant that I decided to get rid of nginx and use mod_proxy_balancer instead.

On 6.06 this turned out to be much harder than it seemed. Essentially proxy_balancer.so did not exist in /usr/lib/apache2/modules .. I would have to compile it with apxs to get it into the installation. I found out that apache 2.2 came with proxy_balancer but when I tried to update the apache package ubuntu said it was already the newest version. I knew this meant I may have to consider an upgrade to the next LTS. Beyond using mod_proxy_balancer I had been trying to get "Phusion Passenger" to work for over a month. (I had to become very familiar with httpd.h and mod_passenger.c to get it to even compile). As of that point I still had no way of serving up rails applications from apache without using Proxy Pass.

It was late on Saturday night and I had the whole weekend to fix anything that broke so I felt pretty confident that everything should be fine.

I did the commands.

#sudo su
#aptitude update
#aptitude upgrade
#aptitude dist-upgrade
#aptitude install update-manager-core
do-release-upgrade

The upgrade was to be 287mb and take several hours. I pressed the "y" key and started browsing reddit on my laptop.
Through the installation I was asked what to do about configuration file conflicts between packages and my own custom versions. There were many times where I honestly didn't care because I didn't even know certain things were still installed. ldap.conf? hylafax.conf? I mean I played around with them .. thought I uninstalled those things. There were several obvious cases where I just kept my existing configs (my.cnf, apache2.cnf, php.ini etc)

The upgrade completed with an error message about /etc/fstab.pre-uuid already existing. I disregarded the error after googling the message for 10 minutes and finding nothing. Everything seemed fine.

I was delighted to finally get phusion passenger working and mod_balancer active. I took the liberty of installing about 10-15 packages I had experimented with but had no further use for. hylafax, bugzilla, otrs, auth-ldap-client etc... then I went home

The fallout

Later that night I went to show off some of the performance benchmarks to a friend and caught a page hanging. I pulled up my ssh terminal and tried to get in to see what was going on. I Couldn't get in! ! .

The next day I went on site to get on the server directly and see if I could get in. I entered every login and password I knew and it wouldn't even accept my username!. I followed instructions for manually resetting the passwords by going into recovery mode. I restarted the machine... none of the logins were checking out. I restarted again and looked at auth.log

Dec 21 06:36:55 www nscd: nss_ldap: reconnecting to LDAP server (sleeping 1 seconds)...
Dec 21 06:36:56 www nscd: nss_ldap: could not connect to any LDAP server as (null) - Can't contact LDAP server
Dec 21 06:36:56 www nscd: nss_ldap: failed to bind to LDAP server ldap://127.0.0.1: Can't contact LDAP server
Dec 21 06:36:56 www nscd: nss_ldap: could not search LDAP server - Server is unavailable
Dec 21 06:37:01 www CRON[9390]: PAM unable to dlopen(/lib/security/pam_ldap.so)
Dec 21 06:37:01 www CRON[9390]: PAM [error: /lib/security/pam_ldap.so: cannot open shared object file: No such file or directory]
Dec 21 06:37:01 www CRON[9390]: PAM adding faulty module: /lib/security/pam_ldap.so
Dec 21 06:37:01 www CRON[9390]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 21 06:37:01 www CRON[9392]: PAM unable to dlopen(/lib/security/pam_ldap.so)
Dec 21 06:37:01 www CRON[9392]: PAM [error: /lib/security/pam_ldap.so: cannot open shared object file: No such file or directory]

It hit me like a ton of bricks. At one point we had another IT guy here who wanted to use ActiveDirectory to manage the users. I hated windows and microsoft for a variety of reasons and wanted to prove to him that I could provide a much easier to use system using linux and phpldapadmin. I installed LDAP ... integrated it into the system and got it running - and we never used it. Now I've removed auth-ldap-client and the authentication client depends on ldap to check if the user is in ldap.

I looked at /etc/pam.d/ and /etc/nsswitch.conf .. where I found references to ldap in /etc .. I also found them in /etc/auth-client-config .. I read up on auth-client-config and found out that it can be used to control nsswitch and pam.d/* config files with profiles. I couldn't find a pre-ldap example so i modified the kerberos example and executed auth-client-config -a -p kerberos_example from the recovery prompt. And everything worked fine after that.

So please.. If you hear about a package, a project or the next biggest thing and you must install something on your machine. Consider doing it in a sandbox VM

Use "less" instead of more

Instead of doing "cat somefile | more" try using "less somefile". Less is a spinoff of more which supports vim-style find (press "/" and type what you need to find) and can read sections of file from disk as opposed to reading the whole file in then displaying it. It's also less typing.

Microsoft

Selling software isn't dead. Its just almost dead for many people. People saw that many great things were possible with computers. You could streamline small businesses, help groups collaborate, coordinate research and so on. I feel the world doesn't think its fair that Microsoft be the final word in software. People weren't comfortable having their aspirations run under one company's flag.
Software isn't dead, in the next few years you will see some of microsoft's largest markets turn their back on the giant.
Microsoft will be big in gaming. The Xbox 360 is amazing, well done.
Microsoft will be big in the business world.
The Microsoft "Personal Computer" will be a relic, a symbol of a dark time for all of human kind and the apogee of Microsoft's reign.
Home PCs will run on Apple operating systems (based on linux) or Linux operating systems.

New Google Gadgets in GMail

Ran across some new features in Google Apps/Google Mail today. There are now mini-managers for docs/calendar that can be embedded into your G-Mail.




I'm using the following plugins here.

• Right-side labels


• Right-side chat


• Navbar drag and drop


• Google Calendar gadget


• Google Docs gadget

Custom Fax Cover Sheets & Hylafax & AvantFax

There is nothing more difficult than making a custom cover sheet for Hylafax. Of all the programming languages I have ever glanced at, Postscript is by far the most cryptic. I spent about 6 hours the other day trying to follow the instructions on the official Hylafax tweaking page. . I designed my cover sheet in photoshop and left the fields blank. Exported it as PNG and imported it into TGIF. I used TGIF's text tool and printed as EPS at first and tried using the script to make a coversheet.

That just didn't work.

I followed the instructions on this page for manual preparation of the raw PS file for faxcover.

I spent 5 hours banging my head against the wall trying to figure out why when I replaced

(XXXX-from-company) SH


with


/from-company IS


and added the macros to the top of my PS file , nothing was working right (Fields were not replaced)

After many hours I had slipped up and accidentally left some fields in the format seen in the PS file (the format the fields were in the raw PS printoff from TGIF)

(XXXX-from-company) SH


Now why would that work? Apparently, All I needed to do was just place the fields with TGIF in that format and things would work. So I did that, tried again and everything worked perfectly. Now just to make that sheet the default cover sheet system wide as opposed to AvantFax's cover sheet. I copied the faxcover.ps file to /etc/hylafax and /var/spool/hylafax/etc and made sure their timestamps were synced (hylafax won't start otherwise).

I left the -C argument out of the sendfax command to use the system template and voila! Failure!. The resulting fax still had AvantFax's coversheet. I remembered that during installation there was a note about replacing hylafax's default coversheet. I looked into the AvantFax installer source code


(debian-install.sh from AvantFax 3.1.2)
mv $HYLADIR/bin/faxcover $HYLADIR/bin/faxcover.old
ln -s $INSTDIR/includes/faxcover.php $HYLADIR/bin/faxcover


Shocked! AvantFax actually replaces the system wide faxcover program with a CLI PHP script! Their PHP script is made to mimic faxcover (but they conveniently forgot to update the man page for faxcover). I looked into the code, and its set to use AvantFax's cover sheet in the avantfax installation directory ($INSTDIR/includes/faxcover.ps). I replaced that file and Voila, My custom coversheet was working!

Lazy Rails Development with Shell Function Shortcuts

I love screencasts, and on every screencast I see that texmate editor I'm so jealous of. I work at a startup so it'll be a while before we'll dish the cash out for a mac, as much as I'd like one, its not in the cards right now.

Once of the things I like the most about that editor is the ability to seemlessly jump between different files in your rails app. I find myself doing certain commands _constantly.

One tip is to create shell functions for your most commonly used commands

ruby script/server #rss
mongrel_rails start -d -e development -p 3000 #mrd_start
mongrel_rails stop #mrd_stop
vim app/controller/some_controller.rb #rvim c some
vim app/models/post.rb #rvim m post
vim app/helper/posts_helper.rb #rvim h posts
vim app/views/posts/_form.rhtml #rvim v posts/_form.rhtml
mongrel_rails cluster::stop #mrc stop
mongrel_rails cluster::start #mrc start


Here is the code (feedback appreciated)

function rss
{
command ruby script/server
}
function mrd_start
{
command mongrel_rails start -d -e development -p 3000
}
function mrd_stop
{
command mongrel_rails stop
}
function mrd_restart
{
command mongrel_rails stop&&mongrel_rails start -d -e development -p 3000
}
function mrc_start
{
command mongrel_rails cluster::start
}

function mrc_stop
{
command mongrel_rails cluster::stop
}
function mrc_restart
{
command mongrel_rails cluster::stop&&mongrel_rails cluster::start
}

function rvim
{
case "$1" in
'c')
command vim app/controllers/$2_controller.rb
;;
'v')
command vim app/views/$2
;;
'm')
command vim app/models/$2.rb
;;
'h')
command app/helpers/$2_helper.rb

esac
}





Hope this helps!

Recording Transport with FreePBX

If you have ever used freepbx in production, you know that there are many cases where its not adequate as is for the needs of a call-center. Chances are you got nowhere by going to get help in #freepbx on freenode. You probably went to #asterisk on freenode where people soon figured out you were using freepbx and then began ignoring you (They hate that). Well, I've had to conquer some difficult requests for features with our PBX and often I've had to make custom dialplans for them. I'll post a few of them.

We were testing out a new outbound sales campaign that involved qualifying a lead and sending it to a third party. We needed to monitor the third party and make recordings, but they were on a traditional PBX. I thought of using DISA (Direct Inward System Access) but could not find any tutorials or articles on how to setup recording with DISA. I had to do four things:

1. Figure out how recording was done on the agent extensions
2. Create a new outbound dial plan pattern and import the recording snippet into the new plan.
3. Create some method of communicating the call to a webserver with a CRM on it.
4. Create a method of automatically transporting the call to the webserver upon completion

I figured that creating a new dialplan would be the easiest, since DISA provides a dialtone from inside the system.


grep -R recording /etc/asterisk


I noticed several macro entries in extensions.conf. The one that was of particular interest was

./extensions.conf:exten => s,7,Macro(record-enable,${MACRO_EXTEN},${RecordMethod})


It looks like record enable takes the extension to record under. In several other places, RecordMethod is set to OUT or IN. After looking at the structure of files in /var/spool/asterisk/monitor. I determined that these variables are probably to help determine the filename of the recording.

I created a new dialplan in extensions_custom.conf


exten => _*123NXXNXXXXXX,1,Answer()
exten => _*123NXXNXXXXXX,3,Macro(record-enable,000, OUT)
exten => _*123NXXNXXXXXX,4,Dial(SIP/icall/${EXTEN:4},,g)
exten => _*123NXXNXXXXXX,5,System(curl http://192.168.1.110/RecordLog.php?args=${EXTEN:4}~${CALLFILENAME})
exten => _*123NXXNXXXXXX,6,Hangup


The dialplan matches a pattern (indicated by _) starting with *123 followed by an area code and 7 digit number. First the call is answered, recording is enabled. The dial is executed. EXTEN:4 removes the first 4 digits from the dial string *1234178575309=4178575309. The missing parameter between the first and last arguments of dial is for the timeout which i want to default to infinite. g is a flag that tells asterisk to continue through the dialplan after the callee hangs up. That means execute priority 5 and 6 after the person being called hangs up.

Before I added Priority 5, I had to figure out how I was going to pass the call information to a second server and get it downloaded. I did asterisk -r and executed "dialplan reload". I dialed the *123 pattern from a softphone and watched to see what variables were being set. I saw that CALLFILENAME was being set. I had used curl previously to send data about calls to another server.


Priority 5: With callfilename set, I call RecordLog.php on my webserver with args=The number dialer~the callfilename. Why did I do it like that? Its because asterisk kept getting confused when I had an ampersand separating multiple variables for the qs. Asterisk would pass the command to bash, and bash would interpret the ampersand as & in bash (background the current command and execute something else). In retrospect, now that I'm writing this I could have quoted the url.

RecordLog.php

//RecordLog.php?cid=${EXTEN:4}&filename=${CALLFILENAME}
$h=fopen("/var/www/web1/web/recordings/call_index.htm","a+");
$args=$_GET["args"];
$bits=explode("~",$args);
$cid=$bits[0];
$filename=$bits[1];
fwrite($h,"$cid\t".$filename."
");
fclose($h);
$url="https://192.168.1.237/recordings/misc/download.php?call=".$filename;
$output=shell_exec("curl -k $url > /var/www/web1/web/recordings/$filename.wav");


We open a call_index.htm file for append writing in a password protected recordings directory. We parse the qs and find the cid (caller id) and filename. We write a link to the htm file. We execute a curl command to retrieve the call with that filename from asterisk and pipe the output directly into a wav file.

download.php(Modified from /var/www/html/recordings/misc/audio.php)

if (isset($_GET["call"])) {

$path="/var/spool/asterisk/monitor/".$_GET["call"].".wav";
if (!is_file($path)) { die("404 File not found!"); }

// Gather relevent info about file
$size = filesize($path);
$name = basename($path);
$extension = strtolower(substr(strrchr($name,"."),1));

// This will set the Content-Type to the appropriate setting for the file
$ctype ='';
switch( $extension ) {
case "mp3": $ctype="audio/mpeg"; break;
case "wav": $ctype="audio/x-wav"; break;
case "Wav": $ctype="audio/x-wav"; break;
case "WAV": $ctype="audio/x-wav"; break;
case "gsm": $ctype="audio/x-gsm"; break;

// not downloadable
default: die("404 File not found!"); break ;
}

// need to check if file is mislabeled or a liar.
$fp=fopen($path, "rb");
if ($size && $ctype && $fp) {
header("Pragma: public");
header("Expires: 0");
header("Cache-Control: must-revalidate, post-check=0, pre-check=0");
header("Cache-Control: public");
header("Content-Description: wav file");
header("Content-Type: " . $ctype);
header("Content-Disposition: attachment; filename=" . $name);
header("Content-Transfer-Encoding: binary");
header("Content-length: " . $size);
fpassthru($fp);
}
}


I know there are some dangerous security flaws in this code. I rationalize it like this, this is not accessible except from in our network as our PBX is not exposed directly to the internet. THe original audio.php had some pesky crypt function interfering with things.

I hope this was helpful to someone