I recently embarked on a little effort to get Twiki 4.1.2 installed on Kubuntu 7.04 (Feisty Fawn). While there is a package for Twiki, it is not the bleeding edge latest, and I also figured it was a good chance to learn my way around some more linux system administration.
The setup went smooth; there are a few other blog posts out there about how to set it up on Ubuntu and Kubuntu. I won’t cover that here.
What I did want to share was a little experience with trying to make Twiki talk to my mail providers SMTP server which requires authentication. It was a little adventure in SMTP authentication, and a little adventure in submitting a bug to an open source project (another task I had not personally tried before).
Mail Providers
Maybe I went wrong right up front – well not went wrong, but started from a little different position than some others. I use a 3rd party mail host, that is, a 3rd party beyond even my ISP or my web host. I was recommended Webmail.us by a friend some time ago and really enjoy their service – very fast servers, complete protocol support, and a great spam filter (the primary reason I moved – server-side spam filtering to replace the Bayesian filters I had installed in all the locations I might check mail). Webmail.us requires authentication against the outgoing SMTP server, requires that the email I send include the same account I just authenticated as in the “from” field, and uses my entire email address as my username. Then again, maybe that is not so strange – I am sure many ISPs have similar requirements these days.
1st Try
So, when first configuring my newly installed Twiki, I used the /bin/configure script included to edit my LocalSite.cfg, the configuration file that Twiki uses. This file includes fields for your authentication username and authentication password – looks good, right?
Not really. Once I filled in my information, and tried it, I immediately received a “error when sending email” anytime the system tried to generate an email.
Next step, I see that there is a debug mode you can change in the same /bin/configure script that outputs SMTP session information to the web server log. Turn this on and I can see the problem right away, the SMTP server is returning an authorization problem:
<<< 220 relay5.relay.iad.emailsrvr.com ESMTP - VA Code Section 18.2-152.3:1 forbids use of this system for unsolicited bulk electronic mail (Spam) >>> EHLO localhost.localdomainr <<< 250-relay5.relay.iad.emailsrvr.com <<< 250-PIPELINING <<< 250-SIZE 75000000 <<< 250-ETRN <<< 250-STARTTLS <<< 250-AUTH PLAIN LOGIN <<< 250-AUTH=PLAIN LOGIN <<< 250-ENHANCEDSTATUSCODES <<< 250 8BITMIME >>> MAIL FROM:<me@danpisarski.com> <<< 250 2.1.0 Ok >>> RCPT TO:<danp@danpisarski.com> <<< 554 5.7.1 <me@danpisarski.com>: Sender address rejected: Access denied >>> RCPT TO:<me@danpisarski.com> <<< 554 5.7.1 <me@danpisarski.com>: Sender address rejected: Access denied
The problem with this block is I don’t see where Twiki is actually trying to authenticate against the SMTP server, so I am not sure what is going wrong. I know a little about SMTP, and next to nothing about SMTP authentication, so I figure the authentication must take place during the connection to the server, not in the session, and it is a problem with my username or password, or my settings.
Easy enough to test, I fire up telnet from the Kubuntu console and connect to the same SMTP URL, port 25. I am able to immediately re-create the exact session above – at no point does the server ask me for a username or password. So how the heck does SMTP authentication work?
One more note, I also temporarily setup my twiki to use a SMTP server I happen to have access to that requires no authentication – and it sends emails right away. So Twiki is working in general, just not sending email to my authentication-required SMTP server.
How does SMTP authentication work?
So next stop is Google to find out how SMTP authentication works, and I find this very helpful little guide to how to test authentication from a telnet session.
You will note in this article that they point out that using the ‘@’ symbol in Perl can be problematic, and Twiki is written entirely in Perl, so I start to suspect that might be the problem. I try it out, creating the Base64 string at the command line using Perl as shown, escaping the ‘@’ sign. I am able to now create a telnet session where I authenticate, by issuing the AUTH PLAIN SMTP command with my Base64 string, and once authenticated, I am able to fully send a test email to an outside email address, using the email address I authenticated with (which is also the email address I have set as the administrator email of Twiki, and thus the one it uses to send emails).
Still suspicious of the ‘@’ sign though, I decide this appears to be a bug in Twiki (I can, after all, send email just fine from a telnet prompt, why can’t Twiki?) and decide I should be a good community member and submit that bug to Twiki.
The Twiki Bug Submission
So, I did. And, pretty quickly, I get a few suggestions – to check inside the LocalSite.cfg to see if the string is encased in single or double quotes, which controls whether Perl does variable substitution in the string. Sure enough, it is in single quotes, which means the ‘@’ cannot be the problem. As you can see by the bug submission though, this wasn’t the only suggestion that came through – right away, the suspicion by the Twiki developers is this is my SMTP server’s fault, and I am troubling them with something that is just some ISP anti-spam rule.
Fair enough, I go back to add the data that yes, LocalSite.cfg is using single quotes and yes, I can authenticate just fine to this SMTP server from a telnet command prompt, using the AUTH PLAIN command. Oh I can also use the service just fine from Thunderbird and other mail clients. So why isn’t Twiki issuing an AUTH PLAIN command, which I can see it is from the debug output? While a few others suggest this might be a shortcoming of the underlying Perl library (sounds sensible to me, though if it doesn’t support this type of SMTP authentication what does it support?). But, if you look at the end of the little conversation after my post, the final verdict is – of course – its my fault, or more specifically my mail provider’s fault, and I should move my question to a support forum and not the bug database.
Fine.
A Support Forum Sounds Helpful, Right?
So, I take my problem over to the Twiki support forum, and try to include all the data I now know that I think is relevant. By this time, I suspect the real answer – that it is indeed a bug in either Twiki or the underlying Perl library, that it does not support using the AUTH PLAIN command or is simply failing to issue that command at the appropriate time in the SMTP session, and that there is no immediate fix to this issue.
But what response do I get? The very same person who discouraged me out of the bug database once again telling me its my mail host’s fault. Why did he recommend I take this issue to the support forum if he was just going to once again tell me its my fault?
Making It Work
Now that I had had about all the fun I could over on the Twiki site, it was time to make it work for real.
I started by downloading the stable Twiki VM, which is running on Debian. I figured, let me see how they have configured SMTP in this pre-configured example, to try and learn what methods might actually work in the software. Interestingly, they have skipped having the user of the VM setup an SMTP right in the /bin/configure section, and instead are using a mail transport agent, or MTA, to forward email.
An MTA makes sense, I have certainly used Microsoft’s variant (the Virtual SMTP Host) as a way to have web servers send email through a single smarthost without having to code SMTP urls into every web server box (and to add a layer of queuing). The Debian install seems to be using exim4, which seems to be a favored linux MTA. It is a touch complicated at first glance, but once you dig a bit it is not hard to configure on Kubuntu.
Here are the steps I followed:
- The first step is to add exim4 and exim4-config, and any dependencies, as they are not installed by default in Kubuntu. Once exim4 is installed, the version that installs on Debian-package versions of Linux like Kubuntu includes a utility to help you configure it. Activate this command with:
sudo dpkg-reconfigure exim4-config - You can select mail sent by smarthost; no local mail as the overall setting, though perhaps your needs would lead you to a slightly different selection – the point being you want to forward all your email to a smarthost
- Once prompted, enter the SMTP url. you do not need to include any username or password data here.
- Complete the rest of the exim4 configuration, I think the settings are pretty straightforward, your needs might vary.
- Once you are done, this tool automatically restarts exim4, and your settings are saved to /etc/exim4/update-exim4.conf.
- You will next need to edit /etc/exim4/passwd.client to add your authentication information. Inside the file is an example of the format to follow, you re-enter your SMTP URL, then add your username and password.
- To make exim4 pick up these new settings, run:
sudo update-exim4.conf - in Twiki (/bin/configure), set Twiki to not use any username or password for SMTP, and set your SMTP server to localhost.
This configuration worked fine for my mail host, and Twiki started sending out email right away. Funny that exim4 is perfectly capable of logging into my SMTP server – I guess I better include that data when I take the advice of the Twiki development community and submit an issue to my mail host.