kief.com

OK, so I've pieced together some more of what's going on with my WAP SSL problem. It turns out that on old-skool WAP phones, SSL encryption does not happen from the phone, but instead is done between the WAP gateway, usually at the network operator, and the web(WAP) site. The connection between the phone and the WAP gateway is encrypted with WTLS, Wireless Transport Layer Security. This is a wireless version of TLS, which is a more general Internet standard.

But newer phones, the ones with their own TCP/IP stack, use regular TLS, which allows them to have proper SSL end to end, from the phone to the site.

This much I have confirmed from googling and foruming, but I still have not found any explicit mention of how this affects the MSISDN issue. It stands to reason the WAP gateway isn't going to be able to insert the MSISDN into the headers in this case, but this has some fairly serious implications for the way secure WAP applications are designed. Having the operator provide the MSISDN is a very useful thing, it gives you a fairly reliable way to identify people, and can even be used to authenticate them to a certain degree.

Well, it's good to be learning new stuff!

Something we're trying to determine is whether or not there is a problem getting MSISDN numbers on SSL connections from Symbian phones. MSISDN is essentially the phone number of the mobile phone, which some network operators provide as an HTTP header to authorized content providers such as ourselves, so it can be used to authenticate the phone.

We get these fine on lower-end phones whether we're using plain HTTP or SSL, but on Nokia 6600's and others we only get the MSISDN on clear connections. This may be an issue with the operators' gateway configuration, or it may be more fundamental. Is it possible Symbian's network stack prevents the operators from inserting headers into the request with an SSL connection?

For that matter, how can they insert the header even on the lower-end phones, given that the phone recognizes our server's SSL certificate, which suggests the gateway is not acting as a man-in-the-middle?

More research needed.

kief.com was down from Thursday evening until a little while ago, thanks to a power outage. Can't complain about free hosting space, can I? Aside from the loss of traffic (dozens of hits, I'm sure), I'm sure I did lose some email. If you tried to send me mail since Thursday, please try again.

I mentioned a few days ago that we're looking for a J2ME person, well we're now also looking for one or more Senior J2EE peeps, as well as a Database/Unix person. As before, we need solid, technically well-rounded people with the right disposition to cope in a small, startup-style environment. We've got solid funding from big-name VC's and we have strong relationships with the mobile network companies, so this isn't a fly-by-night outfit. But we are very much a young company in an emerging field.

The biggest problem I'm seeing in recruiting for us is that many people don't have the stomach to work in this kind of environment. That's fine, not everybody is suited to fly fighter jets, the world needs airline pilots to handle the routine stuff. For me, I can't stand working in big companies where I'm not allowed to deal with an entire system, and I know there are others like me.

Several years ago I had a play with Orion, which seemed like the up and coming Java app server, the preferred tool for the True Techie. It was well-built, running very fast and incorporating all the latest standards, and on top of all that it was cheap, $1,500 per server (regardless of how many CPUs), and freely downloadable and usable for development use with no registration keys, expiration, or other cripplement.

But then Oracle licensed Orion and used it to build their App Server product. While on the face of it this shouldn't have affected IronFlare's delivery of Orion, other than to give them some cash and "enterprise" credibility. But apparently the licensing required IronFlare's developers to spend a fair amount of time supporting Oracle in some ways or other, so their own product has slipped.

Basically, Orion stagnated for a few years, and fell of my radar as well as that of the Java community in general.

But over the past year or so they seem to geared back up, and have the potential to come back into the lead of the pack, especially among more technically driven projects. It makes a lot of sense for our company. We use Weblogic, but WL has a number of disadvantages, particularly that it's massively expensive, it's bulky and slow, and it's tied to commercial distributions of Linux, namely Red Hat.

Orion, meanwhile, is still cheap, with a developer-friendly licensing model. It's still fast, loading up within a second or two on my laptop, and quite simple. It does have its drawbacks; the documentation is only so-so, and there's no real admin interface.

Configuration is old-skool, XML files with no Web/GUI configuration tool, which might put some people off but to me is perfectly fine - text files are much easier to manage, especially for complex, multi-server systems, than having to remember a list of places to point and click.

Logging is pretty minimal, with few options that I've found so far to turn up debugging levels, so figuring out why my application has trouble loading isn't easy. I also haven't found a remote console to track performance indicators. There is a rudimentary GUI console that can be started along with the server itself, but it's local to the server machine, and closing it shuts the server down; so while it's useful for local development, it's useless on a production server. Orion may have JMX/SMTP hooks, if so that would at least give a way to build something to the purpose. But if not this is probably the point that would make it hardest to sell Orion into my company as a replacement for Weblogic.

Another disadvantage is that Orion has yet to catch up to the standards curve, it's still based on J2EE 1.3, including Servlets 2.3 and JSP 1.2.

There's an opportunity for Orion to steal a march on Weblogic if they get their shit together quickly. There are tons of companies around still using Weblogic 6.1, pondering the upgrade to 8.1. That's an expensive upgrade, both in licensing costs and in having to port to the newer APIs. Orion offers a way to cut licensing by at least 90%, and the Weblogic API changes make it no more of a pain than upgrading to 8.1 would. They just need to polish around the edges and they could make a serious leap.

I assume so anyway. For a couple years I've been trying to persuade family visiting from the States that I really did see a restaurant called "Phat Phuc", but I could never remember where it was. The other day when my wife dragged me out to Chelsea I finally found it, or at least the sign.

We're mostly a server-side tech shop - we run a set of applications on Weblogic which users interact with using SMS and WAP, and which makes calls out on the back end to standard web sites. So essentially we provide a bridge so services normally available on the web can be easily provided to mobile phone users. We don't currently have any phone-based applications, but we're looking to do so now.

So if you're an experienced J2ME programmer and can work in London, give me a shout. Keep in mind the "experienced" part - we've got people who have done a bit of J2ME programming, what we need is someone who is initimately familiar with writing and debugging apps that run across many phones. They also need experience with animation, and will need to integrate with server-side applications, so the more you know about HTTP, socket programming, and the like, the better. In general, our company expects programmers to have a solid grounding in CS fundamentals (our founder is a former CS professor).

Oh yeah, we're also looking for server-side Java peeps as well. You don't need experience in the mobile area, although it wouldn't hurt. Mainly you should be into J2EE (we use servlets, Session beans, and JMS mainly, no JSP and no entity beans), and be solid enough to contribute to different parts of the system.

I've been working at a new job for a little while now, and it's a refreshing change from my old job. As I've said before, I quite like my old company, but I got stuck in a rut. I've worked both sides of the programmer/sysadmin line for as long as I've had technical jobs, but the danger of being a sysadmin is that it's too easy to get mired in "IT support", especially at a small company. Herding application servers was fine, but after a reorg I found myself working for a boss who wanted my main focus to be things like new versions of Active Directory, Exchange, and backup systems for laptops - and this is not the direction I want for my life.

So I was really hankering to get back into programming, and I obviously switched just in the nick, considering how hard it was to get through the screens of ignorant recruiters. It didn't matter what I was doing 2 years ago, when they didn't see the right acronyms on my current job they balked at sending my CV through. As far as they were concerned I was a systems administrator, sure I can install and run EJB servers, but my CV doesn't say I have been writing EJB's, so I'm not a programmer. They typically held my CV back for weeks until "better" candidates didn't make the cut, sending me in when they didn't have anything else. They were quite surprised to find that I was actually able to land offers for programming roles. People on the sharp end realize that someone who has done both programming and systems management can be quite useful, but it's not the kind of thing that typically ends up in a job description.

The irony is that now I'm involved in interviewing the kinds of people that recruiters do think are suitable. There is a glut of people who have J2EE on their CV, but for whom this means they have spent the past 3 years working in a large company working on just one thing. I've actually seen people whose entire job was to write stateless session beans. They haven't done much database work (someone else's job), they don't know anything about threading (someone else did that), and they can't write an ant build script (that was a different department), but they know J2EE.

Anyway, I'm enjoying my new role. I'm head of the development team, which is a jolt from heading a small sysadmin team. All of the guys on the team are solid, hard core developers, and given that the system they've built over the past couple years is very involved and not documented for an outsider to learn, I've got some catching up to do. On the other hand, my job here isn't so much to be the alpha geek programmer as it is to be the coordinator between product management and developers. So I'm getting the development process under control and getting a handle on priorities and such.

What's especially cool is that I'm learning a new area of technology. Where most of my work in the past has been web and email focused, now I'm involved in WAP and SMS. Although where the rubber hits the road there's not much difference - it's the same servlets with a different view - there are plenty of new issues involved. At the moment I need to get a handle on various issues with different combinations of phones and mobile operators.

For instance, it turns out Orange had been selling Nokia 6600 phones without Thawte SSL certificates, which means people with those phones can't use our service. This is pretty fucked up considering Thawte are a major provider of SSL certificates. Orange blames it on Nokia, but in fact they install their own version of the OS, and Google doesn't turn up anybody else complaining about 6600 phones not being able to use their secure sites, so you do the math.

One of my company's mail servers got blacklisted by SpamCop last week, which led to our learning a new mail server configuration tweak that's needed in today's spam infested world. We do "e-marketing" mailings, but these are mostly sent to people who specifically sign up for them on corporate web sites - our cients tend to the big and bad-publicity/legal action averse, so everything tends to be very strictly above board. But it turned out the servers we use for this weren't blacklisted, it was the one we use for regular user emails to/from our staff.

We got blacklisted for 48 hours based on a "spamtrap"; an email coming from one of our servers was sent to an email account set up by SpamCop strictly to detect unsolicited mail. The idea is that the spamtrap email address is never given out in a context that would attract legitimate, solicited email. Unfortunately SpamCop is very tight with details on their processes for spamtrap addresses, and there is a lot of variation in how such addresses can be managed.

The strictest spamtrap policy would be simply to set up an address on a server, but never publish it in any way. However some spamtrappers might carefully leak the address in certain controlled ways, for example putting it on an HTML page somewhere for address harvesting crawlers to find, or maybe even using it to send an email to a known address-list seller knowing they'll add it to their lists.

A potential problem with all of these is that spammers and email worms tend to forge email addresses and the sent-from lines that track an email's progress through the Net, and these days they use legitimate addresses and servers. So if they pick your address or your email server and use it in an email to a spamtrap, you can potentially end up on a blacklist.

As it turns out, SpamCop seems to avoid falling for this. The trick that got us blacklisted was actually more subtle, and is a side effect of collateral spam. Collatoral spam is the flood of bounces, "our system detected a virus in your message" alerts, and other useless crap that gets sent to an address that is used as a patsy by the spammers and email worms. If a spamtrap address is used as a patsy address in a spam or worm that is sent to your system, if your system automatically sends a bounce or "you sent us a virus" message, you'll be blacklisted.

At first glance, this is a shitty thing to do in the case of bounces. Mail servers that automatically send "you have a virus" emails to addresses that have been forged in incoming emails deserve to be slammed, they serve no purpose other than to help these people spread chaos to wider audience. My own mail server has filters that refuse to accept delivery of messages that look like a virus alert, with a message explaining why this is stupid.

But bounces are a necessary part of email transportation, in fact they're required by email standards, so at first SpamCop's policy looks like it punishes you for complying with the standards. As it turns out though, there are two ways to bounce mails.

The old way is to take delivery of an email and close the SMTP connection before checking whether the email is addressed to a legitimate user. If not, a new email is created and sent to the appropriate headers of the received email. This has become a Bad Thing, because it means you send a bounce to a potentially (95%+ of the time these days) forged address, not to the original sender; this is collatoral spam.

The new way (probably not new as a configuration option, but certainly new as a "best practice") is to check the address and reject it during the SMTP connection. This tells the server actually attempting to deliver the mail to you that you don't want it, so nobody is involved who wasn't already a part of sending the invalid email. The original sender is still notified that the address they tried to send to doesn't exist, they're just notified by the other email server.

There are a few drawbacks. It used to be considered bad practice to give immediate feedback during SMTP as to whether an address exists at your server. I was taught early on that a mail server should disable the VRFY command, which asks "does this user exist"? The reason being, a script can connect to your mail server and randomly try different addresses in the hopes of discovering a working address to add to a spam-list. The new fix makes it pretty easy to do this, but it's pretty clear that these days most spammers won't bother to see if an address is legit, they'll mail to anything. (Spammers these days just don't have the class that the spammers did in the romantic old days, do they?)

Another problem is with mail servers that actually don't know what addresses are legitimate, such as those providing an email domain relay service, or acting as a gateway into some more complicated system. If you use a Unix mail server on the front end to block spam, and then deliver to an internal Exchange server, you may not want to have to keep user addresses synchronized. But if you don't, you're in danger of getting your mail server blacklisted for sending collatoral spam.

Ugly, ugly, ugly.

So how can you see which way your mail server is configured? Use your favorite terminal program to have this conversation:

kief@icarus:~$ telnet smtp.my.com 25
Trying 192.168.1.10...
Connected to smtp.my.com.
Escape character is '^]'.
220 smtp.my.com ESMTP Postfix
HELO icarus
250 smtp.my.com
MAIL FROM: myaddress@my.com
250 Ok
RCPT TO: nosuchaddress@my.com
550 : Recipient address rejected: User unknown in virtual alias table
quit
221 Bye
Connection closed by foreign host.

The above conversation means you're rejecting it right away. If it accepts the recipient, either that recipient exists, or the mail server is going to send a bounce to whatever address you enter in the Sender or From header in the body, if you continue the conversation.

Alternately, if your mail server is the same one used for SMTP delivery by your mail client, you can just use your client to create a message to a bogus address at your domain and try to send it, but this test depends on your client and configuration.

Map of the US indicating which term is popular in different parts of the country. I'm a "Coke" many myself - trademark dilution be damned, I'll never call it "pop". Found via Neil Gaiman.