next »

[spraxyt]

… there will be some 'interesting' graphs coming RSN. There's nothing quite like ending with a good old tease and watch-this-space stylee comment is there

Shame. Looks like I'll have to manage with the Sunday papers this week then.

[mikeb]

Right then, sorry, kinda missed the press for the Sunday Papers (and Monday as well come to think about it !) but here's some light bedtime or early morning reading instead

It's been a bit strange on the postini front over the last few weeks, even allowing for possible errors in the data due to the MMM bugs various that totally screwed things up for a while. Nice to see a reduction in volume last week although some plausible explanation for the recent(ish) fairly dramatic increase(s) would perhaps be even nicer ! The sad reality is that volume is just about back up to what it was immediately prior to the Christmas peak which is all a bit odd as I don't see much (if any) evidence of a general significant increase over the last few months.  It's fairly easy to spot the point when PN turned off the MX-core first level filtering ... and perhaps not for the first time it has to be said   Rather sad really considering how well it's performed for many years and that the postini alternative is ( A ) not as simple, reliable or effective, and ( B ) has some very serious side-effects that will result in loss of genuine mail and/or necessitate very frequent maintenance to minimise (but not totally prevent) potential problems.

Well, no real surprises here. The short-comings in both postini and PN systems are still being pretty clearly demonstrated. Poor performance due to the whitelisting back door ensuring that spam with forged addresses comes flooding through, PN failure to make best possible use of the various postini scoring data, postini failure to reject even the most obvious of spam and in particular, their failure to make any effort whatsoever to reject spam from non-existent sources that is generally speaking very non-compliant to RFCs various and packed full of very obvious forged headers and other associated data. In addition to that, the PN post-postini filtering doesn't appear to be working correctly again. The Neptune quarantine header isn't always being acted on as it should be and blatant spam (with a zero or near-zero score) is still being delivered along with totally genuine mail due to it having a forged whitelisted address and despite it not originating on the PN network.  Overall feeling: Performance could easily be lots better if PN did the 'right' thing and if they actually fixed the known bugs reliably but it will still never be particularly good IMHO due to the rather fundamental short-comings at the postini end and I can't honestly imagine any positive changes are likely anytime soon.  They'll no doubt simply turn up the aggressiveness in general as/when they feel the need and with the usual complete disregard for customers' 100% genuine mail being rejected. And talking of which ...

Finally, after 3 months, I've seen what I was expecting to be seeing on a much more regular basis. Note the sudden (relatively) dramatic decrease in spam scores for the Test Message and Forum Reply. Bearing in mind that every message has been identical and virtually identical respectively with no changes in sender or routing, there is no particularly plausible reason for such a decrease in scoring other than a general increase in aggressiveness. It is therefore not in any way surprising that many more totally genuine messages seem to be getting lower scores and/or simply not turning up at all. Once again, short-comings or at least some very dubious performance from postini is apparent, this time relating to the "2strike" feature. A large number of forum replies and similar for instance are now being 2striked meaning that messages with spam scores >0.15 (often >>0.15) and BSB scores >90 are frequently being dumped/rejected without any good or even any obvious reason.

[mikeb]

Sorry for mucho lateness ... so much to do, so little time and all that   not to mention my bottle of 'something chilled and refreshing from the fridge' is getting all warm .. and drunk Nothing much to say really as it all looks to be a bit more of the same at a quicky glance but have some shiny new(ish) graphs anyway. Will try and wander back sometime later or probably tomorrow to catch up with other stuff.

[mikeb]

Shiny new graphs being generated as I speak type and coming RSN ... well, after I've made some sort of attempt to catch up with other stuff anyway.  They should appear below as if by magic probably in around 30 mins or thereabouts ... unless I venture outside and get sidetracked by the prospect of enjoying something cold and refreshing from the fridge whilst out there and kinda 'forget' to come back in of course

[bpullen]

Mike. In case you miss it, I just wanted to make you aware of this work that is intended to improve the detection accuracy for Postini filtered spam. Will be interesting to see if the changes reflect on your next set of graphs.

[mikeb]

Sorry chaps, been too busy in general to keep up with this so it's a bit late again this week and lets not talk about last week eh  I also seem to have acquired some unforeseen very time critical additional tasks as well just recently so will have even less time available over the next few weeks. The next update is most likely to be in a couple of weeks I'm afraid.

Thanks for the link to your post Bob, no surprise that using the postini data somewhat more sensibly (from a spam detection point of view) than in the past has improved the detection rate.  As I've said before tho, I'm no longer monitoring false positives other than having a quick look every now and again but there will no doubt also have been an associated increase there going on previous experience. If you care to look back to the earlier discussions on using the neptune-quarantine header then I'm quite certain that you will find several examples of 100% genuine messages that would now be being classed as spam if they were still being delivered. I suspect, however, that most if not all of those messages are now being completely blocked by BSB in any case mind you.

I also note that some users appear to have a different postini configuration to other users once again going by a few example message headers that I've seen over the last several weeks. As reported in detail somewhere on here, my configuration (and headers) suddenly changed mid-way through the MMM 'upgrade' (or rather the nonsense that immediately followed the actual upgrade) without any explanation whatsoever but it would very much appear that certain other customers still have the original postini configuration and associated headers appearing in all of their messages. I cannot believe that PN are still so not in control of the postini configuration that they have somehow managed to end up with different postini configurations being applied to different customers for at least a second occasion. I didn't get any answers last time I raised this sort of thing, just an effective denial that any problem existed (for want of any better description of what was actually said), so I'm certainly not expecting any answers this time either and I guess that the 'problem' will get quietly 'fixed' like it presumably did last time. But as always, I would suggest that Configuration Management ... or lack thereof ... should be something that PN actually takes seriously.

Some shiny new graphs for your viewing pleasure will appear below sometime soon(ish):

Sorry (yet again) had a seriously blond moment at very stupid o'clock ... one of several in fact, so am currently surrounded by backups various and have all fingers etc. crossed The graphs should be here now so if you can't see them refresh the old page and all that.

[mikeb]

Here's this weeks graphs. Next update will most likely be in a couple of weeks time.

[mikeb]

Here's this weeks graphs.  Next update will definitely not be for a couple of weeks as I will still be standing in the middle of a (no doubt very muddy) field somewhere in deepest Somerset this time next weekend ... but having considerably more fun than I would otherwise be having messing around with spam various on ye olde 'puter of course 

Maybe PN or anyone else for that matter could explain the apparent correlation between Spam Detection Rate, Whitelisted Spam and PN tinkering about before I return ?  Very strange methinks.

[mikeb]

Here's this weeks graphs.  Note the scale change on the second graph due to the high volume spike on Thursday 03/07/2008. In addition, around 200 NDA's were received on Monday 30/06/2008 and these have been excluded from the figures due to the unreasonable distortion that would result if they were included in the data.

[mikeb]

Apologies to anyone who may still be 'interested' in the graphs that I've not updated them in a while for reasons various. The latest are below and once again, I've had to excluded several batches of NDAs from the figures in recent weeks due to the unreasonable distortion that would result if they were included in the data. 

[mikeb]

The latest graphs are below and once again, I've had to excluded a small batch of NDAs from the figures this week due to the unreasonable distortion that would result if they were included in the data.  An interesting couple of weeks since the last update !

I'm sure it's no surprise to anyone that there is a huge dip in detection rate due to the deluge of (often) undetected malware "news" spam being sent out recently to grow the bot.  Although this seems to be slowly dying out despite a number of variants appearing, the 'usual' spam has been very much on the increase again over the last few days thus indicating the success of the malware distribution. I'm expecting volume to increase for a while

Also very interesting is another step-change in the scoring of regular identical genuine messages. There seems to be a similar change in the scoring of some other random genuine messages but by no means all messages. I still have various messages that fail to get delivered and plenty of others that regularly get a completely ridiculous low score that simply cannot be justified. The only sort of 'conclusion' you can draw from this is that it appears postini (or some other party supplying them with data) update things 3 times a year. This is now the second occasion that there has been an unexplained but fairly widespread significant step-change in the scoring of genuine messages ... it will be VERY interesting to see what happens around December time

[spraxyt]

Thanks Mike, it's interesting to see the effect of the "news" spam.

In fact native Postini reacted to that very quickly but Plusnet's use of the Postini headers required a mail-server configuration change which was tested and released a few days after the deluge started.

[mikeb]

Sorry for mucho lateness but here are the latest graphs for your viewing pleasure.

[mikeb]

Well, firstly, sorry for the missing updates recently and all that. Unfortunately, I didn't quite get around to grabbing the data in time a couple of weeks ago so it was very much easier to let it run on until this week rather than try to separate the data out. A very 'interesting' and certainly very strange few weeks but at least it seems to have returned to something more like 'normal' again now.

Around the end of August, the first sign of strangeness was a sudden and distinct lack of spam appearing and a distinct lack of flavours as well. This has happened a few times before - a sudden and unexplained decrease in volume and a lack of variety with all the most recent and popular subjects disappearing. postini (or perhaps PN) simply must have updated something somewhere and it seems that rather like message scoring algorithms, they don't do it very often but it's usually pretty obvious when it does happen. In previous cases, it also tends to herald strange problems with genuine mail as well. In this case there were many reported problems with mail from UK2.net (I think) and a few extra missing genuine messages on my accounts from various sources.

In addition to this, for at least some of the time during the strange period, I was experiencing some very odd behaviour with the postini servers. A simple text only message was taking several minutes to be accepted by the servers. I don't mean that an SMTP connection couldn't be established for one reason or another for several minutes but that it took several minutes to complete the session once established. The connection was established almost immediately but then was being routinely stalled for several minutes before completing. Typically this was a couple of minutes or so but I also saw around 15 minutes intentional delay on more than a few occasions !  Yup, that's right 15 minutes to complete the sending of a totally legitimate 2 line text message from my account to my account !!! Postini were apparently intentionally dragging out all SMTP sessions for up to 15 minutes per message for whatever reason. It's not in any way difficult to understand that this would seriously affect anyone sending a large(ish) volume of genuine messages to PN accounts and not that difficult to imagine other problems with incoming requests being missed completely due to all the hung connections unless successfully retried some time later. I've no idea when it started or how long postini were doing this (or whether it's something that they've always done for that matter) but either way it's completely outrageous and definitely was causing problems.  If this is typical postini behaviour then I think it goes a very long way towards explaining why so many genuine messages, particularly from high-volume senders, seem to go AWOL for no good reason whatsoever when sent via postini.

As can be seen from the volume graph below, it's blatantly obvious that something very strange was going on purely from the unexplained and uncharacteristic dip in volume. Particularly so when it corresponds almost exactly with a peak in non-postini (and therefore 100% unfiltered) spam. I would guess that postini were suddenly getting swamped by spam after the success of the relatively recent malware distribution to grow the bot and therefore went even more draconian than usual on all incoming mail regardless of how it actually scored.  The messages that I was getting intentionally dragged out for several minutes were 100% not spam with not even a remote possibility of being spam either according to the various postini headers. I see no particularly good reason to believe that other senders weren't being affected in a similar way at that time. It's not apparently happening now and as can be seen from the volume graph, things have returned to something more like expected as well but it's quite obvious from looking at my spam that certain messages that have been *very* popular for a very long time are no longer being received via postini.

All very strange and, needless to say, no comments or explanation have been forthcoming so far but one thing's for certain ... I can't believe for one moment that there's any coincidence here, postini or PN or both have been quietly tweaking stuff somewhere

[bpullen]

All very strange and, needless to say, no comments or explanation have been forthcoming so far but one thing's for certain ... I can't believe for one moment that there's any coincidence here, postini or PN or both have been quietly tweaking stuff somewhere

Hi Mike, saw your post over on Community yesterday and replied there.