next »

[jelv1]

Totally off the wall, but...

Users have been reporting false positives (ham) to Plusnet (emails marked as [-SPAM-] which are genuine, wanted emails). Presumably Plusnet has been forwarding these to Postini.

As far as I am aware, on the Postini only had a standard facility for reporting missed spam. Now what would happen if on receipt of the ham Postini thought it was spam - would they add it to the BSB lists?

It can't be that stupid, can it?

[mikeb]

OK, not enough time to do much in the way of analysis other than a quicky look this week I'm afraid. Volume is up on last weeks all time low but there were no false negatives at all this week.  Still a fair few false positives tho and several missing messages but I have been getting some (although apparently not all) coming through from senders where all messages started being blocked 2 weeks ago. It's all been a bit strange and somewhat inconsistent to say the least recently with the sudden dip in spam, the apparently associated increase in missing mail and then a sort-of recovery this past week. Still no comments from PN but I have seen a few other customers noting an increase in false positives and possible missing mail over the last 2 weeks as well. Anyways, graphs for your viewing pleasure follow:

[mikeb]

Well, here we go again and still no time to do anything much except bung up the graphs and disappear I'm afraid. I still have almost a months worth of genuine e-mail virtually unread and certainly not responded to at the mo so the absolute last thing I need right now is to go trawling through shedloads of mail to try and establish exactly which did and which did not get eaten by postini !  Still some very obvious false positives tho and still some mail apparently going AWOL.  Reporting "spam" and "not spam" appears to be a complete waste of time and effort as no action appears to result from filing many reports.

Anyway, it's all gone a bit odd again this week.  Another major spike in volume last Sunday but nothing particularly unusual as this sort of thing has been seen before.  However, performance is very much on a downward trend again due to what appears to be a shiny new flavour of spam arriving as of a couple of days ago or, given the date, possibly due to a postini update c*ck-up which is now letting it flood through. Either way, let's see how long it takes postini to respond to the problem this time ... and hope that PN finally get around to responding to postini's apparent changes from some considerable period of time ago of course.

I'm very concerned about the apparent total lack of configuration management so far as the PN/postini system is concerned. Questions have been asked several times now on the PN forum and 'reminders' have been given about 'over-looked' posts containing important points but still no response and some of these go back almost a couple of months now to around the time the "2strike" issues came up. Very suspect methinks. Anyway, it would appear that postini potentially make changes to headers willy-nilly and PN either do not realise that this could break (or has in fact broken ?) their implementation and do not take any appropriate action. It would also appear that some customers have a fundamentally different postini configuration to other customers. All in all, I find it most concerning that no one appears to know what's going on and why ... or even cares enough to find out in a timely manner.

[mikeb]

Still no time for too much messing around but here are this weeks stats for those who may be interested. Going up .....

[mikeb]

Hmmmm, for anyone who hasn't noticed, it's not been a particularly good week on the old spam front. An all-time (postini) high   Plenty of it in general and plenty of it getting through largely unscathed as well.  Volume still very much on the upward trend and several more new flavours that are doing a real good job of getting around postini along with those that have appeared over the last couple of weeks and, to a certain extent, also managing to bypass the PN tagging as well.

One of the new flavours is being handled particularly badly by postini and demonstrates yet another serious short-coming in their detection algorithms. An intentional or simply erroneous line space in a (very badly) forged routing header technically makes all subsequent headers part of the message body.  However, good old postini appear to be accepting all the invalid headers rather than seeing the message as a serious non-compliance with RFC822 and rejecting it outright. The message should be considered totally non-compliant because the "to", "from", "subject", "message ID" and other fields are effectively not present despite some of these being mandatory not optional.

Their processing is totally inconsistent though because they then go on to add their own headers after the stray line break and before all the invalid headers that they had previously accepted and used for routing and analysis purposes. A very poor show and the result is shedloads of them sailing through, aided and abetted by Mr.Whitelist of course.  postini still don't appear to have caught up with the last batch of new flavours so god knows just how long this latest deluge is going to last.  Mr.Spammer is very much better at finding holes than Mr.Postini is at plugging them it would appear.

In addition to that, the PN "subject line tagging" process is (quite correctly) not finding the subject field in this particular new flavour of spam because it is buried in the message body but unfortunately it is rather stupidly doing nothing at all as opposed to simply adding a valid "subject" header with "-SPAM-" in the appropriate place in exactly the same way as a valid "message ID" is added when this is missing.  Any message with a blank, missing or incorrectly placed "subject" field that requires tagging is currently being ignored.

All in all, a bad week necessitating one graph having to be re-scaled. Still no obvious improvement on the false positives and missing mail front either, if anything it's getting worse.  What exactly are PN doing with the "spam" and "not spam" reports ?

[MauriceB]

Thanks for your continued efforts to identify and highlight some of the less good (Carp?) elements of the Postini solution.  I'm optimistic that PN will at some point come back with a convincing story of what is happening with the current implementation and their plans to make significant improvements.  Perhaps we need to start collating a 'little list' of realistic expectations against which any solution may be judged.

Maurice

[mikeb]

Onwards and upwards and all that   Somewhat unfair comparison but I'm thinking ye olde vanilla flavour PN spam filtering is performing significantly better than the shiny new improved recipe with added postini flavour at the mo.

[mikeb]

Well, at least the seemingly never ending upward trend has levelled off at last   It was looking *really* bad early in the week with a daily volume hitting an all-time high (necessitating yet another re-scale of the graph, I'm afraid) and a detection rate nose-diving to around 80% but it's recovered a bit since then. Still well out of spec tho.  In fact, over the 12 weeks that I've been on postini, only one week has actually achieved the Detection Rate SLA (although some are pretty close) but I have to cheekily suggest that was perhaps a bit of a fluke because on the particular week in question there was next to no spam and no false negatives !  The SLA for False Positives is still, needless to say, several miles away from reality of course without any need to work out an actual figure.  The very fact that I've had more than 1 in several years absolutely guarantees that it's not being met !  Virtually all of the false positives (and ridiculously low scoring messages in general) that were first seen some 12 weeks ago are still appearing much the same this week. So much for investigating not-spam reports, training the filter(s) and/or correcting fundamental errors in the algorithms

[mikeb]

The monitoring has all gone completely t*tsup today due to the fiasco that is PN implementing a whole bunch of mail changes complete with the usual collection of avoidable problems various that we always seem to get whenever anything changes just to add to the excitement and all that.

I have just found out that all my A/Cs mail options have been changed by PN without my knowledge so virtually all of my messages (genuine and spam) from the latter part of today have not been delivered at all.  Some are lurking in webmail folders various that can't be easily or readily accessed others are god knows where. And on top of that, all mail that did actually come in early on was also screwed up by PN and has been filed all over the place due to the lack of correct headers being added.

The end result is that I can't at the moment get hold of the missing data and I need to almost manually sort through 10's of 1000's of messages to find the ones that have been misfiled because PN screwed up all my filtering with the various errors they introduced this morning.  The likelihood of a timely update or even any update is just about as likely as a monster lottery win at the mo

[mikeb]

Another day, another c**kup it would appear

As can be clearly seen from the above graphs, all of my A/Cs various are well used, well monitored and routinely produce quite consistent results considering the very variable and totally unpredictable nature of what is being monitored. It would not therefore be in any way unreasonable to expect to see somewhere around a 100 messages per day turning up at the moment based on all of the available historic data.

Statutory Warning: Volume can fall as well as rise, past performance does not necessarily indicate future performance. If in doubt contact your advisers and all that

However, let's be very generous here and say that it should be somewhere between 50 and 150 shall we.  Despite all the b*ggering around I've had to do to try to put right all of the blatantly incorrect settings and changes various that PN managed to apply to all of my A/Cs due to yesterday's 'upgrade' clearly NOT BEING EFFECTIVELY TESTED PRIOR TO IMPLEMENTATION (*) a large proportion of my mail is apparently still going somewhere other than where it should be.  Yesterday, I had a grand total of 22 messages, a significant quantity of which were either bypassing postini or were test messages. So far today, I have a grand total of 3. Are either of those figures in any way reasonable to expect ... NO they're bl**dy well not so don't even think about suggesting that they might be  In addition to this, some genuine messages still appear to be conspicuous by their absence.

(*) Note To PN: Don't even bother to dispute or argue on this point either. No matter what you say, how you say it, who says it or how many times you claim this is not and never was the case, the evidence speaks for itself as it always does. Fundamental, obvious and very easily discovered problems were once again not found during any review or testing process that may have taken place prior to implementation. The only conclusion that any reasonable person can draw from this is that if any review and/or testing did actually take place then it was, at best, ineffective because far too many silly and totally avoidable issues found their way to customers.

I give up. Perhaps PN could advise if/when they have resolved any and all outstanding issues with the mail system and confirm that all A/Cs are now working 100% correctly as they all were prior to yesterday's changes. Until such time, I have better and far more interesting and productive things to do than continue playing Where's Wally my-missing-mail.

[mikeb]

PS. And now it gets even better.  One of the very few messages that I have actually received today, does not even contain the correct set of  headers from postini never mind get the correct action by PN.  And before you ask, YES, all my configuration via the portal is apparently exactly as it should be. Exxxxxxxcellent stuff.  Well done chaps. Is there no end to the incompetence ?

X-pstn-levels: (S: 0.00000/95.44534 CV:99.9000 )
X-pstn-settings: 1 (0.1500:0.1500) cv gt3 gt2 gt1
X-pstn-addresses: from <ErickslagJefferson@credit-land.com> [db-null]
Message-ID: <E1Jhb8h-00009v-CZ@pih-sunmxcore10.plus.net>
To:
X-pn-pstn: Spam 1
X-PN-Spam-Filtered: by PlusNet MXCore (v4.00)
Subject:

A picture always says a thousand words and all that:

[bpullen]

I have just found out that all my A/Cs mail options have been changed by PN without my knowledge so virtually all of my messages (genuine and spam) from the latter part of today have not been delivered at all.

*All* of them Mike? Presumably the option to deliver to mailbox was selected for all of these accounts? Has the setting since reverted back to its original value as part of our problem fix late yesterday afternoon?

Some are lurking in webmail folders various that can't be easily or readily accessed others are god knows where.

The email should be in your Webmail folder or Inbox. We're not aware of any missing email, although some may have been deferred (to be delivered later) on your non-Postini's account. It is easy to access this email via Webmail and bulk move it back to the Inbox as I have explained in other threads.

And on top of that, all mail that did actually come in early on was also screwed up by PN and has been filed all over the place due to the lack of correct headers being added.

Some will have come directly via the mx.cores but other than that the headers should be perfectly fine.

Despite all the b*ggering around I've had to do to try to put right all of the blatantly incorrect settings and changes various that PN managed to apply to all of my A/Cs due to yesterday's 'upgrade' clearly NOT BEING EFFECTIVELY TESTED PRIOR TO IMPLEMENTATION (*) a large proportion of my mail is apparently still going somewhere other than where it should be.  Yesterday, I had a grand total of 22 messages, a significant quantity of which were either bypassing postini or were test messages. So far today, I have a grand total of 3. Are either of those figures in any way reasonable to expect ... NO they're bl**dy well not so don't even think about suggesting that they might be  In addition to this, some genuine messages still appear to be conspicuous by their absence.

Mike, it would really be helpful if you could check the 'Spam' folders of your accounts TBH. Your reluctance to do so really isn't helping us diagnose your problem. I'm not going to argue about the testing element. You know what my opinions on that are.

I give up. Perhaps PN could advise if/when they have resolved any and all outstanding issues with the mail system and confirm that all A/Cs are now working 100% correctly as they all were prior to yesterday's changes. Until such time, I have better and far more interesting and productive things to do than continue playing Where's Wally my-missing-mail.

I'll be providing updates via the thread on Community.

PS. And now it gets even better.  One of the very few messages that I have actually received today, does not even contain the correct set of  headers from postini never mind get the correct action by PN.

I must be missing something here. What's wrong with the headers and why weren't they treated correctly?

Mike, I think the easiest thing for us to do at this stage is for you to PM me all of your account names along with what you think should be happening with the mail sent to them. I can then cross-reference this with the config on ours and Postini's systems and identify any disparities. From what you're saying you seem to be affected to a far severer extent than any of our other customers and at the moment I'm struggling to identify why?

[mikeb]

I was very much hoping to find that on my return almost a week after this started, all would now be well and the newly introduced problems would be fixed. Apparently not it would seem. I'm still seeing all the same issues that began last Thursday and service.status hasn't even been updated as planned in the meantime either.  So, it's back to beating Mr.Head against Mr.Wall it would appear to try to spell out exactly why I think this is wrong and that the problems were introduced solely by PN action last Thursday whereas PN obviously seem to think all is not that bad at all and in any case, no other customers are having similar problems - it's just me.


EXPLANATION OF PREVIOUS COMMENTS FOR THE NON-BELIEVERS

Every single mail received up until around 1200 last Thu has looked something like these 2 recent examples:

Delivery-date: Wed, 02 Apr 2008 22:04:17 +0100
X-pstn-levels: (S: 0.00000/53.65985 CV:99.9000 R:95.9108 P:95.9108 M:97.0282 C:99.7951 )
X-pstn-settings: 1 (0.1500:0.1500) cv gt3 gt2 gt1 r p m c
X-pstn-addresses: from <fkeqsh@My_Account.force9.net> forward (user good) [21/1]
X-pn-pstn: Spam 1

Delivery-date: Thu, 03 Apr 2008 11:39:10 +0100
X-pstn-levels: (S: 0.00000/54.71812 CV:99.9000 R:95.9108 P: 0.0000 M:97.0282 C:98.6951 )
X-pstn-settings: 1 (0.1500:0.1500) cv gt3 gt2 gt1 r P m c
X-pstn-addresses: from <qe@My_Account.force9.net> forward (user good) [21/1]
X-PN-Virus-Filtered: by PlusNet MXCore (v4.00)
X-PN-Spam-Filtered: by PlusNet MXCore (v4.00)

as indeed they have done since just about day 1.  Not necessarily with a forged whitelisted address of course although the majority did last Thu but that is not particularly unusual in itself.  Also, up until this time, I was receiving the sort of quantity of messages that I would expect to see. Although, it was somewhat unusually lower than normal for around midday and quite noticeable that most messages had been received prior to 0600(ish) with only a very few thereafter. The 2nd example above just so happens to be the very last message I received before things apparently changed once again.  Note that header changes are already apparent by now. The missing SPAMx header is a well known error that PN had introduced by this time of course.

All subsequent messages have looked something like this:

Delivery-date: Thu, 03 Apr 2008 12:19:55 +0100
X-pstn-neptune: 51/50/0.98/87
X-pstn-levels: (S: 0.59202/99.77803 CV:99.9000 )
X-pstn-settings: 1 (0.1500:0.1500) cv gt3 gt2 gt1
X-pstn-addresses: from <shedricknf@wwdb.org> [db-null]
X-pstn-neptune-cave-rslt: qtine
X-pn-pstn: Spam 2
X-PN-Virus-Filtered: by PlusNet MXCore (v4.00)
X-PN-Spam-Filtered: by PlusNet MXCore (v4.00)

and not only are they clearly quite different but messages in general were also very conspicuous by their absence after this point. Virtually no messages were received after around midday.  In addition to that, those that were received were all false negatives for one reason or another.

Portal configuration was checked and put right around midnight after it became apparent that yet another PN error had led to them being changed to inappropriate values. Despite this action, last Fri saw a continuation of the trend from the previous day. Virtually no messages being received at all and those that were received were false negatives.

Much the same trend again for last Sat excepting for the fact that a small deluge of NDAs was received during the day. This is the first time ever (in ~10 years) that NDAs as a result of one of my addressees being forged in a spam dump have been received. It's almost certainly pure co-incidence that it happened at this particular time, although there is a slim possibility that it's not I suppose. Configuration options were checked, found to be correct but 'updated' once again anyway but still the problems have continued to date.

Incidentally, postini appear to be completely useless at handling NDAs. They could and should detect them and act accordingly. They could and should detect those that are being received solely as a result of a forged address being used when sending spam in particular. Spammy NDAs could and should be rejected because this is so easy to do very reliably in the vast majority of cases. Only totally genuine NDAs (plus the occasional false negative) should ever be delivered to customers. Sadly, postini don't bother to do anything remotely sensible it would appear so customers can expect virtually unlimited quantities of NDAs should one of their addresses ever be used by Mr.Spammer. I rather suspect that postini will take the easy option at some time and simple dump all NDAs regardless of how genuine and wanted they might be based on their current attitude regarding false positives in general   NDAs are being manually checked and discounted from my data where appropriate because of the severe distortion that would result otherwise.

So, to summarise,

From day 1 until around 0600 last Thu: Just about as expected for most of the time with one or two surprises along the way.

Between 0600 and 1200 last Thu: Uncharacteristically low number of messages being received. Changes started to appear in headers.

After 1200 last Thu: Further header and PN/postini configuration changes were apparent, virtually no messages were being received at all with those few that were received being false negatives. This has continued ever since.


Now tell me that the above sequence of events doesn't tally almost exactly with PN changes taking place and that there isn't something very odd apparently going on as a direct result thereof !!!


But just before you do, let's look at some cold hard evidence shall we just to confirm the situation beyond any doubt whatsoever.  Those users familiar with this and previous threads with my weekly spam stats that have been produced over the last year will no doubt recognise this particular chart which has been detailing postini performance (or lack thereof) since January:



Any doubts now that problems became immediately apparent the very moment PN upgraded the system ? Nope, thought not.

Oh yeah, and talking of problems introduced last Thu, it would appear that whitelisted spam is still sailing straight through untagged despite being blatantly obvious forgeries and not being sent via the PN network.


Now finally, let's address the statement here and elsewhere that it's just me and no other customers that are affected. Really ? It doesn't look that way to me after a very quick scan through recent posts on the PN forum. It seems to me that other customers are also not seeing the configuration they expect happening in reality as I guessed would be the case. I was sort of expecting to be part of the customary "very small number of customers affected" but even I was surprised at the initial off-hand suggestion of being unique here. It is always difficult for me to believe the "very small number of customers" line in most instances anyway simply because I always seem to be one of the "very small number of customers" no matter what the problem is or when it occurred. However, whilst I acknowledge that it is technically possible that these problems are only affecting my A/C, I would suggest that it is just about as close as it is possible to get to implausible for that to be case. >200K customers and the only one of them being affected just so happens to be the only customer who is monitoring PN/postini performance in great detail and on a regular basis so can spot problems immediately ? I really think that there's a greater probability of winning the lottery than that being correct TBH.

I would very much suggest, as I did last week, that these problems are very much a PN problem and they were introduced at the same time as the 'upgrade' and it does affect at least some other customers in one way or another.  All the evidence points to some fundamental problem with the way PN is setting (or rather not setting) the postini and/or PN configuration. At best it is unreliable or at worst it simply doesn't work.  The postini configuration in general has changed subtly from what had existed prior to the upgrade for absolutely no obvious reason and user changes via the portal are not necessarily being acted upon correctly. The end result is that the portal shows one configuration whilst PN and/or postini is merrily doing something quite different. Who knows which options work and which don't or which bits of configuration data are being updated correctly and which aren't and whether it's consistently wrong or intermittent. But the fact is that some options/configuration aren't working and it could apply to any/all options and as suggested last week, this problem simply has to be affecting more users than just me ... and it is of course.  I trust that PN will eventually appreciate this and unlike previous similar reports such as different customers having different postini configurations, not simply ignore the issue because "it couldn't possibly happen" ... and then quietly fix it several weeks later without making further comment. Maybe it's already been mostly fixed, I don't know because I haven't been here watching but what I do know is that at least one of my A/Cs is still acting very strangely, the other being a bit suspect.

Bob, as requested I will PM you details of one A/C that is particularly affected by this so you can see for yourself first hand. HOWEVER, at the risk of sounding just like your mum, remember that you look with your eyes and not with your fingers !!  Please DO NOT go tweaking my A/C settings in any way or go playing trial and error somewhere. The object of the exercise is to see what's going on, go away and find out why and then fix it properly not to generally fiddle about until the problem sort of goes away by itself. Apart from anything else, with several days worth of stuff to catch up on and shedloads to do in any case, I simply cannot deal with several thousand messages suddenly appearing right now, especially as most if not all of them will be incorrectly tagged or have incorrect headers and thus will not be automatically handled, collated and filed correctly on receipt as they usually are.  Once PN have resolved the problems and everything is once again working as it should be, THEN is the time to discuss how to recover from the situation and find/deliver all the data that has been lost over the past week.

I'm not going to be looking at this thread or elsewhere for that matter for a couple of days or so but I would very much like to actually find some GOOD news when I next return along with some GOOD news on service.status for a change.  It's now a week since these problems were introduced after rolling out an allegedly tested upgrade. If it can't be fixed in a week then it clearly wasn't a minor bug that somehow managed to remain undiscovered during the review/testing phase.  Also, a full explanation for why the postini configuration was fundamentally changed at around midday last Thu wouldn't go amiss either seeing that the roll-out took place something like 6 hours prior to that, bug fixes only after that presumably.

[mikeb]

Hmmmm, no response. Strange  Anyway, the graph in the post above will update at around midnight tonight to reflect the data for the week. It's not going to look that much different TBH, just more of the same and no doubt slightly different percentages.

I also looked at my configuration via the portal early this morning and it's now different again   I now appear to have one A/C alias showing correct configuration and two that show what is presumably the PN default configuration.  This is very strange because that's certainly not how it was in the beginning. When I first changed the settings sometime just after midnight last Fri morning, I wasn't sure what they should be in order to go back to the way it always was. Mr.Jelv advised that "inbox" would have the desired effect for routing but I didn't quite 'get' the "domain" bit at first TBH.  I don't have any domains as such, only sub-domains/aliases however, I checked the "all domains" box anyway when I updated the settings on this occasion on the principle that it couldn't possibly do any harm.  When I logged in again to check other things a bit later on, it finally dawned on me that "domains" included sub-domains/aliases and when I viewed the setting for each of them they were all correct and identical. However, when I checked the settings on subsequent occasions, I didn't check each one independently, I only looked at the first page that comes up and noted that it was as expected. I also have a rather nasty suspicion that when I subsequently "updated" the settings (without actually making changes) I didn't check the "all domains" box either.  I just looked at what was shown on the first page, hit the "update" button and then logged out. Is it possible that some settings were returned to default again at some point by one of the PN bug fixes along the way ? Or is it possible that "updating" one lot of settings somehow managed to default the others ?

What's more than a bit odd you see, is that I would expect to see several messages/day to username.force9.co.uk and several 10's messages/day to username.force9.net and none to the 3rd alias. I have received none at all since midday last Thu on force9.net and virtually none anywhere else either but I did actually receive one single message to force9.co.uk this Wed and this has been followed by a slow trickle daily since then.  So that's none at all since the roll-out until this Wed and a regular but small quantity thereafter. And just in case you're thinking otherwise, no, I'm not in any way suggesting that this is a result of me giving you the A/C details etc. because the first message was actually received prior to that.

In addition, I also sent a deliberate spammy test message to force9.co.uk, force.net and elsewhere at the end of last week after the settings had been reset to 'normal' via the portal. The message was delivered elsewhere but NOT to force9.*  I sent the very same message again early this morning and on this occasion it was delivered to force9.co.uk and elsewhere but NOT to force9.net.

Unfortunately, because I haven't been able to spend enough time monitoring exactly what's been going on and conducting 'proper' tests over the last week, it's very difficult to draw any sensible conclusions and provide hard evidence to support them. Whilst it's absolutely clear that it all went horribly wrong around midday Thu 4th, it almost looks like something might well have changed again early(ish) on Wed 10th as well.  Bearing in mind that configuration settings were first altered by me early on Fri 5th and "updated" without changes twice between then and late Sat 7th but haven't been touched since, this is more than a bit strange to say the least.

I quite deliberately haven't (intentionally) tweaked anything at all since Fri 5th because I knew I wouldn't be able to monitor what happened and deal with any possible consequences. However, I will be changing them very early on Sunday (i.e. shortly after midnight) though. I will once again restore all settings to what they should be and ensure that this has actually been applied to "all domains" by logging out and returning sometime later to check each one independently. I will be unhappy in the extreme if ( A ) I still don't get messages as expected or ( B ) the settings mysteriously change again at any time in the future !!!

Now that I will hopefully have a little time available to investigate properly, I will also bung a shiny new disk in one of Ye Olde PCs, install a basic OS from a backup plus, VERY reluctantly, a 10 year old copy of that bug-ridden insecure pile of poo otherwise known as Outhouse Express to have a hunt around for missing stuff various plus anything else that might be of interest.  I don't know whether I'm going to be able to sort it all out sensibly or not because whilst the PN tagging/header issues may possibly have been fixed by now, there will almost certainly be several thousand messages with most of them being a right old PITA to deal with. One thing is for certain though, I'm not going to waste shedloads of my time on this. If things can be recovered and manually processed fairly easily then they will be but if not then I'll just dump the whole jolly lot and give it up as a bad job. It simply isn't worth my while poncing around anymore with something that still isn't stable and isn't actually something of vital importance to me personally in any case.  There is only a reasonable amount of time that I am prepared to invest in worrying about someone else's problem.

[mikeb]

Quick update: I seem to have recovered most of the 'lost' data and after mucho b*ggering around over several hours got it where it sort-of should be. I haven't worried too much about the detailed analysis but I don't think the fundamental figures on the graphs are going to have any significant errors and that's all that really matters. It now looks pretty much as expected rather than nose-diving towards zero. I have also excluded any remaining NDAs that appeared from the deluge that turned up last weekend because of the severe distortion to figures that results otherwise.

Regrettably, I'm not going to be able to post the graphs this weekend so the next update will hopefully be Sunday 27th or shortly after. If the trend that I've noticed over the last few days continues then there will be some 'interesting' graphs coming RSN. There's nothing quite like ending with a good old tease and watch-this-space stylee comment is there