next »

[Simon Day]

Hi Fenland,

I completely understand the importance of being specific about what you are delivering and when. With that in mind, I will have the first of the more specific postings I mentioned, out by Tuesday next week (22nd). It will contain more info about the Mail storage platform project we have on going.

Simon

When I got in this morning and started looking into the plans in more detail, the implementation was scheduled for next week, so I have already posted the specifics, see here for more info: http://usergroup.plus.net/forum/index.php/topic,2921.0.html

[wildmind]

Backing up a mail platform as busy as ours is never going to be an easy task, with the amount of transient data and the sheer volume of data requiring copying to another location. However there is a debate going in internally about back-ups in general, what we should back-up, how often and to what media and as that debate progresses I will share with you the outcomes.

Simon - any chance of having an input into this (either publically or privately)?

I really think that even a once a week snapshot of the email platform would go a long way to appeasing users fears. Nobody expects recovery of single items for single users (well, nobody rational anyway) - and likewise, people shouldn't expect 100% recovery... if they do they need to think for a while

If there were weekly snapshots (although nightly should be possible) and the mirrored replication worked effectively (e.g. with safeguards against the measures we saw recently) then the platform would be more secure, the users happier, and your stress levels eased

[ianwild]

I still want to know how email that is a week old is valuable at all - Your asking us to back up a couple of Terrabytes of aged spam (and a few odd genuine mails) on a high transaction system, when that isn't a done thing in the industry - It really doesn't make sense to me Mike.

We might be convincable, but checkpoints is the way forwards for me, as it not losing any mail in the first place.

Ian

[wildmind]

As an ISP users expect a resilient platform, with due care and attention being taken - neither was the case in the recent disaster.

Those users who were on holiday (of which there were lots) or those who believed an ISP wouldn't lose all their customers email would have been very grateful for a week old backup - as I'm sure the shareholders and the share price would have been!

PlusNets reputation really sunk to a new low with the email fiasco - a backup, even a week old, would have prevented a lot of that.

[ianwild]

Oh yes - I see that, but the fiasco over email wasn't down to us not having a back-up - That would have helped, but so would outsouricing all our mail handling to message labs...

A back-up isn't a solution for a resiliant and unbreakable email platform - A properly designed and supported email platform is the answer!

How are we doing with the email campaign - I'm busting to get itno answering these difficult tickets.

Ian

[MauriceB]

Oh yes - I see that, but the fiasco over email wasn't down to us not having a back-up ...

Oh yes it was!

A back-up isn't a solution for a resiliant and unbreakable email platform - A properly designed and supported email platform is the answer!

But a sensible back-up policy is an integral part of a "resilient and unbreakable email platform" ?

I've been in the game a long time and like the paperless office - the unbreakable system ain't here yet!

Mirroring and fail-over are fine until S**t happens, then you need the backup?

Maurice

[simonflood]

I still want to know how email that is a week old is valuable at all

Because PlusNet allows customers to access their e-mail via IMAP and WebMail, e-mail gets left on PlusNet's servers.  Customers therefore expect that e-mail to remain there.  If PlusNet only offered e-mail access via POP3 then customers would have all their e-mail automatically downloaded locally.

However if PlusNet were to stop offering IMAP and WebMail access customer would leave for another ISP that could offer these.

Your asking us to back up a couple of Terrabytes of aged spam (and a few odd genuine mails) on a high transaction system, when that isn't a done thing in the industry

We're not asking PlusNet to back up the spam - if you know it's spam why keep it?  Or are you referring to customer's filtered spam messages?  I've got the anti-spam filtering enabled yet never get any messages in my Spam folder.  I can't believe I'm not getting any spam.

as it not losing any mail in the first place.

Exactly!  Customers trusted PlusNet not to lose their e-mail. Now that trust has been broken, implementing a backup system is a way to regain some of that trust.

Simon

[fenlandbroadband]

I still want to know how email that is a week old is valuable at all - Your asking us to back up a couple of Terrabytes of aged spam (and a few odd genuine mails) on a high transaction system, when that isn't a done thing in the industry - It really doesn't make sense to me Mike.

I'm not fully clued up with the Data Retention Law that I think "Europe" has proposed, but within about the next year aren't ISP's meant to keep all data for a period of time?

In fact, to quote this BBC News article:

EU countries will have until August 2007 to implement the data retention directive.

Under the legislation, internet service providers and telecoms operators will have to keep details of their subscribers' communications for up to two years.

The measures will require firms to keep records of all phone calls and internet communications for a period of six months to two years, although the content will not be recorded.


How long does PlusNet keep logs of user activity/communication currently? What are you going to do in 12 months time when this (apparently) becomes law - are you going to have to respec and rebuild your mail system yet again, possibly wasting all of the time/money/resources put in to the rebuild that is ongoing currently? What's included in the current spec/plan which will work with making it suitable for this possible two year data retention, or will the system being put in place simply not be able to manage with that?

[Simon Day]

I'm not fully clued up with the Data Retention Law that I think "Europe" has proposed, but within about the next year aren't ISP's meant to keep all data for a period of time?

In fact, to quote this BBC News article:

EU countries will have until August 2007 to implement the data retention directive.

Under the legislation, internet service providers and telecoms operators will have to keep details of their subscribers' communications for up to two years.

The measures will require firms to keep records of all phone calls and internet communications for a period of six months to two years, although the content will not be recorded.

There is still a lack of clarity around this legislation, for example, who is going to hav to pay for the increased stroage requirements the law apparently enforces on companies such as ourselves. We are working closley with the government, along with other ISP's to get to the bottom of the exact requirments and details of the legislation. To be honest, right now, I don't think they (the government) have a clue.

How long does PlusNet keep logs of user activity/communication currently?

We complie with the law on retention of data for radius data, who was on-line, when and with what IP, sorry but I can't rememeber what the specifics are, but I do know we meet the requirments.

What are you going to do in 12 months time when this (apparently) becomes law - are you going to have to respec and rebuild your mail system yet again, possibly wasting all of the time/money/resources put in to the rebuild that is ongoing currently? What's included in the current spec/plan which will work with making it suitable for this possible two year data retention, or will the system being put in place simply not be able to manage with that?

Well, as I have stated above it is not clear exaclty what will be required of us so it is impossible to spec a system without knowing the requirements. I beleive that as you have stated above, we may be required to keep records of who sent something to whom and when, but not the what. Leaving out the what requires a lot less storage of course.

The new system is designed to be able to scale quite considerably, but I don't think that we would use the mail platform to store the data we are talking about here. It would probably pulled off there and stored somewhere on an external media, but really I am just guessing here and will be until such time as we know wha we really have to comply with.

[Ultra]

if PlusNet were to stop offering IMAP and WebMail access customer would leave for another ISP that could offer these.

Not that I've been looking hard, but do *any* other UK ISPs offer e-mail with IMAP access ?  Most I've seen seem to want to charge fees per extra mailbox, though there are a few with a similar method of setup to PN so anything@account.ISP.name will work.

I've got the anti-spam filtering enabled yet never get any messages in my Spam folder.  I can't believe I'm not getting any spam.

OTOH, I've never knowingly enabled any spam protection, but in 3+ years don't recall ever getting any spam.  Mind you, my accountname.plus.com was switched to an IP address for the reverse DNS lookup early on (and since all web sites have the potential to log where visitors come from, there could be thousands of servers which only need put random@name.plus.com  to get spam to most users...

[mikeb]

I'm not fully clued up with the Data Retention Law that I think "Europe" has proposed, but within about the next year aren't ISP's meant to keep all data for a period of time?

EU countries will have until August 2007 to implement the data retention directive.

Under the legislation, internet service providers and telecoms operators will have to keep details of their subscribers' communications for up to two years.

I think the key thing is keeping details but NOT the content hence the storage requirements will be sensible although the system used and its reliability will obviously be of major importance to ensure compliance.  Seeing PN waffling on about storage requirements and other (relatively) minor details of the proposals is worrying to say the least.  I can just imagine the amount of buttock clenching that would have been going on at PN Towers though had this legislation been in force now.  With the amount of data apparently being lost recently, they would have definitely have had to suffer way more than a few customers jumping up and down that's for sure !  I wonder what the penalties for non-compliance will be ... let's hope they're extreme as it might just force providers to consider the consequences of data loss and take 'reasonable' care to prevent it rather than pointing to T&Cs and taking no responsibility whatsoever.

OT: Every time I see one of your posts, I always read your username as FenlandBrewery, generally followed by a Simpson stylee Mmmmmmm, beer with much drooling !  Having just spent the last several days on my annual pilgrimage for virtually 24/7 consumption of very significant quantities of their jolly fine beverages along with similar offerings from a good many other equally fine breweries, they are, unlike PN at the mo, most definitely to be recommended   Seriously missing it already though and it's not even 24 hours since the last pint and around 364 days to go until the next. B*gger

[devnull]

The plan is to replace the 24 Intel boxes with 4 clusters of 4 x Sun T1000 servers and 1 Sun 3510 Fibre Channel Disk Array.

Just out of interest, I was wondering why you are using a single minnow per cluster? Also, I'm guessing you ruled out HDS or EMC based upon cost?

[fenlandbroadband]

OT: Every time I see one of your posts, I always read your username as FenlandBrewery, generally followed by a Simpson stylee Mmmmmmm, beer with much drooling !

 

[Ultra]

let's hope they're extreme as it might just force providers to consider the consequences of data loss

One option is to simply tell customers, as was done with binary newsgroups, that "From January 1st, 2007, PlusNet will no longer handle e-mail for customers." and suggest some alternative services from which that facility can be obtained (also done regarding newsgroups).

It's a fairly drastic one, but the shareholders would be happy, knowing there would be no need to buy more storage, worry about backup of mail, or be responsible for any logging under this EU directive

Even the staff would be happy... after all, they'd have oodles of storage for downloading movies etc, etc, all done automatically with cron jobs, and just pop in to burn new DVDs each day 

When the RIP (Regulation of Investigatory Powers) Act was mentioned in discussions with another ISP, concerning some "black box" logging devices to pass data for selected users to the Authorities, one option that ISP suggested was to move their mail service abroad.  (The systems guy, now MD, also mentioned that the day there is no answer about whether and "black boxes" have been installed, we'd be able to tell that yes, they had been installed and were potentially operational!)

(As it happens, one of the places my mail gets stored is outside the EU - and not in N America, either, though I've nothing to hide, and don't use encrypted mail anyway!)

[Simon Day]

As a result of your feedback about the back-up of the mail platform, and the internal review that has been  going on, we have identified a possible backup solution, an attached SCSI tape library that is apparently capable of backing up the mail platform in 15/17 hours. We are investigating the real-world performance and will see where we get to.