next »

[dtomlinson]

Will check tomorrow, I think it's being planned for this month though.

[bpullen]

It's planned for next Wednesday IIRC.

[bpullen]

The next generation email platform changes (ie. the catch-all mailbox changes) have been put back due to a misunderstanding between the project team about a certain aspect of the specification (nothing major).

The provisional date for roll-out is now April 4th.

It may also be worth me mentioning that we will not be withdrawing existing customers' catch-alls to begin with although we may do this further down the line.

Regarding the email platform on an architectural level, all email is now off the StorageTek (yippeee!) however it's still being used for the quarantining of virulent emails although I'd imagine this will be moved away at some point too.

All our customer email is currently stored on a NetApp FAS940 system.  We have since purchased a replacement FAS3070 system that arrived in Sheffield last Tuesday. It's been racked and configured and we're now in a good place to start moving customers' data across either late this week or early next.

We'll obviously minimise the impact this will have for customers but there's a lot of data to move across and it's likely to take a month or so to do this with a few minutes downtime for each user.

I'll try to get a more formal update posted here and to the portal forums and newsgroups etc. later today. Service Statuses and Planned Maintenance will of course also be posted in due course.

[Oldjim]

Bob,
Appreciate the update.
As part of the new system will there be a simple route through the portal for disabling the catchall without needing to set up a separate postmaster mailbox

[lmartin]

Affermative.  You'll have access to the new "Manage My Mail" tool which will allow you to switch-off your catch-all / add aliases etc...

[Penny]

The next generation email platform changes (ie. the catch-all mailbox changes) have been put back due to a misunderstanding between the project team about a certain aspect of the specification (nothing major).
The provisional date for roll-out is now April 4th.
It may also be worth me mentioning that we will not be withdrawing existing customers' catch-alls to begin with although we may do this further down the line.

It occurs to me to ask, at this point, whether a dedicated mailing will be going out to all customers, advising them of the proposed changes to the default (catch-all) mailbox set-up.

Given at least 50% of all customers will be non-techies like me, would it be possible for such an e-mail (IMPORTANT CHANGES TO YOUR E-MAIL SET-UP) to carry *specific* instructions like the ones jelv provided for me in the "Mail Avalanche" thread - post at http://usergroup.plus.net/forum/index.php/topic,3915.msg48834.html#msg48834 .

Getting the (now in use) mailboxes in place was, for me, a fairly major procedure [I really struggle even with simple "instructions"], but (eventually) I fathomed out what was required, from what John said, and now it's all *so* straightforward ....

Just strikes me that very many customers will need/welcome a very simplistic set-of-instructions, in the e-mail itself, to ensure that
(a) they're aware they actually have to "do" something,
(b) they actually do what is required to safeguard their mail longer term
(c) they know exactly *how* to do it.

Not saying the "portal guides" aren't okay (but I didn't understand/use them previously) but it's perhaps the right "time" for bringing the whole issue to (all) customers' attention with a very specific and info-rich mailing.

With the provisional roll-out on April 4th, and the likelihood of withdrawing all customer catch-alls in due course, now might seem like a very good time to "grasp the nettle" and at least begin to get customers to start getting their specific mailboxes in place, and/or allow plenty of time for them to ask for further info if they don't understand quite how to do so.

Regards,

Penny.

[lmartin]

Penny,

At this time, we'll not be making any changes to mailboxes of existing customers.  For this reason, we're not emailing them.  Customers will have the option to remove the catch-all if they wish, and we'll be mentioning email in the next newsletter.

Cheers,
Liam

[Penny]

With the provisional roll-out on April 4th, and the likelihood of withdrawing all customer catch-alls in due course, now might seem like a very good time to "grasp the nettle" and at least begin to get customers to start getting their specific mailboxes in place, and/or allow plenty of time for them to ask for further info if they don't understand quite how to do so.

Penny,
At this time, we'll not be making any changes to mailboxes of existing customers.

Yes I already understood this, as above.

For this reason, we're not emailing them.

I might be "technically dense" but far from dense otherwise :/

Customers will have the option to remove the catch-all if they wish, and we'll be mentioning email in the next newsletter.

I guess I'd just hoped for a pro-active rather than re-active stance.  If it's considered that "mentioning email in the next newsletter" is all that will be necessary to alert all the non-techie customers and prompt them into the required action, my post above was clearly a complete waste of time

Regards,

Penny.

[northbritish]

It might be a case of spreading the support load on CSC.  Implementing the changes will probably lead to an increase in support calls on it's own.  E-mailing everyone would probably cause strain on the CSC resources as people all got in touch for help at the same time.

If new signups are put onto no catchall by default then churn, added to other customers becoming aware of the option in their control panel will, over a few years, mean most will no longer have the catch-all mailboxes.  Support calls will be spread over a longer timeframe and so not a problem to manage.  When that is the case they can be withdrawn with much less impact on the customer service center.

[Penny]

If it's considered that "mentioning email in the next newsletter" is all that will be necessary to alert all the non-techie customers and prompt them into the required action ...

... E-mailing everyone would probably cause strain on the CSC resources as people all got in touch for help at the same time.

I can see where you're coming from, but AFAIK mailings-to-all-customers are usually staged over quite a long period (I can recall, in the past, even a single edition of "Plus!" taking several weeks before all the batches were sent out) so I don't know that this would necessarily apply.

If new signups are put onto no catchall by default then churn, added to other customers becoming aware of the option in their control panel will, over a few years, mean most will no longer have the catch-all mailboxes.

>>"over a few years" .... with the speed at which changes are happening at PN post-takeover, I suspect the loss-of-catchalls might happen rather sooner than that.

>>"other customers becoming aware of the option in their control panel" ... It's this bit I have the difficulties with.  Yer average Joe Bloggs (like me) knows very little about the mechanics and is very hesitant to change "anything".  If it can be got across by a directed, specific e-mail, that making these changes will radically improve "the PlusNet e-mail experience" then that's no bad thing, and many people would be inclined to try it out (and John's explained-way-of-doing-things provides the safeguard of not messing up, in the initial stages).

Losing the catch-alls has to be the way forward, and I doubt that "including it in the newsletter" is going to be sufficient to prompt action from the bulk of customers (assuming most people even read "Plus!", which I doubt, somehow).

I'd concur that "prompting customer action" can be deferred - but it does need to be done (in a targeted way) at some stage, and with a long lead time to give the less-technical people time to get their act together [quite aside from extended holidays likely in summer months].  For many customers the need to set up mailboxes will seem a "big change" and I guess I'd just prefer to see the groundwork laid thoroughly with the userbase at large.

I fully understand that PN will handle things however they see fit :/  my original post was just by nature of a "heads up" to (hopefully) avoid a bottleneck somewhere further down the track.

Regards,

Penny.

[bpullen]

The latest on the mail storage platform as posted across the community...

EMail Platform Storage Update

Hi all,

Whilst things have been pretty stable for a while, it's no secret that the last year or so has seen more than its fair share of email problems. It's certainly been a sore point for many customers and despite plans last year to overhaul the platform, a number of circumstantial factors combined with some ill advised decisions saw things take an unfortunate turn for the worse.

The purpose of this post therefore is to provide some detail regarding the problems we encountered, what we did to combat them and what our long term intentions are for the future of the email platform.

Background

Before going into too much detail regarding our plans for the email platform it helps to have a basic understanding as to how the email platform is built. All our email is separated across storage volumes. Each volume contains a certain number of customers' mail directories and is mounted to an area of physical storage that contains all the email for these directories.

These volumes are used for the storage of all customer email. Whilst the mail delivery servers (mxcores) handle and are responsible for getting mail from A to B, they are no good without there being somewhere to store the mail whilst it is awaiting delivery, or once it has been delivered to a customer's mailbox and is awaiting collection.

The StorageTek

Early last year we were using a Network Appliance FAS940 for all of our mail storage requirements. This implementation worked well. We were using kit our engineers were very accustomed to and the platform was reliable, trustworthy and suited its purpose well.



Despite being satisfied with the implementation, scalability was a concern. Email was on the rise and the volume of spam in circulation had hit new heights and was still rising.

As we began to run out of disk space on the existing storage implementation, it became clear that action was required. The decision was therefore made to rebuild the mail platform in order to future-proof our email service and allow for future expansion in line with both mail volumes and customer numbers.

Last Summer we spent £170,000 purchasing 2 Sun 5310 head units, each with 3 trays of disks.



With this kit in place we started looking at moving email from the NetApp and onto the new StorageTek.

Trouble at the Mill

Those who were customers at the time will undoubtedly remember the problems we had with the migration of mail onto the StorageTek last July which saw the unfortunate loss of some customers email. Sadly this was to mark the start of a number of reoccurring problems we encountered with the StorageTek over the following months.

In August we began the migration of customer email onto the StorageTek and by September had moved all Free-Online customer email and most Force9 email.

It was around this time that the reoccurring mail problems started manifesting themselves in the form of repeated mail delays and mail collection problems.

The reoccurring problems last year were often caused by the same things. Either we'd be hit with excessive volumes of mail (spam bombs) or one of the volumes on the StorageTek would become unwriteable. When the latter happened it caused immediate problems where the mail queues would start increasing. This caused email delays and had a knock on effect on the mail collection servers. This resulted in timeouts and errors when customers tried to pick up their email.

The reoccurring problems threw a spanner in the works and the mail migration was consequently halted. This left us in a position where email was spread across both of our storage volumes. This still allowed for problems and it became clear that further action was required.

Upgrading the NetApp

It was clear at this point that the new platform was not fit for purpose and that we had made a poor decision. After careful consultation and a series of informed discussions we decided that we needed to move away from the StorageTek kit.

Late January we made a positive step in the right direction by scaling the original NetApp FAS940 storage platform.



This additional storage space was beneficial for a couple of reasons. Firstly it provided us with the space to move most of our customer email off the 'New' Sun StorageTek and back onto the tried and tested NetApp platform. Secondly it gave us breathing room to deal with the continuing increase in email volumes.



Obviously this expansion was a good thing, but was only the start of the plans we had for the mail platform.

Migration Back to the NetApp

With the additional storage in place we were almost in a position to get all customer email back on the original NetApp.

The last time the problems with the StorageTek occurred was towards the end of February as detailed here. This affected the last of the email left on the platform and provided an opportune moment to move away from the problematic system once and for all!

New kit was delivered as a priority to ensure we could facilitate all customer email. Our engineers worked around the clock to install and configure the new equipment and before long we were transferring the last of the email onto the NetApp.

So where does that leave us..?

All email is now on the NetApp FAS940 with the StorageTek being used to quarantine virus emails (although it's possible that this will be moved away at some point too).

This means that customers will never see a repeat of the problems witnessed over recent months and during the latter part of last year. This is fantastic news but there's more yet!

The New Email Platform

Whilst the NetApp FAS940 storage system has been functioning with few problems, available storage space and scope for future expansion is still a concern so the decision was made to engineer a replacement.

I'm pleased to say that we've pulled out all the stops when writing the specification for the platform. We recently parted hands with almost a quarter of a million pounds which has bought us a shiny new NetApp FAS3070 system that landed on our doorsteps last Tuesday.

This equipment has since been fully racked and configured and we're now in an ideal place to start moving customers' data across to the new platform.



There is of course a risk involved when migrating mail from the old NetApp to the new. This is something we're only too aware of from experience so customers will be pleased to know that we will not be removing any email from the older platform until a month has passed following installation of the new system. This gives us something to fall back on should the unspeakable happen.

Due to the fact that all customers' data is being moved this work is likely to take a number of weeks. During this time each customer will experience short period of time during which they will be unable to access their email. The length of this service interruption is dependent on the size of each customers' mailboxes and will typically last no longer than 10 minutes. Some customers with very large mailboxes may find this service interruption to last slightly longer. Every effort will be made to keep this downtime to a minimum and we will announce details regarding when customers can expect their data to be moved where  possible.

There's a Service Status I posted regarding the maintenance work here.

We'll be trying to provide frequent updates regarding the work over the coming weeks via Service Status and the forums. I sure people will see this as a big, big step in the right direction for the provisioning of the email platform. I for one am confident that we'll see a return to the reliable and stable days of old.

We've already overhauled our mxcore mail delivery platform with a £120,000 re-fit and if that's anything to go by then the future of email is looking bright

Feel free to discuss the plans and ask any questions you may have. I'll be more than happy to try and source the answers.

Kind Regards,

[Penny]

we're now in an ideal place to start moving customers' data across to the new platform

Appreciate the very long and detailed overview

... just a tiny bit wary about losing any more mail [I understand the bit about the back-ups].

Given my own experiences so far this year, could I perhaps request that someone at your end keeps a *very* pro-active eye on mailboxes actually *working* on a continuous basis?  [ie all customer mailboxes throughout the whole transition period]

Really can't afford to lose any more mail (or have it delayed) in respect of several areas of the site right now, particularly as I have major time commitments elsewhere.

Regards,

Penny.

[Oldjim]

Bob,
You shouldn't have posted that - email timing out again 

[bpullen]

Sod's law isn't it

I had a quick look at the problem from home yesterday but I'm not sure if we know what caused it. Will have a word with Josh when he gets in.

[bpullen]

OK guys,

Following the problems experienced over recent days, we'll be adding another 4 mail collection servers this evening.

We also hope to have another 2 Solaris boxes installed at some point tomorrow. That will have doubled the number of mail collection servers we have following the recent problem with time-outs.

We are also looking to address the spam issues.

The next generation email project that allows existing customers to remove their default mailboxes is now expected to roll to live on 18th April. This also changes the fashion in which email accounts are created on signup by providing a single address and postmaster alias instead of the default 'catch-all' address we have provided up until now.

It is also probable that we will be making further changes to the mail platform next week to start dropping spam email that is sent directly to the mx last servers.

We cannot be 100% as to the exact cause of the recent problem but strongly suspect the cause to be related to nfs/spam/mail directory size problems. We are addressing all of these points and will also be kicking of the migration over to the new NetApp at some point tomorrow all being well.

Kind Rgds,