In a bid to make sure this threads up to date, I guess it makes sense to provide the update that was posted across the portal forums and newsgroups recently...
As some of you are no doubt aware, the last 18 months has seen a long series of problems on the email platform. All of the problems are in some way or other related to the email storage platform, either because there were problems specific to the Storage itself, or because we were busy trying to fix storage so weren’t focused on the other areas when we should have been.
As has been covered extensively before, when we were running out of space on the mail storage platform, we made a poor decision, didn’t scale the NetApp and purchased a poor substitute.
Last Wednesday morning however, we made a positive step in the right direction by scaling the NetApp storage platform. As you can see from these graphs we have significantly increased the storage capacity:
This additional storage space is beneficial for a couple of reasons. Firstly it provides us with the space to move all the mail remaining on the “New” Sun Storetek mail storage platform back to the tried and trusted NetApp system. Secondly it gives us breathing room to deal with increased mail volumes over the coming months. Obviously this expansion is a good thing, but it is only one part of the plans we have for the mail platform. While storage has been one of the key issues we have had to deal with, it is not the root cause of the problems. That honour goes to your favourite and mine, spam!
spam is an issue on the rise. The increase in the volume of unsolicited email is becoming headline news, not just for technology news web sites, but for mainstream sites and newspapers as well. The common themes in all the articles are that the amount of spam flying around the Internet is increasing; that 2006 was the worst year for it so far; and that this Christmas period was the worst of all. This view is backed up by the information that we gather from our own systems. We can clearly see that the volume of mail being processed by the platform increased by about 55% in the 2 weeks before Christmas.
What does an increase in overall spam volume mean for us?
- We need extra resources to process the mail as it arrives. Without enough processing power mails begin to back up in the queues and are delayed.
- The mail has to go somewhere of course, so we have to have an appropriate amount of mail storage available.
- All those GB of mail have to get to us somehow so of course we have to scale the transit connections to the rest of the Internet to cope with the additional traffic.
What are we doing about this? Well in the medium to long term we are committing some of our development resource to changing the way our mail offerings work. What follows is a summary of the work we will be carrying out over the coming months.
Although we have had spam Detection as an option that can be enabled from the Member Centre of our Website for quite some time, we have recently revamped the spam solution to use a combination of DSpam and Spamassasin for added improved detection rates. The new system allows customers to train the spam detection system themselves so that it will continually improve itself. Detection is only the beginning of the battle however. Once customers trust that the detection system is accurate and never tags a mail as spam when it is not, we need to provide them with the ability to choose for us to drop those mails before they even enter the customer’s inbox.
We recognise that if customers are to buy into this idea, we need to provide additional functionality. Customers already have the ability to request that we black hole specific addresses, however we aim to improve the mail control panel options within the Member Centre of our website to allow customers to configure their own black hole lists, as well as their own white lists.
One of the features of our mail offering has been that customers have access to what is called a catch-all address. This means that mail sent to any address at a customer’s domain would be delivered to the default mailbox regardless of if it is an address that the customer has configured. While this used to be a sustainable way of providing an email service, in light of the rise in spam, this can no longer be said to be the case. With that being the case we are working on changing the mail offerings so that this catch-all functionality will no longer be the default behaviour.
We are also investigating the option of implementing what is known as Grey-Listing on the mail platform. This is a method of ensuring that only valid, “well behaved” email servers are allowed to send email into a mail platform, while “badly behaved” mail servers, like the ones used by spammers and compromised computers are rejected. For a full explanation of grey listing, please see the following link:
http://en.wikipedia.org/wiki/GreylistingWhile Grey Listing is probably a sensible step in the right direction, it will not really help us against spammers who use legitimate email servers. For more information on how this is possible, please see the following link:
http://en.wikipedia.org/wiki/BotnetAdded to all of this, is the recent delivery of a spam detection appliance from a company called Critical Path. Many other service providers use this solution in their mail platforms, so we want to evaluate its performance. First off, it will be positioned in front of our internal mail platform by way of testing, and if successful, we will consider placing it in front of the customer mail platform.
While all these developments will help us in the future, in the shorter term we have to manage our mail platform to ensure a great service for our customers. As has been commented on before, we have implemented a new platform for mail delivery. The new mail delivery platform, also known as the MXCores, utilises more powerful servers and improved configuration of the mail delivery software to increase throughput of mail, as well as provide site resiliency. New hardware only goes so far however, so we decided to invest some time in a full review of how the mail platform is used so that we could better manage it.
We offer customers unlimited mailboxes, something that I know many customers think are a great option, and are used by many of them. We have been analysing the usage of all the mailboxes on the platform and have found a proportion of mailboxes that have not been accessed for some considerable time. These mailboxes are still receiving mail; mail that has to be processed by the servers responsible for delivering mail from external sources to an individual customer’s mailbox; mail that takes up space on the storage; mail that is not being read. We have chosen to set a minimum number of days up to which we think it is acceptable for a mailbox not to have been accessed. This level is 90 days.
What are we going to do with these mailboxes? We are going to remove the mail in those mailboxes from the platform and apply a black-hole to the address for that mailbox. This not only frees up space on the storage for customers who are actually using the platform, but also prevents mail destined for unused addresses from taking up resource on the mail delivery servers.
I am aware that some customers will be concerned that as with all human endeavours, mistakes are possible. It is for this reason that any mail that is selected for removal from the storage will be archived for 30 days, rather than deleted. This will allow us to return mail to a customer if a mistake should be made. Once the archiving of mail has occurred, and the black-holing has been applied, a single mail will be placed into the inbox of the affected mailbox, explaining what has happened, why it has happened, and how the customer can have the black-hole removed if they wish to start using the mailbox again.
While we have set the minimum level at 90 days, it is important to note that there is no immediate requirement for us to use this minimum level. We currently implement this on mailboxes that have not been accessed for 180 days. That is 6 months without once logging in to the mailbox.
I hope this information has been useful to you, gives you some insight into the plans we have and the actions we are currently taking to ensure that we offer you the best value mail service possible.
Simon Day
Network Improvement Consultant
PlusNet
pp.
