Home   Help Search Groups Login Register  
You are not logged in. To get the full experience of these forums, we recommend you log in or register
Plusnet Usergroup » All Users - The Open Forum » Plusnet Network and Technical Issues » Network Spending and Development Plans
Pages: [1] 2 3
  Print  
Author Topic: Network Spending and Development Plans  (Read 33559 times)
Simon Day

Posts: 263


« on: August 15, 2006, 05:09:48 pm »

Afternoon everyone.

One of the trends that we have noticed in some threads on the forums and other communication methods is that some customers believe we are not investing enough in the platforms. We believe that that this does not reflect reality, but the fact that people think this to be true is our own fault for not keeping you all informed of our plans. In this posting I am hoping to give you an insight into our spending plans, what we want to spend money on, and why. It is important to note that as you will all be aware, we work in a very fast paced industry, time behaves differently in the Broadband industry, as our Chief Operating Officer often reminds us, we operate in mouse years! This means that these plans are of course subject to change, but I hope they give you the clear view that we do plan for the future and are constantly trying to provide the customers with the best platform that we can. Also, some of these plans are more fleshed out than others, the further they are away in time, then the less detail we have. So, without further ado...

Capital Expenditure plans for 2006 / 2007

Over this year we have planned for a total CAPEX spend of over 3 million, of which these are the highlights of items yet to be implemented: 

MXCores - 120,000.

The MXCores are responsible for the delivery of mail to customer's inboxes.

The current MXCore platform consists of 24 1U Intel based servers, and was designed to scale sideways, i.e we could just throw more kit at the platform as the traffic increased. This design has proved very effective, and was perfect in its day, however 24 servers already take a lot of environmental resource, as well as a lot of management overhead. To scale the platform sideways further is only going to add to this burden. So, to reduce the resource required to run the platform, we are planning to significantly change the architecture. 

The plan is to replace the 24 Intel boxes with 4 clusters of 4 x Sun T1000 servers and 1 Sun 3510 Fibre Channel Disk Array. There will be 2 clusters per site providing true site resilience and the option to scale sideways again.

Core Database - 100,000.

The Core Database contains all the information related to the customer base and is the heart of the business in many ways.

The technology used in the Core database platform is due to be refreshed to reflect the increased performance and storage requirements we have with an increased user base. As with the MXCore platform, we also want to take advantage of the multi-core technology that powers modern servers, which reduces our rack, power and cooling requirements. In light of this, the plan is to replace the six Sun V440s and associated various storage back ends we currently use, with 5 x Sun T2000 servers and 5 fully populated Sun 3220s for storage.

Core Network - 262,000

We define the Core Network as the equipment in London that takes traffic from the BT/LLU networks, and passes it on to PlusNet's own services, or the Internet, either over Peering or one of our Transit providers. We currently have a switched 10 Gigabit Ethernet layer 2 core in London consisting of Foundry Big Iron Switches and a routed Gigabit Ethernet Layer 3 core consisting of Juniper M10i's. These devices are reaching the edge of their performance and scalability limits so need to be replaced with more powerful equipment. Again we are taking the opportunity provided by needing to scale the platform to change the architecture. We are looking to both simplify and reduce the environmental footprint requirements of the Core Network. 

This project's aim then is to collapse these two types of device into a single device and to remove the layer 2 switched core architecture. To do this we will be implementing some new Cisco 7609's as a replacement for both the Juniper routers and the Foundry switches. As with most of our projects, we prefer the phased approach. So this work is being spread over several months, with the first phase already completed.

Customer Termination & Traffic Management - 135,000


Customer Termination equipment is defined as the equipment that customer's DSL connections are terminated on, and of course, as everyone knows we use Ellacoyas for our traffic management.

To scale the Broadband platform we have already placed the Purchase Order with both Juniper and Ellacoya for some new kit. However we have not purchased a new ERX1440, but instead have made the move to the new E320 which is Juniper's next generation of Broadband Termination device. This improves our environmental requirements and gives greater performance for the money.

Portal Servers - 12,000

As part of the long term plan for the Portal platform, we have initially scoped two lots of budget for the portals. This first allocation of cash is really just to sideways scale the platform to stay in line with the increasing number of customers using the portals. So all we are doing here is buying some additional 1U servers with 4Gig of Memory, 2 x 3Ghz processors, hardware raid and 2 x 73GB  SCSI disks. I will talk about the next phase of this work later.

Parbin Refresh - 22,000

Most of the Parbin brands (Port 995, Just The Name etc) that we purchased along with Metronet still reside in the original Redbus Co-Lo facility in London and although we have made some additions to the platforms running the services, i.e. additional memory, we need to completely replace the hardware to bring them in line with our current hardware standards and move the equipment into our main co-lo facilities in London and Sheffield as appropriate.

Out of Band Management Network - 50,000

Out-of-band management refers to system console access provided, even in the event of primary network subsystem (hard and/or software) failure.

We currently do not have a consistent separate network over which to control and manage our servers and network devices. This project is to build an Out Of Band Management Network at every Point of Presence, be it in Sheffield or London. This will entail implementing a pair of Cisco 3825 Integrated Service Routers into each PoP to provide security and serial access to devices and ISDN routers to which we will attach ISDN lines for access from remote locations as well as all the internal LAN infrastructure. 

Improved Load Balancing - 20,000

We are planning on implementing Global Server Load Balancing across our Data Centers so that if we have a platform failure in the Primary site, the traffic will automatically fail over to the stand-by. 

Data Centre Resilience Improvements - 45,000

The aim here is to ensure that no back-end system is a single point of failure and provide a truly resilient platform.

Component Add Refresh - 6,000


This is a piece of development work to improve the functionality of our Component Add system and improve the servers it runs on. This system is used every time you add a component to your account and has become slower to run as we've added additional functionality. This work will speed up its operation and allow us to remove any legacy code.

CBC/VMBU

CBC stands for Capacity Based Charging, and is the current system we use to bill our Pay As You Go customers.

This is a piece of development work to allow us to decommission the CBC servers and move the functionality to VMBU. Currently PAYG customers are billed based on RADIUS data, which is processed by the CBC servers. Once this work has been completed all customers will be billed/managed by the VMBU system. There will be some prerequisites fixes to VMBU to ensure the validity of the data.

Authoritative DNS

This is a piece of development work to ensure that our current Authoritative DNS servers that contain all of our customer domains are able to scale and handle the increasing number of domains that we host. This includes work to make sure that the servers are able to restart in a timely manner whilst loading up the domain list.

Monitoring - 5k

This work is to provide a dashboard view internally and also provide resilience of this view. Currently we only have the very detailed technical view of what's going on with the Infrastructure that can be difficult to read and see quickly what the health of the network is.

New Building

We are moving from our current two buildings into a brand new office in the Centre of Sheffield, this is the link to the new developers website http://velocity-space.com. As part of this we are fully reviewing and replacing our internal infrastructure to give the best possible secure access to everyone in the office.

Mail Storage - 170k

This is the new Sun NAS system that has been purchased to provide additional capacity for customer e-mail storage to cope with the growth in traffic.

Scale of the platform in line with increased customer numbers

There is an ongoing task to constantly upgrade or improve the current platform to cope with the increases in customer numbers and volumes of data/traffic being passed or processed.


CGI Back-Ups

There is a plan being formulated to improve the current back-up infrastructure for the customer CGI platform. This is to ensure that the infrastructure can handle the increasing customer traffic on the platform.


Capital Expenditure plans for 2007

The total expenditure amounts to more than 3 million again, of which the highlights are as follows:

Customer Termination and Traffic Management - 520,000

Core Network - 151,000

Data Centre Improvements - 21,000

New Portal Server Hardware - 50,000

This is what I mentioned earlier, and follows the same idea as the MXCore changes we are making in 2006. Basically the idea is to reduce footprint, while improving performance.

The attached GANT Chart shows the current view of the order of work.


I hope you have found this information interesting and goes someway to showing that we do have plans for investing in the platforms. As we get close to the date of implementation, I will post individual threads so that everyone knows what our plans are and has a chance to comment and question.

Best regards


Simon


** Modded by IW to add some bold tags.
« Last Edit: August 15, 2006, 05:41:09 pm by ianwild »

Simon Day
Network Improvement Consultant
PlusNet Plc
Oldjim

Posts: 1014

« Reply #1 on: August 15, 2006, 05:30:20 pm »

Simon,
That is very detailed and totally over my head but can I ask if the portal improvements will permanently speed up the portal access as it is still the slowest website I visit by a long way.
ianwild

Posts: 3979


Not to be confused with Mike, Wildmind.

WWW
« Reply #2 on: August 15, 2006, 05:42:31 pm »

Hi Jim,

They will make a big difference, as will the database improvements.

There is another aspect of this in the development (ie not networks) workstack, which is about optimizing the queries and code used on the portal.

Combined, those three pieces of work will speed up the portal infinately.

Ian

Regards,

Ian Wild
PlusNet Support
noelshaw

Posts: 48

« Reply #3 on: August 15, 2006, 07:43:17 pm »

A most interesting, relevant and well presented report.

The plans are most impressive and the new building superb.  You all deserve it!

Regards.......Noel
Arthur

Posts: 39

WWW
« Reply #4 on: August 15, 2006, 07:51:31 pm »

As someone pointed out on the F9 forum they have made no mention of staffing levels or plans to increase them.

ianwild

Posts: 3979


Not to be confused with Mike, Wildmind.

WWW
« Reply #5 on: August 15, 2006, 08:56:32 pm »

This was an update from the networks team - They don't deal with staffing in the CSC (or anywhere else)!

Over the coming weeks we have a lot of stuff planned in terms of presenting details about other areas of the business. At the top of that list is a CSC Update, (which I know is overdue, and I should have posted by now in all fairness).

There is certainly no shortage of resources being put into this area either either though. That said, the proof is in the pudding on that one and it is clear that currently we aren't where we want to be. Although I don't think staffing levels are at the root of the issues within the customer support area, there are certainly plans in place to recruit as needed across the board.

Ian

Regards,

Ian Wild
PlusNet Support
fenlandbroadband

Posts: 176

« Reply #6 on: August 15, 2006, 11:33:05 pm »

I really want to go in to one of my typical long winded posts where I just blurt out all of my thoughts on this and make a few points/raise a few questions, but I can't think of how to word what I want to say (or even what I want to say)  sad

Must have spent at least half an hour trying to think of what to type (I've probably been timed out from inactivity by the time I hit post...) because I don't want it to come across as negative or as if I'm ungreatful for this info, which I'm certainly not.


That said, the proof is in the pudding on that one

It's been said many times before that we should judge PlusNet on their actions and not their words, and whilst many of us do constantly judge PN and pull you all up on various bits and pieces, and individuals have judged PN and stayed/left, we've never really got to the stage in recent months where PlusNet have actually set down some real accountable targets and figures, and then said if they've been achieved them or not. When is the pudding going to be served up so as we can see if it's actually ready? Like I've mentioned to you before Ian, it's a bit like when the announcement went out the other month that the CSC call waiting times were to be cut as part of the improvement plans, but no details of what the call wait was hopefully going to be cut by/to - it could have been cut by 30 seconds and PlusNet could have claimed they'd managed to achieve their aim, pat themselves on the back, and throw a big (well deserved) party for the staff - trouble is, it'd still be a ridiculous and unacceptable call waiting time which makes it absoloutely no better in comparative terms for customers than it was before. I really do feel PlusNet needs to give dates for when things will be ready (or the "judgement day"?) and real targets/aims/figures for what is trying to be achieved. It's all very well saying we're going to do x y and z, but we never really seem to get to the point where we see it pulled off and where we (and no doubt you guys too) can sit back and think "yeh it's actually there, it's working, job done".

I know you can't see in to the future (despite the "PlusNet vision" statement  tongue) so saying that you'll achieve something by a certain date isn't going to be easy, but I think it's "too easy" to simply claim something will be done without saying really what the something is and how we (and PN?) can judge if it's actually been achieved. And whilst that is the case, it's words and not actions that we're getting.
Simon Day

Posts: 263


« Reply #7 on: August 15, 2006, 11:58:33 pm »



I know you can't see in to the future (despite the "PlusNet vision" statement  tongue) so saying that you'll achieve something by a certain date isn't going to be easy, but I think it's "too easy" to simply claim something will be done without saying really what the something is and how we (and PN?) can judge if it's actually been achieved. And whilst that is the case, it's words and not actions that we're getting.

Hi Fenland,

I completely understand the importance of being specific about what you are delivering and when. With that in mind, I will have the first of the more specific postings I mentioned, out by Tuesday next week (22nd). It will contain more info about the Mail storage platform project we have on going.

Simon

Simon Day
Network Improvement Consultant
PlusNet Plc
portmoak

Posts: 214


WWW
« Reply #8 on: August 16, 2006, 09:28:01 am »

Although I don't think staffing levels are at the root of the issues within the customer support area, there are certainly plans in place to recruit as needed across the board.

Ian

With the very greatest of respect Ian, you are wrong. There are clearly too few staff for the job. You could argue that if there were less problems you wouldn't need so many staff but that's another issue entirely seeing as the number of problems seems to be increasing anyway. More better-trained staff are what's wanted. If PlusNet aren't vigorously addressing this then the outlook is bleak.

Accounts theadamsons and portmoak
F9 customer since 1998.
wildmind
Guest
« Reply #9 on: August 16, 2006, 09:44:14 am »

Guys,

Cna we keep staffing discussions to a different thread please - this one is about investment into the network infrastructure as per Simons announcement.

Cheers laugh
jbpnug

Posts: 4

« Reply #10 on: August 16, 2006, 10:21:08 am »

Quote from: simonday
CGI Back-Ups
There is a plan being formulated to improve the current back-up infrastructure for the customer CGI platform.
huh Too little too late! I've had all my CGI space wiped out - a month after PN destroyed all the emails which had accumulated while I was on holiday - and I'm still waiting to hear what's being done about restoring my cgi space (ticket I.D. 20045007)

Quote from: fenlandbroadband
It's been said many times before that we should judge PlusNet on their actions and not their words, and ... we've never really got to the stage in recent months where PlusNet have actually set down some real accountable targets and figures
Too true - and PN's actions of late have been very much lacking in competence and customer-friendliness. Capital spending plans are all very well but the crisis is now and it also needs staffing changes from the top down.  angry
« Last Edit: August 16, 2006, 10:23:44 am by jbpnug »

Regards

John B
Colin
Usergroup Member

Posts: 6339


WWW
« Reply #11 on: August 16, 2006, 10:47:19 am »

I'm guessing if your CGI space hasn't been restored by now, you're one of the 10 or so customers affected who couldn't have their site restored.

Colin Ogilvie
Plusnet Usergroup Member
Using: Plusnet Extra
JBailey

Posts: 2141


« Reply #12 on: August 16, 2006, 11:31:52 am »

As someone pointed out on the F9 forum they have made no mention of staffing levels or plans to increase them.

Hi,

I just posted this in the PlusNet Portal forum:

https://portal.plus.net/central/forums/viewtopic.php?p=346567#346567

Quote
Not sure how facile replies like that improve the relationship with the customer, James.

Of course customer service isn't part of network improvements.

But you would have thought that given the obvious problems with customer service, communicating how it would be improved would be no 1. priority, not detailing the make of your new routers.


Hi there,

Appreciate your comments. My post was initially out of irritation. Predominantly this was because Simon had gone to a lot of trouble to let everyone know that we are spending money on our infrastructure, because this has been brought up a number of times.

This isn't saying that network infrastructure is our number 1 priority. The CSC Redevelopment is a massive priority at the moment. Are we sorry that our friends in the CSC are taking a lot of flak currently? Of course we are. They're not just doing their work without rewards though. All of them have received payrises to 18k pa now, some of them were earning as little as 14k beforehand, and some other members of staff have received further payrises in recognition of their good work since.

What a number of you will not have seen is that we are continuing our recruitment plan, and we've had a few more new starters, and I understand that there's another 8 that have started this week and are currently going through an intense training plan currently. We've also got a massive amount of development focus on the CSC currently, fixing things that irritate you, and we're constantly working on improving the Help Assistant in response to feedback that we have been getting from you guys and PUG to make things easier for everyone.

Also what you won't be seeing is that practically everyone in the busniess (even Danny O'Sullivan, our COO) has been working on tickets. Everyone from Marketing to WPMT, to Product and Development has been getting training in annoying things like house moves and provisioning, and a number of people in this building have been staying on as late as 9pm every day, after starting at 8am, to help out with tickets, and we're confident that we will be able to get the backlog of tickets down to an acceptable amount shortly.

I hope this goes some way to providing some insight to the fact that we do actually care about our customers, and we don't just say things to annoy you all, and we are focused in progressing forwards and want to get back to being able to provide a high level of quality support so that our customers will once again feel proud to refer our services to their friends.
_________________
Kind Regards,
James Bailey / jbailey@plus.net
PlusNet Customer Comms Co-Ordinator

Kind Regards,

James Bailey
Complaints Manager
Plusnet
simonflood

Posts: 88

« Reply #13 on: August 16, 2006, 04:11:07 pm »

MXCores - 120,000.

The MXCores are responsible for the delivery of mail to customer's inboxes.

The current MXCore platform consists of 24 1U Intel based servers, and was designed to scale sideways, i.e we could just throw more kit at the platform as the traffic increased. This design has proved very effective, and was perfect in its day, however 24 servers already take a lot of environmental resource, as well as a lot of management overhead. To scale the platform sideways further is only going to add to this burden. So, to reduce the resource required to run the platform, we are planning to significantly change the architecture. 

The plan is to replace the 24 Intel boxes with 4 clusters of 4 x Sun T1000 servers and 1 Sun 3510 Fibre Channel Disk Array. There will be 2 clusters per site providing true site resilience and the option to scale sideways again.

Mail Storage - 170k

This is the new Sun NAS system that has been purchased to provide additional capacity for customer e-mail storage to cope with the growth in traffic.


CGI Back-Ups

There is a plan being formulated to improve the current back-up infrastructure for the customer CGI platform. This is to ensure that the infrastructure can handle the increasing customer traffic on the platform.

I hope you have found this information interesting and goes someway to showing that we do have plans for investing in the platforms. As we get close to the date of implementation, I will post individual threads so that everyone knows what our plans are and has a chance to comment and question.

The whole post was very interesting, as much for things you didn't mention as for things you did.

As you'll probably have already noticed I've picked out the items to do with e-mail and CGI.  Whilst you specifically mention backups in association with CGI I don't see anything relating to backups and e-mail.  Do PlusNet have any plans to do a proper backup of the e-mail storage system (rather than relying on mirrors/snapshots) to cover future systems failure (not to restore individual e-mails for individual customers)?

Many thanks,

Simon
Simon Day

Posts: 263


« Reply #14 on: August 16, 2006, 05:07:08 pm »

As you'll probably have already noticed I've picked out the items to do with e-mail and CGI.  Whilst you specifically mention backups in association with CGI I don't see anything relating to backups and e-mail.  Do PlusNet have any plans to do a proper backup of the e-mail storage system (rather than relying on mirrors/snapshots) to cover future systems failure (not to restore individual e-mails for individual customers)?

Backing up a mail platform as busy as ours is never going to be an easy task, with the amount of transient data and the sheer volume of data requiring copying to another location. However there is a debate going in internally about back-ups in general, what we should back-up, how often and to what media and as that debate progresses I will share with you the outcomes.

Simon Day
Network Improvement Consultant
PlusNet Plc
Pages: [1] 2 3
  Print  
 
Jump to: