next »

[dhookham]

Thing is, like OE, you don't *have* to allow 60k mails to pour in. Start it pulling in headers, then once it's got a sensible amount in (start with around 1k, but I've had it happily handling 3-4k mails), stop it. It will then have a stab at tagging the mails as good or bad - see how well or badly it's done. Make sure to make use of marking domains as spammy if you spot any obvious patterns in the list of blocked addresses that you start to build up eg sd232asdfew@spammydomain.com  2442ds42@spammydomain.com  etc.

Also might be worth added a few good spam blacklist servers to it so help the identification.

Then process what you've got. Ensure the rest is marked as legitimate and that legitimate male gets hidden. Then download some more headers and repeat the process.

[northbritish]

But don't forget to add filters to keep CC LegitAddresses@yourdomain.com and BCC as well.  I get copied in on quite a few e-mails that don't get sent directly to me.

[mikeb]

Yeah, good point, if there are likely to be valid CC or BCC as well at TO headers then include these in your "legitimate mail" filters.  You can easily test the CC field but not BCC (or envelope) directly in Mailwasher filters.  Probably the easiest thing to try first is to check the ENTIRE HEADER instead of TO FIELD for all valid addresses so filter1, for instance, becomes:

Filter1: If the ENTIRE HEADER contains "valid_name1@your_domain.co.uk" then mail is legitimate.

You definitely can stop Mailwasher at any point but I'm not sure just how many messages it will handle sensibly before falling over.  You should certainly be able to leave it running for some reasonable time retrieving and displaying messages to get a good idea of what's going on. But the problem as I see it is that with 60k+ messages, it's pretty painful to do the whole thing manually and difficult to make a definitive judgement on whether the filters are totally OK or not.  Spotting the occasional 'good' email in the average bucket of spam is often not that easy but with 60k messages it's going to be virtually impossible. Without wasting an age trying, a 'good guess' at likely legitimate mail and just dumping the rest is probably the best that can reasonably be achieved.  The only real problem will be if there is lots of spam/bounces sent to 'valid' addresses.

Similarly, I'm not sure that consulting spam databases and so on is going to be a particularly sensible thing either.  I find it takes bl**dy ages in most cases even for only a small amount of spam.  I'm not certain but I think it also means that you can't delete immediately and silently so again, that either means loads of manual intervention or hoping Mailwasher can handle the whole 60k in one hit without either it or Windoze falling over.

I would doubt Mailwasher/Windoze will handle 60k headers without complaining lots at best.  In fact, I doubt the PN mailserver will allow a connection to be open for the length of time necessary to get 60k headers in any case !  I'll refrain from suggesting that the PN mailserver might not even run for such a period of time without falling over in some way or other even on a good day

Obviously, Penny will have to decide what is a reasonable amount of time to spend on the task .vs. the benefit in doing so.   However, if legitimate mail is only likely to be being sent to a few known addresses (with minimal spam to those addresses) or from certain people or on certain subjects etc. then the 'simple' approach of keeping those messages and just deleting the rest unchecked is likely to be the sensible and certainly the fastest option I would suggest ... well assuming that you consider around 16 hours to do it fast anyway !

[Penny]

Thanks for all the extra info, guys   Some of it is maybe a bit beyond me, but I can see the logic of most of the stuff re filters etc.

However I'm not entirely convinced that the F9 end of things is working properly.  Just tried logging on to the default account in OE (which has maximum server time out of 5 minutes but usually allows extensions) and within a minute or two, got:
"Your server has unexpectedly terminated the connection. Possible causes for this include server problems, network problems, or a long period of inactivity. Account: 'Project HappyChild', Server: 'mail.force9.net', Protocol: POP3, Port: 110, Secure(SSL): No, Error Number: 0x800CCC0F"

Webmail refuses to let me in "Username/Password match not found, please try again", Mailwasher times out after 60 seconds because the server breaks the connection, and Outlook (using IMAP) with a max server time of 10 minutes also times out because the server breaks the connection, iirc.

I'm not getting any header downloads at all in Mailwasher, OE or Outlook because the connection seems to be severed before that point is reached, so at this stage it seems pretty academic whether I have filters or not because nothing ever reaches the point of anything being downloaded.

Can however confirm that the two new mailboxes are working okay and a very small trickle of mail is coming in by that route.

Realistically don't want to lose the essential stuff already in the default mailbox (which will be racking up a further 5,000 e-mails a day the longer this is left) but I can't get at any of it.  More than happy to try out the filters etc. that have been explained here but I can't do any of it if the connection keeps getting broken.  Main access is 4Mb-ish cable by the way (free!) so reasonably fast, and it isn't viable to attempt downloading 60,000 e-mails via a PN connect lite account (dial-up type) or Netstart weekend 0800 access (also dial-up type speed).

Would be helpful if F9 could find a way to export the existing mail so it can be accessed somehow via methods people have suggested here, because I don't seem to have any routes to download it by conventional means.  At worst, as John (jelv) has said, I can probably just get by, by means of setting up mailboxes for all the main addresses, but realistically that's going to (a) leave the current 60,000+ mails in the default mailbox, and (b) over a period of say 6/7 months, stack up a million e-mails into that box.  That can't be a positive long-term progression for the mail server

Still stuck in surreal mode here and little spare time anywhere.  If F9 can either (a) fix my mailserver access continuity, or (b) export the current default mailbox content in a file, or (c) fix Webmail access, or (d) re-queue the existing messages (assuming I or they can set up default blackholing before that point) then at least the current situation will be fixable.  Otherwise I don't see there are any alternatives to letting the e-mails build up on the mail server till it reaches a point where it will presumably crash

Really don't see there is any more I can do till Support make some comment here   Like I said it's all a bit surreal but then life generally has been a bit like that, lately.

Regards,

Penny.

_{[edited slightly for clarity]}

[wildmind]

Presumably you have raised a ticket on the matter? Due to the Christmas hols it would be best to ensure that a ticket has been raised.

[mikeb]

There is most certainly some problem with the PN mailservers and with your A/C in particular   I have had no problems with any of mine recently either on PN or F9. Until that is sorted and you can get access to your mailbox normally there isn't much you can do with Mailwasher apart from setting up as many "legitimate mail" filters as you can think of.  Have a good scan through your previous messages and come up with as many sensible looking filters as possible.

Regarding time, connection type or speed is just about irrelevant when determining how long processing 60k messages will take.  It's the protocol and PN mailserver response time etc. that will largely determine this and there really isn't anything much you can do about it.  The very best I see is probably around 3 messages per second and the worst generally being around 3 seconds per message on a 2Mb/s connection. You can do the maths but either way it's a long time !  Whether you have a 'good' dial-up or 'bad' broadband connection wouldn't seem to matter.

You really need to be able to take a look through a reasonable number of the messages first and test out the filters on a sample of them to determine what are 'good' and simple/quick filters.  Whilst Mailwasher can be setup to do a near 100% correct job this does usually take some considerable time not to mention maybe some clever filters and a lot of messing around etc.  What I'm suggesting is trying get rid of >90% of the [Censored] with minimal risk of deleting any significant amount of 'good' stuff and without too much messing around. If you can put a good set of simple filters in place fairly easily, you should then be able to leave Mailwasher to get on with it and only have to restart it when something falls over or the server connection gets dropped etc. If you find that the PN mailserver regularly boots you off after, say, an hour then try setting Mailwasher to automatically check for messages every 90 mins or something to see if it will then restart automatically.  Each time you restart you should at least be seeing far less messages to DL, well, assuming that you can delete them faster than they are arriving in any case !

Once you've got rid of >90% of the most likely [Censored], then is the time to think about being more clever with the filtering or checking against blacklists etc. to further remove rubbish from the relatively small number of possibly legitimate messages that remain.  When you're happy that all the messages listed in Mailwasher are mostly legitimate messages or those that aren't can easily be deleted manually then DL these into your mail program and consign the default mailbox to that great blackhole in the ether to just dump all future messages ... providing that all your individual mailboxes for legitimate future messages are set up and working correctly of course

Here's another maybe useful filter to add.  If there are a significant number of bounce messages to one or more of your valid addresses then add a filter right at the top of the list to delete any most likely bounce messages before checking for legitimate messages:

If the TO field contains "mailer-daemon" OR the SUBJECT contains "failure notice" OR the SUBJECT contains "returned mail" OR the SUBJECT contains "unsolicited bulk mail" OR the SUBJECT contains "undelivered mail" OR the SUBJECT contains "delivery status notification" OR the SUBJECT contains "SPAM" then delete the message.

One final thing, I really don't think that PN bundling all the messages up into a single mbox file or similar for you to import is particularly sensible or likely to happen either for that matter.  Re-queuing would certainly be beneficial but again, it's probably not as easy for PN to do as it sounds so I wouldn't expect it to happen either.  In any case, providing you can get the access problem sorted and Mailwasher running, simply tagging the legitimate mail will effectively achieve the same thing as re-queue would.

[Penny]

There is most certainly some problem with the PN mailservers and with your A/C in particular   I have had no problems with any of mine recently either on PN or F9. Until that is sorted and you can get access to your mailbox normally there isn't much you can do with Mailwasher apart from setting up as many "legitimate mail" filters as you can think of.

Thanks for all the extra info, Mike.  Checked on tickets but nothing had been logged (I'd phoned twice but really only to ascertain volumes so there wouldn't have been much info for Support to open a ticket with) and obviously at that stage I hadn't recognised that I hadn't actually got access to the mailbox.  Duly requested now

The only two critical addresses are those quoted already (the rest I can live without for the time being) and new mail is getting through, so once the mail server will give me access I'll have a go at setting up Mailwasher along the lines you've suggested.

Appreciate the very detailed instructions   No guarantee I won't stuff it up but I'll do my best if there's time anywhere over the next few days.  Happy Christmas all

Penny.

[Colin]

Id suspect the problem is with the volume of mail -- 60000 is a lot of mail to index on loging in to the mail server.

[mikeb]

Hmmm, that's what I was thinking ... along with trying to blackhole the default mbox maybe causing problems.  I have a real bad feeling here that getting to these 60k+ messages may well turn out to be a tad more difficult than it ought to be and perhaps they're not even there any more    Will just have to wait for CS or whoever to lose the hangover and update/action the ticket I suppose.

Anyway, here's hoping PN haven't had a tidy up and one more thing regarding Mailwasher setup:

In TOOLS/OPTIONS in might be a good idea to reduce the number of lines DL'd from each message.  The default is 200 lines but if you are not trying to filter on the message body then reducing this to, say, 50 lines (or even the minimum 20) should ensure that all headers are obtained and checked but not very much of the body.  The possible benefit is in speeding the process up and reducing processing and storage requirements.  Once the number of messages is down to reasonable levels then the default setting can be restored so you can have a quick look at a significant chunk of the message contents if you want to before deciding if it's useful stuff or junk.

The only two critical addresses are those quoted already (the rest I can live without for the time being) and new mail is getting through, so once the mail server will give me access I'll have a go at setting up Mailwasher along the lines you've suggested.

Don't forget that if you can actually get Mailwasher trying to sort this nightmare out, you will only get one chance to retain the possibly legitimate stuff.  Anything that doesn't match one of the 'legitimate mail' filters will be deleted on the first pass and will not be recoverable.

[Ultra]

Still times out    presumably because it isn't designed to download 60,000 messages at one go.  Realistically I feel I've done everything that is reasonably possible from this end; 

If it helps, I have a runbox.com account (which can import mail from another a/c, and has a 10 GB capacity, and 2.5 GB limit on traffic each week...  so it could probably import all the mail, and hold it in a folder or folders for you, or forward to one or more Google Mail accounts...  I've sent you a PM with some other comments).

The only other possible solution (which James suggested) would be if F9 can export the entire contents of the rollo... account into some type of file which could be downloaded (FTP?) in some sort of format which could be imported into OE or Outlook, so I can extract anything relevant before deleting the rest.  Either way, sorting this out has realistically to be done at the F9 end as I can't do anything further from here :/

I know MS OE gets slower and slower as the message count goes up...  I'd guess that trying to identify and zap unwanted messages prior to downloading / importing would be better, though must admit today was the first time I'd seen this thread, so am not up to speed on full set of details - just reading through...

[MauriceB]

Hi Penny,

following this thread and your problems / solutions.  You may want to try Mailmaint a program I suggested back early in the thread?  It was written to help in various situations that cause problems in POP3 mailboxes.

In particular in your case you can set up a preference to 'Ignore headers older than...' and also to only download the headers without any message content.  Setting 'some' or 'all' debug options (also in preferences) will allow us to see what is happening between your pc and the mailbox.

So you could try setting it to 1 day's worth of headers and see what you get?  It should give you the latest mail first? If this works, then it is possible to set up filters and automatic deletion later.  It will still be a pain with 60,000 messages, but it might just work?

NB  Although I've used MailMaint for a number of other problems, this is a new and untried area.  (end of health warning )

If I can be of any help and you want to PM - please feel free.

Maurice

[Penny]

Much appreciate the offers of help and PMs.  Have to admit though that I'm struggling with flu here and it seems to be getting worse (started about four days back) to the point where I'm just really not able to deal with anything that's not ultra-simple.

My ticket's been replied to, asking me to use delmail, but as Phil Richardson said further up this thread that I can't, and I still don't have any usable access to Webmail or the mailservers, I really don't feel there's much of a constructive nature I can do right now.

I'll be in touch with those who have kindly offered phone and PM help, when this bug's passed off a bit.  Thanks guys.

Penny. 

[Penny]

I'll be in touch with those who have kindly offered phone and PM help, when this bug's passed off a bit.

Feeling somewhat better now   but not been in touch with anyone yet, for which my apologies.

Just an update to let you guys know that Networks are going to try to sort out something next week - I can't use any of the proposed solutions right now as there must be somewhere between 100,000-150,000 mails in the default box (support tried to access it tonight but no news so far of a figure so it's probably at the high end of that) and that's why I can't access any of the headers from this end via OE / Outlook / Mailwasher.

I'll post back when I know more.

Regards,

Penny.

[northbritish]

You know how the new employee always gets the rotten jobs to do.  Guess who's going to be working his way through 150k of spam tomorrow to find the few dozen legit ones. 

[chillypenguin]

Here's a tip sort the messages by subject line if you can.
With 150K of spam then a lot of  that will be the same spam, but from different senders.
So this should allow you to pull the spam out in small chunks rather than one at a time.

Chilly