VOICE Home Page: http://www.os2voice.org
[Newsletter Index]
[Previous Page] [Next Page]
[Features Index]

March 2000
editor@os2voice.org

Junk Spy version 1.0
Part 2

By: Wilson Rook wrook@walkabout.org


(Part two of the Review, part 1 was in last month's Newsletter - http://www.os2voice.org/VNL/past_issues/VNL0200H/vnewsf2.htm)

As a 4 year veteran hometown ISP, I have learned a lot about e-mail. Over the past few weeks, I have learned more about e-mail, e-mail terrorism, and e-mail counter terrorism.

E-mail, at its most basic form, is a collection of keystrokes that can be recreated on the screen of any connected computer in the world. E-mail is a modern day telegram. "Have received the funds, will ship today, expect delivery on Friday.", is a very quick status update for ebay traders. The electronic telegram contributed to the demise of the pony express with same day messages such as "Bank notes received, expect delivery to Arizona Territory by Fall". In some cases, e-mail is delivered on paper by a courier (or family member), however, most e-mail is typed and submitted by the author and received and decoded by the recipient. As an ISP, I am simply a lineman with no knowledge of what actually travels across this new strange copper and light that we call the Internet. E-mail is fast, cheap, simple, and ... did I say fast? ... yes it is very fast.

With all powerful technologies come diversified use. One use of the 22nd century telegram is to Send Phenomenal Amounts Of Mail (SPAM) or Unsolicited Bulk E-mail (UBEM) [this acronym does not lend itself to use in lyrics like the famed canned ham product does]. Spam is the result of simplicity and cost. The ability to put a message on anyone's screen must be a dream come true for advertisers. Whether you're in the comfort of your own home or at the office researching products, SPAM turns your Internet surfing into a cheap network commercial break on late night TV. This great task is accomplished at a very low cost to the sender.

The best counter attack to misused technology is better counter technology, organization, and cooperation. Junk Spy has values in all three of these areas. As discussed further below, Junk Spy uses computers to automate the distribution of database updates of junk e-mail characteristics. Junk Spy users can submit samples of junk mail to Sundial for evaluation and inclusion in database updates. Like computer virus and anti-virus programs, spam is a dynamic target. Any solution to prevent unsolicited e-mail will be circumvented tomorrow. Sundial's Junk/noJunk e-mail procedure promotes cooperation of many users to identify and submit spam to a central clearing house.

Let's Get Technical

Sundial's approach to junk mail is very comprehensive. In my prior article, I addressed the general problem of junk mail as well as the second generation problems that can be caused by the solution to the first problem, misidentification of good mail. I have been running Junk Spy for over a month. For two weeks, the program ran using all its default parameters. Junk Spy was used for the next two weeks with the first database update, customization and participation as a contributor. With these changes, I increased the effectiveness of my personal battle against junk mail.

In the first two weeks, Junk Spy identified 69 e-mails as junk mail. There were a few that should not have been marked as junk mail.

The marked mail included: Jade Search Smart, Rodopi Software, wrook@oceana.net, Brent Blare, Abel McClendon, qwmancq3f1nkrz5xza@m300.swipnet.se, and Real Player News.

Jade Search Smart - I'm on their list but I don't remember requesting to receive it. I don't read it even though some of the titles look interesting. I had previously decided not to try to stop this mail. If I ever have a slow day, I'll read some of these. This should not be marked as junk mail.

Rodopi Software - Oh no! This one cost me $150. Yes, really. It was my own fault. I knew I should be reviewing the junk pile every day or so. This should not be marked as Junk Mail.

wrook@oceana.net - Hey, this one is from me. The following header print data shows on line 3 that the mail was received from an IP address of 209.211... This is not me. I'm a 216.111... type of guy. The next few lines show the smtp server info. The from and reply to show my address. I do send my self e-mail at times. It is a way to get info into my e-mail. This mail is not from me and very well should have been listed as junk. The last 4 lines in the following header is the result of Junk Spy at work. With information, PM Mail is able to sort the identified junk mail to my junk folder. Good spy job.


Brent Blare - I don't know who this is. I know to just delete it. Now Junk Spy is doing it for me. Good spy job.

Abel McClendon - Oops, this is secretary@os2voice.org. We'll have to find the fix for this one. I know this problem has already been addressed. This should not be marked as junk mail.

qwmancq3f1nkrz5xza@m300.swipnet.se - This is a no brainer. Good spy job.

Real Player News - I know I did tell real networks that they could send me mail. I'm not upset that junk spy trashed this one. Good spy job.

I have only listed a few items above. The other 69 were all junk. I also had a number of e-mails that did not get listed as junk. At this point, Junk Spy was not working optimally. It required a more aggressive approach than a simple static default installation.

The next task was to update Junk Spy and tune it to my needs. My first concern was the e-mail from Rodopi Software. I opened the missorted e-mail in my junk mail folder. I selected to view the e-mail "Full Header". In PM Mail, I did this by clicking on the menu item "Window" and then "View Full Header". The reason for the filter was shown at the end of the header in the "X-" section. Two "X-Comment"s told why and where action was taken to mark this as junk. The first comment stated "X-Comment: Found MASS E-MAIL in entire message". On review of my e-mail, I found the paragraph:

"... Gee, if you have 10,000 customers and you send one mass e-mail to all of them and only two percent respond you will generate enough profit from domain registrations to buy your Rodopi."

While other information in the message was of value to me, this statement that promotes "mass e-mail" is very offending. I don't want to change my settings, but you may. The next line in the header stated "X-Comment: For more information see "Junk Mailers" detector." Now I know where to make the change. In the small Junk Spy window, Click on the Trash Can icon and then click on "Junk Spy", then on "Detectors...". The last of 4 items in the list was "Junk Mailers", which I highlighted and then clicked on the "Modify" button. Near the top of the list I found "MASS E-MAIL (All)", which I highlighted and clicked on the "Modify" button. The next page gave me two options to apply the search to the e-mail To, From, Subject, Body, Header fields or the entire message. An "Unless" option was also provided. To test this "Unless:" field, I typed "Long Live OS/2" in the field. I sent two e-mail messages to myself as shown below. Only one message had the second line in the body. The Long Live OS/2 message was placed in my inbox while the other message was marked as junk mail.


The Jade Search Smart, Brent Blare, and qwmancq3f1nkrz5xza@m300.swipnet.se had similar valid reasons to filter based on the "Junk Mailers" detector. The primary use of the "Junk Mailer" detector is to search for key phrases that are very often contained in general junk mail.

cableplans@flashmail.com had a "Buy My Product" detector alert. As the name indicates, this detector looks for key phrases that would be found in a hard sale "Buy My Product" detector.

The infamous Valentine's Day/Secret Admirer e-mail from last month's article contained phrases picked up by the "Adult Content" detector. The fourth standard detector searches on phrases found in "Get Rich" junk mail.

The detectors use a very simple approach of searching for phrases and keywords that are contained in junk mail. You can disable, delete, edit, or create new detectors. The detectors work very much the same way that the filters work in PM Mail. Junk Spy's advantage is that it provides a pre-compiled list and the test search engine to back it up. The Detection Update Center will also provide a simple and automatic way to update the list. PM Mail's filters compliment Junk Spy by adding the ability to perform complex filter evaluation of marked junk mail.

Here is another e-mail that I do want to receive. The Abel McClendon e-mail (secretary@os2voice.org) had MEMBERS@ in the header and was filtered by the "Junk Mailers" detector. This needs to be corrected. There are many ways to do this. The Members@ can be deleted from the "Junk Mailers" detector. Or, @os2voice.org can be added as an exception to "Unless:" field on the MEMBERS@ Clue Definition. A Global Exception for @os2voice.org can be added as a new Clue Definition. I would like to keep the protection of the Members@ clue, so I don't want to delete this item. Adding and exception in the "Unless:" field of the MEMBERS@ clue would work, however, the list is long and I did not want to spend the time finding it. A search option to find the clue would be a helpful future addition to Junk Spy.

I chose to add a Global Exception for any mail containing the term os2voice.org in any part of the e-mail. This will create a much broader exception for any e-mail containing os2voice.org. I could also have limited the exemption to the from field.

The wrook@oceana.net from me that is not from me, and the Real Player News had addresses listed in the "Relay Spam Stopper" detector.

With the above simple changes, I continued to use Junk Spy. I now have my normal inbox, with child inboxes for Inet.Mail, PMMail, and WarpCast mailing lists from my prior PM Mail configuration. I also have my new junk mail box. My junk mail box contains 46 new messages, all junk. I have received about as many messages in my normal inbox that should have been filtered as junk mail. Many of these were forwarded to Sundial for Evaluation. I look forward to the next update and improved detectors.

One of the benefits of the Internet is the ability to receive information from many sources in an inexpensive and very manageable manner. Junk Spy is designed to eliminate mail that uses the same procedure to distribute unwanted mail. Without recognition of the daunting task, it is very hard to appreciate Junk Spy. With proper use of Junk Spy, and Sundial's Junk/NoJunk review process, the detectors will continue to improve. Like the war on drugs, brute force only works so far. The real solution is to eliminate the desire or demand for the product. Comprehensive use of products like Junk Spy will eventually result in decreased effectiveness and desire of junk mailers.

Junk Spy uses public data bases such as the Real Time Black Hole and the Relay Spam Stopper lists. Junk Spy also uses its own database. All three of these databases are living, ever changing archives. To support Sundial's efforts, spam or junk mail that is missed by Junk Spy's detectors can be forwarded to Junk@sundialsystems.com. Before you send just any junk mail, could this be mail that you requested (or forgot to unchecked a box on a web site that said "Keep me informed of new products")? If the mail is from a legitimate mailer, your best bet is to unsubscribe. Some of you may consider the Voice Mail list to be Junk mail - it is not. It is simple to unsubscribe to most mailing lists by just sending a reply e-mail with unsubscribe in the subject and/or body. Most legitimate newsletters have the unsubscribe information on the bottom of every message. By legitimate, I am referring to real companies such as HP, Real Networks, Postmaster.com that received your address through legitimate means. It is much better to unsubscribe from repeat unwanted e-mails than to rely on Junk Spy. First, the bandwidth is still wasted as your computer does still have to download the file. Second, your Junk Mail Folder will contain fewer items to review, and third, you do not want to cause a legitimate company to be blacklisted.

Will Junk Spy do it all? No, not at this time and I don't ever think it will. "All", as in 100% is a very tall order. I believe that clearing 1/2 the junk is a very very good place to start. Junk Spy is a tool that we need to learn to use. It is a new tool, the only one of its kind, that Sundial is learning to build. Junk Spy users need to be responsible and assist Sundial in building the Junk Mail Database with submittals of Spam. "Just Unsubscribe", when that is an option. For true Spam, we must rely on databases and filtering - We must rely on Junk Spy for our e-mail counter terrorism.

Other Reading on Spam include.
Coping with SPAM: Should You Try to Unsubscribe?
SPAM - What can I do??
Internet FAQs Spamming
Your can reduce Unwanted Mail [This site refers to snail mail - but the logic is the same]
JunkBUSTERS Junk Email Headlines - Spamming must be stopped"

For more on Junk Spy see Sundial's site - http://www.sundialsystems.com/junkspy/index.html Junk Spy can be ordered directly from Sundial Systems for $49, which includes one year of detection database updates.

PMMail/2 also mentioned in this article is a product of Blueprint Software Works Inc. - http://www.blueprintsoftware.com/


Wilson Rook is the president of a small ISP in Michigan, Internet Walkabout. <http://www.internetwalkabout.net> started providing Internet Service in late 1995 in rural Western Michigan. Walkabout provides full Internet dialup, dedicated, and hosting services. Internet Walkabout uses OS/2 as our primary server platform. We also have Linux and NT servers. We selected the best product/OS platform for each service we offer. OS/2 6;Linux 3; NT 2

[Features]
editor@os2voice.org
[Previous Page ] [Index] [Next Page ]
VOICE Home Page: http://www.os2voice.org