SPF Helps Legitimate E-Mail Get through Spam Filters

SPF (Sender Policy Framework) is an e-mail protocol that fights return-path address forgery and makes it easier for ISPs to identify spoofed addresses. For this reason, e-mail newsletter publishers who have published an SPF record stand a better chance of their e-mail being delivered. Is it a cure-all for spam? By no means, but it is an important step in the attack on spam. E-mail publishers can use SPF to get more of their e-mail delivered.

SPF is an open e-mail standard developed in 2003-2004 by Meng Weng Wong, founder of POBox.com, and brought to the stage of becoming a recognized standard through the help many in the technical e-mail community.

How Does SPF Work?

The major ISPs, such as AOL, EarthLink, Yahoo, and MSN, are overwhelmed by spam. In order to defend themselves from a glut of spam they run every e-mail through a number of tests designed to identify spam from legitimate e-mail. These tests include whitelists (lists of certified e-mail senders), blacklists (lists of purported spammers), keyword checks (that filter out subject lines with FR** in capital letters followed by exclamations points), and lists of e-mails that that been already been identified as spam by some recipients. An SPF check is an additional check to verify that the "Return-Path:" field hasn't been forged -- pretending to be something it isn't.

The Return-Path: field is part of the header of every e-mail, but isn't usually visible unless the recipient takes another step to see the full header. To see the full header, first open the e-mail, then:

• For Office Outlook, select "View | Options" from the menu at the top of the e-mail.
• On Yahoo, selecting "Full Header"
• For AOL 9.0, select "(Details)"

The Return-Path: field is added by the sender's e-mail system.. The field is intended to contain definitive information about the address and route back to the message's originator in case the e-mail bounces It is different from the From: field that the sender has control over. However, spammers typically forge the Return-Path: field in order to disguise their identity.

When you publish an SPF record for your domain, you declare which IP addresses are authorized to send out legitimate e-mail from the domain identified in the Return-Path: field.

When an ISP does an SPF check, a pass doesn't automatically give your e-mail a clean bill of health or declare it to be spam if a record isn't found. But if an e-mail fails an SPF check, it will probably be subjected to a number of other checks -- and may get lost in the filtering process. E-mails that pass an SPF check stand a better chance of being delivered. That's why this matters.

Is SPF It Hard to Set Up?

Web hosting services are often confused about SPF. Some think they have to install it in their e-mail system. Wrong. Your web hosting service can publish an SPF record for your domain quickly and easily (item #1) without setting up SPF on their mailserver (item #2). Let me explain:

1. Publishing an SPF record. Publishing this SPF record for your domain involves pasting a line of code (generated by the SPF wizard) into the DNS record. This helps you get your e-mail delivered.
2. Installing SPF on a Mailserver. Setting up a mailserver to sort incoming e-mail based on an SPF check is more complicated, but this isn't what you are asking your web hosting service do. Some major ISPs, however, do use SPF to check incoming e-mails. Most web hosting services haven't implemented SPF for incoming e-mail at this time. However, SPF implementation is possible using patches available for MS Exchange, Postfix, Sendmail, Qmail, Exim, Courier, and others.

How Do You Self-Publish an SPF Record?

The easiest way for small businesses to set up an SPF record is to use one of two SPF wizards available online:

• SPF Setup Wizard from POBox.com.
  http://spf.pobox.com/wizard.html
• Sender ID Framework SPF Record Wizard from Microsoft.
  http://www.anti-spamtools.org/SenderIDEmailPolicyTool/Default.aspx

Both of these produce the same result -- a line beginning with: v=spf1 that will be added to the DNS zone file where you domain is identified.

Both of these wizards were designed more for IT professionals than marketers, though Microsoft's provides more explanation. Neither are easy to understand.

If your e-mail is sent out by an e-mail service such as GotMarketing, VerticalResponse, or Topica, usually that e-mail service will send out e-mail with the Return-Path: field containing an e-mail address from their own domain name. In this case they would publish their own SPF record for their own domains and you don't have to worry about it. However, if an ISP does a full Sender ID check, it would be good to publish an SPF record for your own domain also. (See below for details.)

In my case, I lease a dedicated server with a different domain name (wilsonweb.info) to send out my e-mails using AutoResponse Plus. Since the SPF protocol checks the "Return-Path" field, I need to publish an SPF record for this domain name so that all the e-mail newsletters I send from this e-mail server will pass an SPF check.

Example of SPF Wizard

(Note: What I am going to describe below covers my situation only for the sake of example. If you need to do something different, ask your webmaster or e-mail service for assistance with this.)

It will help if you open another web browser window to look at the SPF Setup Wizard while I'm explaining it, so that you can understand the instructions below. (Alternatively, you can view a PDF document showing my information filled in using the wizard.)

SPF Record for my e-mail sending domain

I set up the SPF record for my e-mail sending domain and press the "Begin" button. It does a look-up and tells me the site's IP address.

For simplicity, I say "no" to the entries for "a", "mx", and "ptr", since information using these parameters can cause unnecessary work by the recipient's server and may not be necessary for simpler cases.

Under "a:" I leave it blank and the same for "mx:", "ip4:" and "include:". There are times when each of these fields might be important, but for my simple needs for SPF, I'll leave them blank. When the server's IP address and domain name have the same reference as "a:", "mx:", "ip4:", and "include" it is redundant to list them again, and cause extra load on the receiver e-mail system for unneeded look-ups. 

Finally for "all" I put "yes." Sendmail recommends that SPF records "should end in ~all (soft fail) to avoid over-zealous rejection of messages that have their authentication broken due to unexpected forwarding by receivers."

The final text is pasted by technical support into your DNS zone record as a TXT type

And when I press the "Explain" button, each part is explained in technical terms.

Please realize that this is a very simplified example that fits my case. I could have made it more complex, but deliberately kept it as simple as possible -- even more simple than the wizard wanted to make it.

SPF Record for my main domain, wilsonweb.com

For my main website domain wilsonweb.com, I set up the SPF record as follows:

"v=spf1 ip4:69.11.214.150 ~all"

This tells anyone querying the record that another server, IP address 69.11.214.150 (wilsonweb.info), can  send out e-mails for wilsonweb.com. (See a PDF document showing this configuration in the wizard.) The ip4: field is pretty important, since it contains any other servers that can send out e-mail for my domain.

If my e-mail were sent out by an e-mail service, such as Topica, I would check with their technical support staff and then set up an SPF record for my home domain something like:

The include mechanism means that this record will include all information from the SPF record for that domain (topica.com). This assumes, of course, that topica.com has an SPF record already (which it does), or the include mechanism would be useless.

Your applications may be different, so please consult your web server's technical support person to help you with your situation.

One way to learn about SPF is to see how various e-mail senders have set up their SPF record. Both of the wizards mentioned above will search for an SPF record and display the current record if any exists. SPF Tools at spfTools.net has an Adoption Roll and Validator that will check if an SPF record exists and contains correct syntax -- and will add a domain to the list of domains that now have SPF records (nothing official, but a tally of how widely SPF is being adopted). http://spftools.infinitepenguins.net/register.php

What's the Difference between SPF and Microsoft's Caller ID?

In June 2004, Microsoft announced its agreement to merge Caller ID, its own proposed its own anti-spam authentication scheme, in addition to "classic SPF." The joint standard is called "Sender ID." This is pretty technical, so if your eyes glaze over, just move on, but here goes....

Classic SPF compares the SPF record with an e-mail's return-path: field -- that's it.

Sender ID uses classic SPF to check the return-path: field. Then it uses Microsoft's Caller ID algorithm to check the From: field in the header -- but if there is a Sender field present in the header, it will override. If there is a Resent-From field present in the header, that overrides Sender. Confused yet? :-)

Microsoft's CallerID defines an algorithm to extract from the headers what it calls a Purported Responsible Address (PRA). It is a more rigorous check, but is backward compatible with the SPF syntax and semantics that appear in the SPF record. Thus most publishers don't need to publish a separate record for Microsoft CallerID or Sender ID.

Some, however, are upset about the way Microsoft insists on licensing its technology. While Microsoft offers to license the technology to everyone for free, there was suspicion that, over time, Microsoft would hold too much control over how the standard evolves. For these reasons, the Internet Engineering Task Force (IETF) officially rejected the proposed Sender ID specification in mid-September 2004. At this point few ISPs are doing a Sender ID check.

SPF Is Controversial. Whom Should I Believe?

Some webmasters think that installing an SPF record is useless, since it won't stop spam. Others feel that SPF is only putting a band aid on a larger problem -- that the entire e-mail architecture needs to be revamped (which is probably true).

It is true that this technology won't stop spam, but it will help e-mail marketers get their opt-in e-mails delivered more consistently, since ISPs that do a SPF check may not have to analyze an e-mail in several other ways that could cause it not to be delivered. But we can't wait for that to happen.

It's a matter of perspective To webmasters and ISPs the chief issue is the curse of spam. To e-mail marketers the chief issue is curse of not being delivered consistently.

The parties who are recommending publishing SPF records include: MarketingSherpa, Sendmail, Habeas, Sophos, Symantec, Brightmail, IronPort, Ciphertrust, MailArmory, MailFrontier, Roaring Penguin Software, and Communigate Pro. ISPs that use SPF include AOL, Earthlink, SAP, Gmail, and others.

As for me, I think it's enough of an issue that I am setting up SPF records for all the major domains I own -- even though my web hosting services gave me a blank stare the first time I inquired. It can't hurt -- and it may help significantly with certain large ISPs that are desperately looking for clues to help them separate spam from legitimate e-mail. The time has come.

I want to thank technical people from Habeas.com, Topica.com, and SPF Help who suggested how I could express technical statements more precisely. However, any errors that may remain are mine alone, since I have the sometimes difficult responsibility of taking technical topics and making them understandable to mere mortals. :-)