Introduction
Today, over 95% of all e-mail traffic on the Internet is spam. Spammers have grown increasingly sophisticated, using innovative methods to trick spam filters and penetrate email inboxes. The majority of spam and malware are sent by networks of compromised computers ("botnets") that launch millions of distributed attacks without the user's knowledge. To make matters worse, new classes of spam are extremely difficult to detect and harmful, containing spoofed addresses, rapidly-mutating images and embedded links to malware sites.
Traditional spam filters that rely on static rules, traffic patterns or heuristics have been ineffective against advanced messaging threats due to their slow response time and tendency towards high positives. In order to win the war against spam, businesses must use a real-time defense system that can instantly respond to evolving attack vectors.
Cloudmark accomplishes this through a combination of Advanced Message Fingerprinting™ and the world's largest threat detection network, the Global Threat Network. These components, together with automated analytical processes, drive the system's unsurpassed 98 percent spam filtering accuracy with near zero false positives.
The core of Cloudmark's solution is its Advanced Message Fingerprinting, which employs high performance fingerprinting algorithms to target different threat attributes embedded in a message. As each message comes in, Cloudmark uses these algorithms to generate a set of "fingerprints" to represent unique aspects of the message. Once a fingerprint has been determined to be associated with a spam, phishing or virus attack, future messages containing that same fingerprint will be automatically blocked. Through this method, Cloudmark is able to identify mutations and variants in real time.
For new outbreaks, Cloudmark provides extremely fast coverage through its Global Threat Network, consisting of over 300 million reporters in 200 countries. With Cloudmark, threat monitoring comes not from employees of an anti-spam company, but from a worldwide network of trusted users, enterprises, honeypots and system administrators. Feedback from these reporters enables Cloudmark to block the latest threats within minutes of attack origination, this approach contributes to faster threat detection and more accurate message classification.
All feedback by the Global Threat Network is corroborated and analyzed by the Trust Evaluation System. This system tracks the reporter's reputation and determines fingerprint classification based on the number of reports and reporter's reputation. Reputation or "trust" is earned over time by consistently reporting correct abuse feedback. Also, since feedback is continuously corroborated, any inaccuracies in message classification, such as false positives or false negatives, are corrected in near real time. No other system offers this kind of feedback review and seamless remediation.
Cloudmark Advantages
- Comprehensive protection against spam, phishing and viruses
- Language and content-agnostic analysis stops spam in all languages (including double-byte characters such as Chinese, Japanese and Cyrillic) and formats, including image spam, .pdf spam and virus executables
- Through 24 x 7 monitoring, the Global Threat Network ensures the fastest response to new spam outbreaks as well as highly-targeted phishing attacks and virus outbreaks
- Advanced Message Fingerprinting algorithms detect variants and rapidly mutating attacks before they reach e-mail inboxes
- Automatic remediation of false negatives or false positives through continuous administrator and user feedback
- Cache of threat fingerprints is updated every 60 seconds for protection against the latest attacks
Administrator and User Controls
- Email users can add senders, domains, and IPs to their safelist to ensure that email from these senders is never filtered as spam
- Email users can add senders, domains, and IPs to their blacklist to ensure that email from these senders is always filtered as spam
- Email users have several options for handling email once it is flagged as spam
- Email users can choose the exclusive setting, which allows them to block email from senders outside of their safelist
- Email administrators can also manage global spam preferences, safelists, and blacklists at the domain and the user level
Email Antivirus Process
Our antivirus system scans all inbound and outbound emails using a multi-stage process. The process is broken down into the following four stages:
Stage 1: Restricted Attachments
Virus protection starts with scanning messages for dangerous types of file attachments. Dangerous files are those that can execute code, which can be used by malicious persons to spread viruses or do harm to your computer. Restricted file types include, but are not limited to, program files (.exe, .com), script files (.bas, .vbs, .js), and shortcuts to files (.lnk, .pif). When an email is sent or received that contains a restricted file attachment, the email is rejected and the sender receives a "bounced" email notification informing them of the restriction.
Stage 2: Normalization
This stage of the email antivirus process searches for email formatting vulnerabilities that can be used by viruses to hide from virus scanners. If any vulnerability is found, our system corrects the formatting of the message so that it can be thoroughly scanned for viruses. This is called "normalizing" the message, and most notably this process protects against all known Microsoft Outlook security threats.
Stage 3: Decompression
Next, if the email contains any compressed attachments such as zip files, the compressed attachments are temporarily unzipped so that the contents can be scanned for viruses. Many of today's viruses use compression as a way to sneak their way past virus scanners, sometimes even compressing themselves in several layers to try to hide from scanners. If an attachment cannot be decompressed, such as might be the case with password protected zip files, the original file is scanned for virus signatures that occur within compressed attachments.
Stage 4: Virus Scan
After the above preprocessing is complete, an email antivirus scanner is used to scan the email and all of its uncompressed attachments. Everything is scanned to ensure maximum protection against new virus threats. ClamAV (www.clamav.net) is the current scanner of choice, although our system was designed to be able to plug-in any virus scanner on the market, should the need to do so arise. Updated virus definitions are automatically pushed to our system. This gives our customers protection from new viruses within minutes. Virus definitions are updated hourly. In contrast, most desktop and server anti-virus programs are configured to check for new virus signatures only once per day.
Virus protection is included at no additional cost with all Email and Microsoft Hosted Exchange accounts.
Threat "Fingerprints"
Cloudmark's Advanced Message Fingerprinting™ algorithms are designed to target sophisticated spamming and virus proliferation techniques. By fingerprinting only relevant threat attributes in messages, Cloudmark is able to identify spam, phishing and viruses that have undergone polymorphic changes in text, URL, image, sender or other attributes. As each message comes in, these algorithms generate a set of fingerprints that represent unique aspects of the message. Message fingerprints are matched against a database of known "bad" fingerprints, which is updated every 60 seconds. If there is a match, the message will be blocked without the need for reporting.
Completely new outbreaks are rapidly detected through a global network of millions of reporters, consisting of system administrators, end users, honeypots and other sources. All feedback is corroborated and analyzed by Cloudmark's Trust Evaluation System (TES). TES tracks the reputation of each reporting source. Trust is earned over time by consistently reporting correct abuse feedback. This system preserves the integrity of reports and ensures that the system is extremely accurate. Since feedback is continuously corroborated, any inaccuracies in message classification are corrected in near real time. No other system offers this kind of continuous feedback loop.
Zombies, Open Relays, and Known Spam Sources
A "zombie" is a computer that has been taken over by a spammer, and which is used to send out bulk mailings without the knowledge of the computer's owner. Normally this occurs because the computer owner opened a virus, which gave a spammer a back door into their system. This is one of the most common sources of spam. Another prevalent source of spam is "open relays"—insecure mail servers that can be used freely by spammers. Spammers use automated tools to scour the Internet in search of vulnerable mail servers, and then hijack those servers to increase the amount of spam they can send.
To combat this problem, there are several third-party organizations that maintain databases ("blacklists") that list the IP addresses of these compromised machines. There are also databases that list known professional spammers. We have arrangements with approximately 15 of these organizations, so that our system can download full copies of the blacklists hourly and incorporate them into the spam filtering system.
DNS and RFC Violations
Spammers tend to be careless in how they send email. Therefore it is important to scrutinize each inbound email to see if it followed the rules defined by current Internet standards. For example, below are just a few of the tests that examine the sending mail server and the message:
Did the mail server falsely identify itself in the "HELO/EHLO" data?
Does the mail server have a missing or invalid reverse DNS record?
Is the domain missing "A" and "MX" DNS records or using illegitimate values?
Was there an SPF violation? (Was the email sent from a mail server that is not authorized to send mail using the sender's domain name?)
Are message headers improperly formatted or missing required data?
There are more than 1,000 of these tests. Blocking based on any test alone would block a large amount of legitimate email; however, these tests are extremely effective when used with a filtering system that combines message fingerprinting and real-time threat reporting.
Elusive Spammers
Spammers are very aware of the filtering techniques used by top-tier business email service providers such as Blackrock Networks, Inc.. This has led them to develop creative tactics and advanced software in an attempt to beat anti-spam systems. For instance, many spammers use binary encoding to hide their text and HTML email from signature-based filters. It is also very common for spam to include invisible HTML code intermixed with the visible content, and subtle variations in wording and punctuation, as well as purposely misspelled words.
The Cloudmark system is completely text-, language- and format-agnostic. Cloudmark's Advanced Message Fingerprinting algorithms is resistant to polymorphic changes in text, URL, image, sender or other attribute as only relevant ("spammy") parts of the message are fingerprinted and tracked.
Combining the Tests
After rigorous testing, a final confidence level is assigned to each email. The confidence level is compared against a threshold to determine if the email should be identified as spam. If the email confidence level is lower than the threshold, the email is deemed to be spam-free and is delivered normally. If the confidence level is greater than the threshold, the email is identified as spam.
What to do with spam Once a spam email has been identified, there are several actions that can occur based on administrator and/or user preferences: Deliver to Spam folder. This will allow each user to review the emails that have been tagged as spam. This folder can be viewed from webmail or IMAP, and settings are available to automatically delete old spam from this folder after a certain number of days or number of emails Tag the subject. The word "[SPAM]" will be added to the beginning of the subject line, and then delivered normally. This allows each user to set up custom filtering rules inside of desktop mail programs, such as Microsoft Outlook. Forward to an alternate email address. This is useful if a company wants to have a single administrator review all of the spam that their users receive. Delete the email. In this case, users will never see the spam.
False-Positive Prevention
Every effort is made to ensure that legitimate emails are not falsely identified as spam ("false-positives"). The Cloudmark system is able to automatically self-correct, therefore reducing incidents of false positives as well as false negatives. Reporters can provide feedback on a missed spam message or on a legitimate message which has been misclassified as spam. Once this feedback is corroborated, the message is automatically re-classified. In addition, Cloudmark's system factors in reputation data from leading reputation services providers Return Path and Habeas.
Sender Score
ReturnPath's Sender Score system provides us with access to a comprehensive email reputation database. The data is compiled from more than 40 million mailboxes across a number of email providers, including Blackrock Networks, Inc.. Sender Score looks at things such as complaint data, mail volume, bounced messages, unsubscribe functionality, security practices, identity stability, and more. Think of it as a credit score for email senders. When an email arrives, particularly bulk mail, we are able to tell if the sender is a "good" sender or a "bad" sender.
Safelists (aka "whitelists")
While every effort is made to ensure that legitimate emails are not identified as spam, a small number of false-positives are unavoidable. To solve this problem, domain administrators and email users can specify trusted email addresses, domain names, and mail server IP addresses that should always bypass the spam filtering system. This feature should be used when specific emails sometimes get identified as spam, such as opt-in newsletters or emails from colleagues whose mail servers are blacklisted or configured improperly.
Blackrock Networks, Inc. also maintains a system-wide safelist. This list tracks mail servers run by large companies such as IBM, FedEx, and Paypal, as well as popular discussion lists such as those hosted by SourceForge.net and Yahoo Groups.
Internal Mail
When you send an email to other people at your company, you obviously do not want it to be blocked as spam. Therefore, whenever our customers send mail from our servers, that mail automatically bypasses the spam filtering system. This ensures that internal mail will be delivered reliably.
In addition, there is both a domain-level and a user-level safelist. Domain administrators can add senders to the safelist, and it takes effect for all users in the domain. Users can create their own personal safelist, and it takes effect just for their account. Both the domain's safelist and the user's safelist work in conjunction with each other to ensure that mail
For customers desiring extreme spam protection, the Exclusive filtering level can be used to block all email from senders not appearing on their safelist. This will cause email from all unknown senders to be rejected by the SMTP server.
A distributed database tracks real-time information about connections to our servers and where spam and viruses are currently being sent from. This information is used to rate-limit mail from likely spammers and from emerging spam and virus sources. Rate-limiting is a safe way to reduce spam because mail is never blocked. Instead, mail is allowed through, but at a much lower rate. This permits legitimate mail to be delivered, while slowing down high-volume spammers.
This system also protects the Blackrock Networks, Inc. servers from denial of service ("DoS") attacks by limiting the number of concurrent and successive connections third-party servers can make.
A directory harvest attack is an attempt by a malicious person to find out the email addresses that exist within a domain so that they can later send spam to those addresses. Attackers do this by sending a series of connections to a SMTP server pretending to deliver mail to a large quantity of randomly selected email addresses, and collecting the responses from the server. The SMTP responses normally indicate whether or not each email address exists, thus allowing a spammer to compile a list of valid email addresses. This is also known as a "Dictionary Attack," because the attacker literally runs through a list of thousands of common names that can make up an email address.
We protect customers from Directory Harvest Attacks by automatically disconnecting spammers who send mail to too many unknown recipients. Subsequent connections are throttled so that the attacker cannot establish new connections at a rapid rate. This greatly reduces the chances of our customers' email addresses ending up on spammers' mailing lists. Similarly, our servers reject mail when a spammer uses a forged FROM address; i.e., when they try to guess an email address within the domain that they are spamming in hopes that this gets their mail whitelisted and delivered. If they guess an unknown address, we reject the message.
In the rare case that one of our IPs becomes blacklisted, engineers receive an alert from the monitoring system. Immediately, our engineers take action to diagnose and stop whatever caused the blacklisting to occur. During this time, the IP address that was blacklisted is removed from use so that no outgoing mail is sent from it. If an entire range of IP addresses become blacklisted, fallback routing is used to route mail through an alternate cluster of SMTP servers with clean IPs. Once the issue has been resolved, we contact the blacklisting organization in order to get the IP address removed from the blacklist.