How SmarterMail Uses SpamFoo

This page describes how SmarterMail installs and runs SpamFoo Client, how SpamFoo's result feeds SmarterMail's spam score, and where the files live. It is for administrators who want to understand or troubleshoot SpamFoo in SmarterMail. To turn features on, see Set Up SmarterMail.

How SmarterMail Runs the Client

When the Client Runs

SmarterMail runs spamfoo-client only while SpamFoo is needed. SpamFoo is needed when both of these are true:

  • The SpamFoo service is running in Manage > Troubleshooting > Services. It runs by default. Stopping it there is saved, so the client stays stopped across SmarterMail restarts until someone starts it again.
  • SpamFoo spam checking is on (Enable Spool Filtering), or at least one domain's Email Classification mode is not Disabled.

SmarterMail checks this when it starts, when you save the SpamFoo spam check, and a few seconds after a domain's Email Classification mode changes. It starts the client if needed and stops it if not. It also starts a stopped client on demand when a message needs scanning or when someone opens the SpamFoo dashboard.

First Start

The first time SpamFoo is needed, SmarterMail downloads and installs SpamFoo Client, sets up its license, and starts it. No setup is needed on your part. See Licensing.

The Local Endpoint

SmarterMail talks to the client at http://localhost:16253. The client listens only on localhost by default. Users reach the SpamFoo dashboard through SmarterMail's own web server, so no extra port needs to be open.

What SmarterMail Sends to SpamFoo

Incoming Mail

  • SmarterMail scans each message once per local recipient at delivery time, because results are personalized per mailbox.
  • With spam checking on and licensed, SpamFoo checks the message for spam and, when Email Classifications are on for the recipient, also sorts it into a Classification.
  • With spam checking off or not licensed, and Email Classifications on for the recipient, SpamFoo only sorts the message into a Classification. It adds no spam weight.
  • SpamFoo uses SmarterMail's own SPF, DKIM, and DMARC results instead of checking them again.

Existing Inbox Messages

When a mailbox is queued in Mailbox Processing, SmarterMail sends each Inbox message that has no Classification yet to SpamFoo to be classified. Mailboxes with the smallest Inbox go first. Messages moved into the Inbox later are classified the same way.

Corrections

Spam and not spam reports come from the webmail Report menu, from Move to Junk and Move to Inbox, and from moving a message between Inbox and Junk Email in any mail app.

  • SmarterMail holds each report for at least two minutes. Moving the message back in that time cancels it.
  • Messages larger than 50 MB are not reported.
  • Spam and not spam reports are only learned when Send classification and spam corrections to SpamFoo is on. When it is off, these reports are not sent to SpamFoo.
  • When a user moves a message to a different Classification, SpamFoo always learns from it for that user's mailbox, whether or not correction sharing is on.
  • A copy of the message text is shared with SpamFoo only when correction sharing is on and the user agreed. For a report made in a shared folder, both the person reporting and the mailbox owner must have agreed.

See How Corrections Are Used.

The SpamFoo Dashboard

SmarterMail serves the SpamFoo dashboard at /spamfoo-dashboard/ on its own web server and signs users in with their SmarterMail account and role. In a high availability (HA) cluster the path includes the node, as /spamfoo-dashboard/<node>/, so each node shows its own client.

  • Dashboard sessions last 24 hours, renew while in use, and are tied to the browser that opened them.
  • If the client can't be reached, the dashboard shows a page that explains whether SpamFoo is turned off, starting, stopped, or not responding. The page refreshes every 5 seconds, and a stopped client is started.

How the Spam Score Combines

SpamFoo doesn't decide on its own whether SmarterMail treats a message as spam. Instead:

  1. SpamFoo returns a spam verdict and probability.
  2. SmarterMail picks Spam Weight when the verdict is spam, or Not Spam Weight when it isn't. The defaults are 20 and 0.
  3. SmarterMail adds that weight to the weights from its other spam checks, such as SPF, DKIM, RBLs, and URIBLs.
  4. SmarterMail applies its spam filtering actions to the total. A total below zero counts as zero.

When SpamFoo scans a message for spam, SmarterMail records the result in the message:

  • X-SmarterMail-Spam gets an entry such as SpamFoo [Spam,prob:0.97]: 20. The first value is Spam, the Classification, or Ham (not spam) when Email Classifications are off for the recipient.
  • X-SpamFoo-IsSpam, X-SpamFoo-Probability, X-SpamFoo-SenderIP, X-SpamFoo-SenderEmail, and X-SpamFoo-Reason (the decision path, plus any phishing reasons) are added.
  • X-SpamFoo-Classification and X-SpamFoo-Classification-Prob are added when Email Classifications are on for the recipient.
  • When SpamFoo only classifies the message, with no spam scan, only X-SpamFoo-Classification is added.
  • Any X-SpamFoo-* headers the sender included are removed before delivery, so a sender can't fake a SpamFoo result.

Some mail skips the SpamFoo spam scan:

  • Authenticated senders (users who signed in to SmarterMail to send): not scanned. With Email Classifications on, the message is stamped Primary.
  • Trusted senders and contacts that pass authentication: not scanned for spam. With Email Classifications on, the message is sorted by the user's own corrections when they match, and is Primary otherwise.
  • IP addresses set to bypass spam checks and SmarterMail system messages: not sent to SpamFoo.

When SpamFoo Is Unavailable

SpamFoo never blocks mail delivery. When SpamFoo is unavailable or doesn't answer within 30 seconds:

  • The message gets a SpamFoo weight of 0. SmarterMail's other spam checks still apply.
  • If Email Classifications are on for the recipient, the message is stamped Primary.
  • Corrections that fail are logged and not retried.
  • The dashboard shows an unavailable page instead of an error.

Where Files Live

All paths start from the SmarterMail program folder, the folder that holds the SmarterMail service program files. Where the configuration and data directories go depends on how SmarterMail is installed:

Install Configuration directory Data directory
Standard SpamFoo/data SpamFoo/data
Docker spamfoo in the system data volume (/app/SystemData/spamfoo by default) Same as the configuration directory
HA cluster spamfoo in the shared system data folder, used by every node SpamFoo/data on each node

In Docker, both directories are on the mounted system data volume, so the license, models, and database survive a container rebuild.

  • SpamFoo/bin/: the client program. It is replaced as a whole on each update, so don't keep your own files here.
  • <configuration directory>/etc/: the SpamFoo license key and the keys SmarterMail and the client use to work together. Treat this folder like a password.
  • <data directory>/admin.db: the client's database.
  • <data directory>/trained-models/: downloaded models.
  • <data directory>/logs/: SpamFoo Client logs.

Updates

SpamFoo Client stays up to date in two ways:

  • While it runs: SpamFoo Client checks for new models and new versions of itself about every 6 hours. New models load without a restart. When a new version is available, the client downloads it and stops, and SmarterMail starts the new version the next time SpamFoo is needed.
  • When SmarterMail starts it: SmarterMail checks for a newer version first and installs it before starting the client. It checks at most once a day. If the check fails, it starts the version already installed.

SmarterMail never stops a running client just to update it.

For SpamFoo Client installed without SmarterMail, see Update and Back Up.

Licensing

  • If the SmarterMail license includes the SpamFoo add-on, SmarterMail uses that add-on's SpamFoo license.
  • Otherwise, SmarterMail gets a SpamFoo license for Email Classifications on its own.
  • Without the add-on, SmarterMail sends no spam scans to SpamFoo, and the SpamFoo spam check shows (Classification Only).
  • Opening Settings > Licensing makes SmarterMail check the license again, so a new or removed add-on takes effect without a restart.