How to find your domains MX record - NSLOOKUP

If you recently added a new domain to your mail server you may need to wait for DNS to propagate the MX records, which could take up to 72 hours, here's a couple of tools you can use to lookup the MX records to see if it did propagate.

• NSLOOKUP (DOS command)
• IP Tools (Online utility)

NSLOOKUP. Open a DOS prompt and type 'nslookup', without the single quotes, it should show what DNS server it's using (usually the default for your Ethernet adapter) and the DNS server IP. Type 'help' for a list of all commands. Then type 'set q=mx'. This tells nslookup to lookup the MX records for the domain entered at the nslookup prompt. For example, type your_domain.com and the MX records will be returned, if the domain has MX records. For example, type google.com and you will see results similar to the following,

Non-authoritative answer:
google.com MX preference = 10, mail exchanger = smtp2.google.com
google.com MX preference = 10, mail exchanger = smtp3.google.com
google.com MX preference = 10, mail exchanger = smtp4.google.com
google.com MX preference = 10, mail exchanger = smtp1.google.com

smtp2.google.com internet address = 64.233.165.25
smtp3.google.com internet address = 64.233.183.25
smtp4.google.com internet address = 72.14.221.25
smtp1.google.com internet address = 209.85.237.25
>

The results show that google.com has 4 MX records, all set to preference 10 (kind of like a round robin approach), and then it list the IP's for each MX hostname.

Another way to use nslookup is to change the DNS server it uses to see if other DNS servers report different results than your default DNS server configured on your Ethernet adapter (TCP/IP settings). After entering the nslookup prompt type 'server new_dns_server_IP', where new_dns_server_IP is the DNS server IP or hostname you want to use to check records against, it could be a secondary DNS server on your local network or your ISP's DNS server. Then perform the lookup on your domain to see what that DNS server reports.

IP Tools. IP tools, located at www.iptools.com, is a great tool for checking more than just MX records, you can check rDNS, IP's, A records, etc... what's handy about IP Tools is that you get a "second opinion" on DNS records or MX records for a particular domain, just in case your concerned about your DNS cache or TTL on your DNS servers.

Must have Anti-spam Tools for Mail Servers

Most mail servers come with anti-spam tools that are not enabled by default and I thought I'd share my thoughts on what options are effective in fighting spam. If these anti-spam technologies are included in your mail server you should consider enabling these options. Hopefully this guide will encourage you to explore these options and reduce spam before it reaches your inbox, and who doesn't want that!

1. DNS Blacklist (DNSBL), or sometimes called RBL, is a very effective method for fighting spam. When your mail server receives an incoming connection the IP address of that mail server is sent to a DNSBL service. If the IP is listed in the database for that DNSBL service it returns a true statement to your mail server. This tells your mail server that the IP is a known spammer and your mail server will typically reject the incoming connect immediately. If DNSBL returns false the SMTP session will continue. This usually happens after the SMTP 'Mail From:' command. This is a very efficient method for fighting spam since no data is sent to the mail server, only the sender information.
   
2. Reverse DNS (rDNS). This is another very efficient technique. This involves finding the hostname of the connecting IP address. How is this different from regular DNS? When resolving a domain name your system uses DNS to find the IP for a particular domain, your DNS server will then return the IP address. When your mail server performs rDNS it reverses the IP and sends it to the ISP's DNS server responsible for issuing that IP address and looks for the existance of PTR or hostname entry. Typically PTR records are created for static IP's but not dynamic and a fair number or spammers send from dynamic IP.
3. Spam Assassin. This is likely already enabled on your mail server but I wanted to mention it because there's many different components of Spam Assassin that's worth checking into; taking full advantage of all it's features. For example, it has header test, Bayesian, charset, DNSBL, Razor2, and many other features that can be adjusted to better combat spam, usually by increasing the score weight for a particular test.
   
4. Bayesian Filtering. This approach takes spam probabilities of individual words (in the body of the message) and weighs the combination of words to conclude the percent probability for being spam. Usually this is used in conjunction with your mail servers spam scoring system and increases the total spam score.
5. Sender Policy Framework (SPF). SPF uses DNS to publish a policy record for a senders domain. When checking for SPF your mail server takes the sender domain and looks up the policy to see if it complies, which typically includes the IP address of the mail server and what hostnames are allowed. Your mail server may have options that temporarily reject failed SPF or add to the spam score if SPF fails.
   
6. Grey Listing. This is a fairly straight forward approach that takes advantage of the fact that spammers are lazy. When your mail server accepts a connection it will look up the IP in a database of allowed Greylisting IP's, if it's not in the list the IP will go through the Greylisting process. Your mail server will typically allow the connecting server to report the sender address and respond with 451 temporarily unavailable. This is where spamming servers usually give up. Normal mail servers will resend the message at a later time and then be accepted for delivery.
   
7. Greeting delays. Spammers don't usually want to wait around trying to deliver a single message and will timeout in a very short period of time. If you put a delay (usually 20-25 seconds) on your mail server greeting message this can stop some spammers. Use this with caution, some mail servers won't wait more than 30 seconds for greeting messages.
8. Relaying Security. Although relaying is not technically an Anti-spam tool it's important to ensure that your mail server is not allowing relaying as spammers take advantage of open relays to send most of their spam. See my post on Open Relay - Stop Spam Relaying.
   

Couple of notes about using DNSBL services. There are many different DNSBL that provide different services. If you can customize your mail server to use a couple different services consider using one for spam and another for relaying. I'd recommend using bl.spamcop.net for spam. Refer to DNSBL comparison at Wiki for more services.

Open relay - Stop spam relaying

You come into the office in the morning, login to your computer to check mail and find that the mail server is really slow or not responding at all. So you remote in to the box and find that your mail servers outgoing queue is full of messages destine for domains you never send to and from senders not from your domain. "What the heck's going on?" Have you been a victim of spam relaying? Very possibly, yes, here's a few tips on how to diagnose this problem and how to prevent future spam relays.

Topic of discussion:

1. What is relaying?
2. How spammers are able to relay, what methods are used?
3. How to determine if relaying is occurring.
   
4. SMTP log details of spammer relaying mail.
5. How to secure relaying attempts and prevent future relay attempts.
   

What is relaying:

When a mail server receives an incoming message and it's recipient address is to a local domain defined by your mail server it is considered local mail and placed in the local POP3/IMAP account. If the recipient is to a remote address (to a domain that is not defined by your mail server) this is considered relaying. The mail server should perform several checks on the connecting IP to ensure that it is trusted. The connected IP is typically trusted using one of three relay checks:

• Is a predefined trusted IP. Usually your private network.
  
• POP3 before send. The IP must have authenticated to POP3 or IMAP before sending.
  
• SMTP Authentication. Can be used by the connecting IP and is part of the ESMTP protocol.

How spammers are able to relay:

Use private network. Most mail servers are configured to only allow relaying from the local private network and spammers are less likely to find ways to send spam via your private network, unless they find a way into your network using a virus or some other means. In my experience spammers are lazy and will first attempt to use automated ways to find weak passwords on your mail server so they can relay, however, when troubleshooting relaying issues never count out the possibility that your local network has been compromised, virus or otherwise.

Authenticate to mail server. POP3 before send or SMTP authentication can be used to relay mail, but how do spammers find your username and password allowing them to authentication? One common method used is POP3 harvesting attacks. What happens is the harvester connects to your POP3 server and sends a common username and password, if it fails they send another username and password until they find a valid username / password combination. I've come across this issue 100's of times and it's unbelievable how many usernames and passwords are the same, typically email users change their password to match their alias making it easier to remember, which opens a huge security hole in your mail server.

How to determine if relaying is occurring:

Stop the SMTP service and clean out the outgoing queue. If a spammer (this is usually an automated process running on a zombie computer) is activity relaying mail through your mail server you need to stop your SMTP service immediately. If not, you might find your mail servers IP being added to black list servers. Removing your IP from black list servers can be just as time consuming as cleaning up spam relaying issues. Most mail servers provide you with the ability to delete messages in the outgoing queue, simply delete what's obviously spam and check the following to determine what happened.

Check the POP3 logs for harvest attempts, and note the connecting IP. Here's an example.

+OK test.com Mail Server v1 POP3
user sales
+OK sales
pass sales
-ERR Unknown user or incorrect password
user info
+OK info
pass info
-ERR Unknown user or incorrect password
user test
+OK test
pass test
+OK 0 messages (0) octets
quit
+OK test.com closing connection

POP3 Logs. Your POP3 logs will show the username but not the password so you will need to look for reoccurring attempts where a valid username/password was found, look for the +OK after the pass command. Once you know the username you can reset that accounts password. Another tip is to look for the largest sized log, this is likely when the harvest started. Also search your SMTP logs for the same IP address. If you find the IP, and your mail server allows POP3 before send, this will allow that IP to relay.

SMTP Logs. A more common scenario is that the newly discovered open relay is posted to the big cloud in the sky and zombie spamming computers will use SMTP authentication to relay spam through your mail server.

SMTP log details:

Encoded username and password. Successful SMTP authentication.

>>> 250-host-name Hello remote-client [1.1.1.1].
<<< AUTH LOGIN
>>> 334 VXNlcm5hbWU6
<<< dGVzdA==
>>> 334 UGFzc3dvcmQ6
<<< cGFzcw==
>>> 235 2.0.0 Authentication successful

This is only a partial log showing successful authentication. This is what you will see in the SMTP log to determine what account was used for relaying. Search the logs for 'AUTH LOGIN', without the single quotes. The username and password are base64 encoded and can be decoded from the following site. For example, the username above is dGVzdA. The password is cGFzcw. Copy the data and paste into the site below to decode.

Base64 Decoding

Decoded username and password. Here's an example of the data decoded.

>>> 250-host-name Hello remote-client [1.1.1.1].
<<< AUTH LOGIN
>>> 334 Username:
<<< test
>>> 334 Password:
<<< pass
>>> 235 2.0.0 Authentication successful

Once successfully authenticated you should see the session continue with delivery to 100's if not 1000's of remote recipients. Once you know which account was used to relay you can change the password. In our example above it was account test. Keep in mind that more than one account may have been compromised. Continue to search the logs for other login attempts.

How to secure relaying attempts and prevent future relaying attempts:

• AUTH policies. Most mail servers allow all authentication types for SMTP. This is best practice so your mail server supports the widest range of connecting clients. For example, AUTH CRAM-MD5, PLAIN, LOGIN, and DIGEST-MD5 are defaults. If you change the AUTH defaults you may encounter problems where some clients (Outlook only supports AUTH LOGIN) can not authentication. This will result in a 'We do not relay' error. The logs will report some type of AUTH exchange to be considered trusted.
• Restrict failed login attempts. Set POP3 login failures to 5 to slow down POP3 harvesting.
  
• Trusted IP addresses. Know which IP addresses are trusted on your network and when possible trust only private IP addresses.
• Password policies. Create password policies that force mail users to change passwords often. Passwords should also contain alpha-numeric and non alpha-numeric characters.
• Limit recipient deliveries. Limit the amount of recipients your mail server will accept in a single SMTP session. This will slow down the spammer.

How to view SMTP traffic with a Network Capture tool

To troubleshoot or analyze a particular problem on a mail server you may need to take a closer look at the actual SMTP traffic being sent to and from your mail server. Using a Network Capture tool can uncover problems not seen by the mail server. Let's take a look at what can be accomplished with this tool.

• You can check whether SMTP traffic is arriving at the mail server, verifying that traffic is not blocked by a firewall or router sitting between the mail server and the Internet or an email client.
• You can check if DNS requests are being made to your DNS server (MX lookups).
• You can check the exact format of DATA being sent to and from your router.
• You can view email client request for POP3, IMAP, or SMTP.
  

How to create a SMTP Network Capture:

1. Download and install a Network Capture tool. In our example we'll use Wireshark, which can be downloaded from www.wireshark.org. Note: if you have a firewall installed on your mail server computer it may interfere with how the network capture works and you may need to temporarily disable the firewall to get a complete capture.


2. Start capture. Start the capture by selecting Start from the Capture --> Interfaces... menu. If you have multiple Interfaces select Start on the Interface that sends and receives email. After pressing the Start button the network capture will begin.

3. Enter filter term. Enter the term 'smtp', without the single quotes, in the Filter box of the main screen and press Apply.

4. Reproduce problem. Attempt to reproduce the particular problem you want to analyze, for example, have an email client send mail.


5. Displayed SMTP traffic. You will see the SMTP traffic appear in the main window. When done, you can stop the capture and optionally save the capture to file for future analysis.

Definitions of capture:

Source column. The source is where the packet originated, for example, if your local IP is in the Source column this means the mail server initiated that packet.

Destination column. The Destination is the IP that the Source is sending to, for example, if the destination is a remote IP address this is where the mail server is sending the packet.

Protocol column. The Protocol is the type of request made by the Source. In my example I'm filtering on SMTP so you should only see SMTP in the Protocol column. If you changed filtering to DNS you would see all captured DNS/ICMP packets.

Analysis:

If you want to get a little more involved in analyzing the packets on your mail server you can remove the 'smtp' filter, press Apply, and view all packets on the selected interface. On a busy system this will likely flood the live capture with a lot of packet data and requires a good understanding of networking to analyze all the data. For the sake of simplicity I'm not going to go into a complete packet capture but only focus on SMTP. However, it may be necessary to analyze all packets to find problems on your network not related to the mail server. For more info on packet errors and other network captures see the Wireshark help, they actually have very good help files.

Once you have captured the required SMTP data you can analyze it by clicking on an SMTP line at the top and it will show the related capture in the corresponding fields below that, the actual data is located in the bottom field and provides you with the data being sent back and forth between the mail server and the connecting client or server.


SMTP Capture Explained. I'm showing a successful SMTP session but in the event when SMTP fails you would see responses from the connected client or server that should provide details of why the session failed. For example, let's say your analyzing the DATA command to see if all data is sent to your mail server, including the "." on a line by itself indicating the end of DATA transmission (see the Wiki SMTP description for more info on communications). If your network capture does not show the "." on a line by itself your mail server will assume that there's more DATA to receive and will wait until it does receive the ".", or it will timeout. The error in your SMTP log will likely report a failed session without real cause for failure.

Causes for Failure. Likely causes for this type of failure are down stream gateways, routers attempting to filter mail prior to sending to your mail server, desktop virus scanner on your mail server interfering with delivery, the sending client or server is not complying with SMTP protocol, or network configuration issues (MTU, etc...) or there could be a hardware failure or network card driver issue.

Conclusion. If you are unable to determine why email delivery is failing and your mail server logs are not providing enough information about the failure, try using a network capture tool to capture the actual SMTP traffic, the capture may provide enough details that lead you to a solution to the problem.

Check Security Cipher on Webmail or Web Site

I recently found a free tool that does a good job of checking SSL/TLS Ciphers for your secure Webmail or web site. This is handy when you need to determine if your server needs SSL updating or if you simply want to know what ciphers your server supports, some ciphers are known to be insecure.

SSLDigger v1.02 can be downloaded here.

Sysinternals Troubleshooting Utilities

Sysinternals Utilities has some of the best troubleshooting tools for Windows operating systems. They were created in 1996 by Mark Russinovich and Bryce Cogswell and later acquired by Microsoft in 2006. I've been using these tools since before Microsoft acquired them and find that 4 of the utilities are a must have, pskill, regmon, filemon, and process explorer. Filemon in particular provides real-time file monitoring showing what applications are accessing what files and whether there were read/write errors, access or otherwise; extremely valuable information when troubleshooting file based servers, such as, most mail servers. Pskill can terminate processes that will not stop. Very useful on Windows 2000 boxes where task manger will not allow you to end a task.

Now called Windows Sysinternals, accessible from Microsoft's Technet web site or www.sysinternals.com. I highly recommend that you check out these utilities to see if one of them can get you out of a troubleshooting bind!

What ports are in use - error 10048 - NETSTAT

Ever attempt to start a service that will not start or pops up a warning stating that another service is listening on that port with error 10048? This happens when a service is using the listening port that your service is attempting to listen on.

This commonly happens with SMTP on Windows 2003. For example, lets say you want to trial a new mail server and you install it on Windows 2003. After the install you receive error 10048 or the SMTP services will not start. The reason this may happen is because Windows 2003 has it's own SMTP server and will occupy port 25 not allowing other services to listen on that port (this is true for all services not just SMTP and the same applies to other operating systems). Here's a way around this problem and how to use NETSTAT to see if port 25 is occupied.

Open a DOS prompt and type the following (type netstat -? for help).

netstat -an

The response will list the TCP listening ports first and then the UDP ports. If the Windows 2003 SMTP service is listening on port 25 you will see this entry,

TCP 0.0.0.0:25 0.0.0.0:0 LISTENING

or

TCP 0.0.0.0:smtp 0.0.0.0:0 LISTENING

To disable Windows 2003 SMTP service go to Administrator Tools - Services and stop the Simple Mail Transfer Protocol service and set it to disabled. Now issue the netstat -an command again to see if port 25 is available, if so, you should be able to start your SMTP service.

Another side affect of the Simple Mail Transfer Protocol service on Windows 2003 is that it magically starts by itself, I've seen this happen a few times and suspect it's directly related to updating windows or updating IIS. If the Simple Mail Transfer Protocol service does start after you've disabled it your email clients will report error 'We do not relay' (which is a response from the Simple Mail Transfer Protocol service) and you will need to stop and disable the service again or at least check to see if another SMTP service is listening on port 25 using netstat.