Background:

Windows Small Business Server 2008 with built in Exchange 2007 SP1
External email screening service (anti-spam and antivirus)

Problem:

Users complaining that some emails from external senders are randomly bouncing back with NDR 550 error.

Investigation:

Searched external email filtering service logs for a particular bounced back email.
Error found:
2011-08-12 18:33:55    Recipient Disposition: [250 Backend; Mode: normal; Queued: no; Frontend TLS: no]
2011-08-12 18:33:55    Message Disposition: [550 Backend Replied: System error (Mode: normal); Backend TLS: no]

It looks like email was rejected by the in-house Exchange (SBS) server.

Checked Exchange Tracking Logs. Unsurprisingly there was no trace of the rejected message. It means message was rejected at an earlier stage.

Checked Receive Connector Protocol Logs located in C:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\ProtocolLog\SmtpReceive
Note: By default Protocol Logging in Exchange 2007 is disabled. To enable it read this article.
Searched for a particular  bounced back email.
Error found:
2011-08-12T18:33:33………[removed]………..,550 5.7.1 Message rejected as spam by Content Filtering.,

Now it’s clear that message was rejected by Exchange internal Anti-Spam system

Resolution:

Exchange Management Console > Organization Configuration > Hub Transport > Anti-Spam > Content Filtering > Properties > Action
Disable Reject messages that have a SCL rating greater than or equal to 7

In this case when company uses external email screening service we could disable Exchange Content Filtering completely. Alternately we could just increase SCL threshold.