# Dspam + Postfix = timed out while sending end of data



## cr4sh (Dec 4, 2013)

Hello. I posted this problem on the dspam mailing list but I didn't get any response. Maybe anyone here has this problem too, and can help me. I have a weird problem with dspam+PostgreSQL+Postfix.

From time to time (1-2 times per day) I get many emails in my mailq with error 
	
	



```
status=deferred (conversation with mx2.mail.pl[/var/run/dspam.sock] timed out while sending end of data -- message may be sent more than once)
```
 and I need to kill dspam, start it again and make `postsuper -r ALL` to deliver these emails.

`root@mx:/home/cr4sh # uname -a`

```
FreeBSD mx.mail.pl 9.1-STABLE FreeBSD 9.1-STABLE #0 r250047M: Wed May  1 00:34:31 CEST 2013     cr4sh@mx.mail.pl:/sys/amd64/compile/GENERIC  amd64
```

`root@mx:/home/cr4sh # dspam --version`

```
DSPAM Anti-Spam Suite 3.9.0 (agent/library)

Copyright (c) 2002-2009 DSPAM Project
http://dspam.sourceforge.net.

DSPAM may be copied only under the terms of the GNU General Public License,
a copy of which can be found with the DSPAM distribution kit.

Configuration parameters:  '--sysconfdir=/usr/local/etc' '--with-logdir=/var/log/dspam' '--with-dspam-home=/var/db/dspam' '--with-dspam-home-owner=root' '--with-dspam-home-group=mail' '--with-dspam-home-mode=0770' '--with-dspam-owner=root' '--with-dspam-group=mail' '--enable-syslog' '--enable-debug' '--enable-daemon' '--enable-clamav' '--with-pgsql-includes=/usr/local/include' '--with-pgsql-libraries=/usr/local/lib' '--with-sqlite-includes=/usr/local/include' '--with-sqlite-libraries=/usr/local/lib' '--with-storage-driver=pgsql_drv,sqlite3_drv,hash_drv' '--enable-virtual-users' '--with-dspam-mode=4510' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/info/' '--build=amd64-portbld-freebsd9.1' 'build_alias=amd64-portbld-freebsd9.1' 'CC=cc' 'CFLAGS=-pipe  -g' 'LDFLAGS= -L/usr/local/lib -L/usr/local/lib' 'LIBS=-L/usr/local/lib -pthread' 'CPPFLAGS=-I/usr/local/include -I/usr/local/include' 'CPP=cpp' 'CXX=c++' 'CXXFLAGS=-pipe -g'
```

`root@mx:/home/cr4sh # pkg info | grep postfix`

```
postfix-2.10.1,1               Secure alternative to widely-used Sendmail
```

`root@mx:/home/cr4sh # dovecot --version`

```
2.2.6
```

`root@mx:/home/cr4sh # cat /usr/local/etc/dspam.conf | grep -v ^# | grep -v ^$`

```
Home /var/db/dspam
StorageDriver /usr/local/lib/dspam/libpgsql_drv.so
QuarantineAgent "/usr/libexec/mail.local"
DeliveryHost            192.168.200.6
DeliveryPort            10025
DeliveryIdent           localhost
DeliveryProto           SMTP
OnFail error
Trust root
Trust dspam
Trust apache
Trust mail
Trust mailnull
Trust smmsp
Trust daemon
Debug *
DebugOpt process classify spam fp inoculation corpus
TrainingMode toe
TestConditionalTraining on
Feature whitelist
Algorithm graham burton
Tokenizer osb
PValue bcr
WebStats on
Preference "trainingMode=TOE"           # { TOE | TUM | TEFT | NOTRAIN } -> default:teft
Preference "spamAction=deliver" # { quarantine | tag | deliver } -> default:quarantine
Preference "spamSubject=[SPAM]"         # { string } -> default:[SPAM]
Preference "statisticalSedation=5"      # { 0 - 10 } -> default:0
Preference "enableBNR=on"               # { on | off } -> default:off
Preference "enableWhitelist=on"         # { on | off } -> default:on
Preference "signatureLocation=headers"  # { message | headers } -> default:message
Preference "tagSpam=off"                # { on | off }
Preference "tagNonspam=off"             # { on | off }
Preference "showFactors=off"            # { on | off } -> default:off
Preference "optIn=off"                  # { on | off }
Preference "optOut=off"                 # { on | off }
Preference "whitelistThreshold=10"      # { Integer } -> default:10
Preference "makeCorpus=off"             # { on | off } -> default:off
Preference "storeFragments=off"         # { on | off } -> default:off
Preference "localStore="                # { on | off } -> default:username
Preference "processorBias=on"           # { on | off } -> default:on
Preference "fallbackDomain=off"         # { on | off } -> default:off
Preference "trainPristine=off"          # { on | off } -> default:off
Preference "optOutClamAV=off"           # { on | off } -> default:off
Preference "ignoreRBLLookups=off"       # { on | off } -> default:off
Preference "RBLInoculate=off"           # { on | off } -> default:off
AllowOverride enableBNR
AllowOverride enableWhitelist
AllowOverride fallbackDomain
AllowOverride ignoreGroups
AllowOverride ignoreRBLLookups
AllowOverride localStore
AllowOverride makeCorpus
AllowOverride optIn
AllowOverride optOut
AllowOverride optOutClamAV
AllowOverride processorBias
AllowOverride RBLInoculate
AllowOverride showFactors
AllowOverride signatureLocation
AllowOverride spamAction
AllowOverride spamSubject
AllowOverride statisticalSedation
AllowOverride storeFragments
AllowOverride tagNonspam
AllowOverride tagSpam
AllowOverride trainPristine
AllowOverride trainingMode
AllowOverride whitelistThreshold
AllowOverride dailyQuarantineSummary
PgSQLServer             /tmp/
PgSQLUser               dspam
PgSQLPass               MWwDHj*********
PgSQLDb                 dspam
HashRecMax              98317
HashAutoExtend          on 
HashMaxExtents          0
HashExtentSize          49157
HashPctIncrease         10
HashMaxSeek             10
HashConnectionCache     10
IgnoreHeader DKIM-Signature
IgnoreHeader X-Bogosity
IgnoreHeader X-Spam-Checker-Version
IgnoreHeader X-Spam-Flag
IgnoreHeader X-Spam-Level
IgnoreHeader X-Spam-Status
IgnoreHeader X-GMX-Antispam
IgnoreHeader X-GMX-Antivirus
IgnoreHeader X-UI-Filterresults
Notifications   off
PurgeSignatures 14      # Stale signatures
PurgeNeutral    90      # Tokens with neutralish probabilities
PurgeUnused     90      # Unused tokens
PurgeHapaxes    30      # Tokens with less than 5 hits (hapaxes)
PurgeHits1S     15      # Tokens with only 1 spam hit
PurgeHits1I     15      # Tokens with only 1 innocent hit
LocalMX 127.0.0.1
SystemLog       on
UserLog         on
Opt out
ParseToHeaders on
ChangeModeOnParse on
ChangeUserOnParse full
ServerPID               /var/run/dspam.pid
ServerMode auto
ServerParameters        "--deliver=innocent -d %u"
ServerIdent             "localhost.localdomain"
ServerPID               /var/run/dspam.pid
ServerDomainSocketPath  "/var/run/dspam.sock"
ClientHost      /var/run/dspam.sock
ProcessorURLContext on
ProcessorBias on
StripRcptDomain off
```

`root@mx:/home/cr4sh # cat /usr/local/etc/postfix/master.cf`

```
#
# Postfix master process configuration file.  For details on the format
# of the file, see the master(5) manual page (command: "man 5 master").
#
# Do not forget to execute "postfix reload" after editing this file.
#
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
smtp      inet  n       -       n       -       -       smtpd
        -o smtpd_milters=inet:192.168.200.6:8891
        -o content_filter=lmtp:unix:/var/run/dspam.sock
```

in dspam.debug and dspam.messages *I* get nothing special, but today I get this sql.error

`root@mx:/home/cr4sh # tail -n 200 /var/log/dspam/sql.errors`

```
[12/03/2013 13:37:22] 34892: ERROR:  invalid byte sequence for encoding "UTF8": 0xb3
PODPOWIED'Z:  This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".
: SELECT uid FROM dspam_virtual_uids WHERE username='aałaa.aaaa@aaaaa.pl'
```
And in messages I have hundreds of these:

`root@mx:/home/cr4sh # tail -n 20 /var/log/messages`

```
Dec  3 17:09:22 mx postgres[34895]: [30517-1] ERROR:  duplicate key value violates unique constraint "dspam_token_data_uid_key"
Dec  3 17:09:22 mx postgres[34895]: [30517-2] STATEMENT:  PREPARE dspam_update_plan (bigint) AS UPDATE dspam_token_data SET last_hit=CURRENT_DATE,innocent_hits=innocent_hits+1 WHERE uid=41 AND token=$1;PREPARE dspam_insert_plan (bigint,int,int) AS INSERT INTO dspam_token_data (uid,token,spam_hits,innocent_hits,last_hit) VALUES (41,$1,$2,$3,CURRENT_DATE);
Dec  3 17:09:22 mx postgres[34895]: [30518-1] ERROR:  duplicate key value violates unique constraint "dspam_token_data_uid_key"
Dec  3 17:09:22 mx postgres[34895]: [30518-2] STATEMENT:  PREPARE dspam_update_plan (bigint) AS UPDATE dspam_token_data SET last_hit=CURRENT_DATE,innocent_hits=innocent_hits+1 WHERE uid=41 AND token=$1;PREPARE dspam_insert_plan (bigint,int,int) AS INSERT INTO dspam_token_data (uid,token,spam_hits,innocent_hits,last_hit) VALUES (41,$1,$2,$3,CURRENT_DATE);
Dec  3 17:09:23 mx postgres[34895]: [30519-1] ERROR:  duplicate key value violates unique constraint "dspam_token_data_uid_key"
Dec  3 17:09:23 mx postgres[34895]: [30519-2] STATEMENT:  PREPARE dspam_update_plan (bigint) AS UPDATE dspam_token_data SET last_hit=CURRENT_DATE,innocent_hits=innocent_hits+1 WHERE uid=41 AND token=$1;PREPARE dspam_insert_plan (bigint,int,int) AS INSERT INTO dspam_token_data (uid,token,spam_hits,innocent_hits,last_hit) VALUES (41,$1,$2,$3,CURRENT_DATE);
Dec  3 17:09:23 mx postgres[34895]: [30520-1] ERROR:  duplicate key value violates unique constraint "dspam_token_data_uid_key"
Dec  3 17:09:23 mx postgres[34895]: [30520-2] STATEMENT:  PREPARE dspam_update_plan (bigint) AS UPDATE dspam_token_data SET last_hit=CURRENT_DATE,innocent_hits=innocent_hits+1 WHERE uid=41 AND token=$1;PREPARE dspam_insert_plan (bigint,int,int) AS INSERT INTO dspam_token_data (uid,token,spam_hits,innocent_hits,last_hit) VALUES (41,$1,$2,$3,CURRENT_DATE);
Dec  3 17:09:23 mx postgres[34895]: [30521-1] ERROR:  duplicate key value violates unique constraint "dspam_token_data_uid_key"
Dec  3 17:09:23 mx postgres[34895]: [30521-2] STATEMENT:  PREPARE dspam_update_plan (bigint) AS UPDATE dspam_token_data SET last_hit=CURRENT_DATE,innocent_hits=innocent_hits+1 WHERE uid=41 AND token=$1;PREPARE dspam_insert_plan (bigint,int,int) AS INSERT INTO dspam_token_data (uid,token,spam_hits,innocent_hits,last_hit) VALUES (41,$1,$2,$3,CURRENT_DATE);
Dec  3 17:09:23 mx postgres[34895]: [30522-1] ERROR:  duplicate key value violates unique constraint "dspam_token_data_uid_key"
Dec  3 17:09:23 mx postgres[34895]: [30522-2] STATEMENT:  PREPARE dspam_update_plan (bigint) AS UPDATE dspam_token_data SET last_hit=CURRENT_DATE,innocent_hits=innocent_hits+1 WHERE uid=41 AND token=$1;PREPARE dspam_insert_plan (bigint,int,int) AS INSERT INTO dspam_token_data (uid,token,spam_hits,innocent_hits,last_hit) VALUES (41,$1,$2,$3,CURRENT_DATE);
Dec  3 17:09:23 mx postgres[34895]: [30523-1] ERROR:  duplicate key value violates unique constraint "dspam_token_data_uid_key"
Dec  3 17:09:23 mx postgres[34895]: [30523-2] STATEMENT:  PREPARE dspam_update_plan (bigint) AS UPDATE dspam_token_data SET last_hit=CURRENT_DATE,innocent_hits=innocent_hits+1 WHERE uid=41 AND token=$1;PREPARE dspam_insert_plan (bigint,int,int) AS INSERT INTO dspam_token_data (uid,token,spam_hits,innocent_hits,last_hit) VALUES (41,$1,$2,$3,CURRENT_DATE);
Dec  3 17:09:24 mx postgres[34895]: [30524-1] ERROR:  duplicate key value violates unique constraint "dspam_token_data_uid_key"
Dec  3 17:09:24 mx postgres[34895]: [30524-2] STATEMENT:  PREPARE dspam_update_plan (bigint) AS UPDATE dspam_token_data SET last_hit=CURRENT_DATE,innocent_hits=innocent_hits+1 WHERE uid=41 AND token=$1;PREPARE dspam_insert_plan (bigint,int,int) AS INSERT INTO dspam_token_data (uid,token,spam_hits,innocent_hits,last_hit) VALUES (41,$1,$2,$3,CURRENT_DATE);
Dec  3 17:09:24 mx postgres[34895]: [30525-1] ERROR:  duplicate key value violates unique constraint "dspam_token_data_uid_key"
Dec  3 17:09:24 mx postgres[34895]: [30525-2] STATEMENT:  PREPARE dspam_update_plan (bigint) AS UPDATE dspam_token_data SET last_hit=CURRENT_DATE,innocent_hits=innocent_hits+1 WHERE uid=41 AND token=$1;PREPARE dspam_insert_plan (bigint,int,int) AS INSERT INTO dspam_token_data (uid,token,spam_hits,innocent_hits,last_hit) VALUES (41,$1,$2,$3,CURRENT_DATE);
Dec  3 17:09:25 mx postgres[34895]: [30526-1] ERROR:  duplicate key value violates unique constraint "dspam_token_data_uid_key"
Dec  3 17:09:25 mx postgres[34895]: [30526-2] STATEMENT:  PREPARE dspam_update_plan (bigint) AS UPDATE dspam_token_data SET last_hit=CURRENT_DATE,innocent_hits=innocent_hits+1 WHERE uid=41 AND token=$1;PREPARE dspam_insert_plan (bigint,int,int) AS INSERT INTO dspam_token_data (uid,token,spam_hits,innocent_hits,last_hit) VALUES (41,$1,$2,$3,CURRENT_DATE);
```

If anyone can help me with this issue, I will be grateful.


----------



## cr4sh (Dec 6, 2013)

No one can help me with dspam?

Best regards.


----------



## erwin (Feb 16, 2014)

Setting "standard_conforming_strings = off" in your postgresql.conf should help with the escaping problems. But i myself have still problems with primary key.


----------

