Need a little help SMTP email send program syntax, terminating string.

[Jester]

I’m trying to resurrect an Arduino 8266 based email sender program that was working fine in 2023. Since then I changed email provider service and now with the new email, this program fails at step 32 of 33.
The program successfully:
  Logs into local wifi,
  Logs into email providers server,
  Passes authentication,
  Provides to and from email addresses,
Appears to send body text of email, and then the server times out at step 32 of 33

I think the problem is with the terminating string that is supposed to convey the end of the text in the email. In the 2023 version the text was terminated with a simple “.” On a new line, it appears that the 2024 version wants a <CR><LF>.<CR><LF>

Using the 2023 terminating string yields 550 5.7.1 Message Rejected (104) [F3EC52000E/1721829661-832712]

Using the 2024 terminating string appears to try and send the email , however it times out after about 20 seconds trying.

I have attached both the program (extension changed from .ino to .c) as well as a line by line comparison of the SMTP messages for the working and non working versions.

[Ian.M]

Your '2024' version sends

Code: [Select]

<.> preceded and followed by blank lines at the end of the message data.  This is not a valid RFC2821 end of mail data indicator (end of message body), so of course it hangs till timeout - its expecting more lines of data (message body).

The SMTP response code 550 is "Requested action not taken: mailbox unavailable (e.g., mailbox not found, no access, or command rejected for policy reasons)".  The fact you received it indicates the mail data block (message body) was correctly terminated,  The 5.7.1 *should* indicate why the message was rejected, but it seems to be a 'portmanteau'  code used for 'Unable to Relay', and various detections of spam or unacceptable content.  Obviously mail server admins don't want to elaborate!  Its also used for excessive message length.

[Jester]

Your '2024' version sends

Code: [Select]

<.> preceded and followed by blank lines at the end of the message data.  This is not a valid RFC2821 end of mail data indicator (end of message body), so of course it hangs till timeout - its expecting more lines of data (message body).

The SMTP response code 550 is "Requested action not taken: mailbox unavailable (e.g., mailbox not found, no access, or command rejected for policy reasons)".  The fact you received it indicates the mail data block (message body) was correctly terminated,  The 5.7.1 *should* indicate why the message was rejected, but it seems to be a 'portmanteau'  code used for 'Unable to Relay', and various detections of spam or unacceptable content.  Obviously mail server admins don't want to elaborate!  Its also used for excessive message length.

I appreciate your help.........

I was using the following termination based on receiving  -> 354 End data with <CR><LF>.<CR><LF>  from server just after sending the body text of the email  [step 30]

Code: [Select]

  espClient.println(F(""));
  espClient.println(F("<.>"));                                                             
  espClient.println(F(""));       

If I instead use:

Code: [Select]

  espClient.println(F(""));
  espClient.println(F("."));
  espClient.println(F(""));

The server responds with:
550 5.7.1 Message Rejected (104) [D5B2E20010/1721861964-932527]
-> 500 5.5.2 Error: bad syntax

So I think your saying the second termination version is actually correct, even though it's giving a 500 5.5.2 Error: bad syntax message?

I'm still lost. How do I determine what is causing the 5.5.2 bad syntax error message?

[madires]

From https://datatracker.ietf.org/doc/html/rfc5321#section-4.1.1.4:

   The mail data are terminated by a line containing only a period, that
   is, the character sequence "<CRLF>.<CRLF>", where the first <CRLF> is
   actually the terminator of the previous line (see Section 4.5.2).
   This is the end of mail data indication.  The first <CRLF> of this
   terminating sequence is also the <CRLF> that ends the final line of
   the data (message text) or, if there was no mail data, ends the DATA
   command itself (the "no mail data" case does not conform to this
   specification since it would require that neither the trace header
   fields required by this specification nor the message header section
   required by RFC 5322 [4] be transmitted).  An extra <CRLF> MUST NOT
   be added, as that would cause an empty line to be added to the
   message.  The only exception to this rule would arise if the message
   body were passed to the originating SMTP-sender with a final "line"
   that did not end in <CRLF>; in that case, the originating SMTP system
   MUST either reject the message as invalid or add <CRLF> in order to
   have the receiving SMTP server recognize the "end of data" condition.

[Ian.M]

You are sending an extra blank line after the ".".  The .println() method includes a terminating EOL sequence, which on your system is presumably <CR><LF> or it wouldn't work at all!
The extra <CR><LF> is known to cause 500 5.5.2 Error: bad syntax.

[Jester]

You are sending an extra blank line after the ".".  The .println() method includes a terminating EOL sequence, which on your system is presumably <CR><LF> or it wouldn't work at all!
The extra <CR><LF> is known to cause 500 5.5.2 Error: bad syntax.

Okay, I commented out both the preceding and post <CR><LF>.  The previous body text is sent with a espClient.println(), so that should serve as the preceding

Code: [Select]

//    espClient.println(F("")); 
     espClient.println(F("."));                                                              // and this should be the final  <CR><LF> after the '.'
//    espClient.println(F(""));     

When I run this, I no longer get the bad syntax error, so I guess that's progress and I'm probably correctly terminating the email, now I get the following error:

550 5.7.1 Message Rejected (104) [2396E20026/1721933938-342398]

Any suggestions?

[rich t]

Have you tried just talking to the SMTP server interactively using telnet?  You should be able to send commands by hand and see what works.

rtestardi@rich-pc3:/mnt/c/Users/rtest$ telnet smtp.hostedemail.com 587
Trying 216.40.42.128...
Connected to mail.hostedemail.com.
Escape character is '^]'.
220 smtp.hostedemail.com ESMTP

[Jester]

Have you tried just talking to the SMTP server interactively using telnet?  You should be able to send commands by hand and see what works.

rtestardi@rich-pc3:/mnt/c/Users/rtest$ telnet smtp.hostedemail.com 587
Trying 216.40.42.128...
Connected to mail.hostedemail.com.
Escape character is '^]'.
220 smtp.hostedemail.com ESMTP

I actually have tried, the time allowed before it kicks me out is quite short, so unless I type very quickly and accurately I get kicked out. This combined with having to enter base64 coded information makes it all but impossible. Keep in mind that it's failing at the last step.

[WatchfulEye]

550 5.7.1 normally means message blocked for security or content policy reasons (e.g. Suspected spam).

Try using a normal mail client on the same account to send the same message. If that doesn't work contact your mail providers tech support.

[westfw]

In the 2023 version the text was terminated with a simple “.” On a new line, it appears that the 2024 version wants a <CR><LF>.<CR><LF>

The termination has always been "a dot on a line by itself", where "lines" are delimited by CRLF (both of them.  "NVT ASCII" per the RFCs.)
using Arduino or similar, do ESPCLient.print("\r\n.\r\n");

[shapirus]

What's in the SMTP server logs?

If you have no access to those, try setting up a local one for testing and try to reproduce the problem with it using the same client.

[Jester]

What's in the SMTP server logs?

If you have no access to those, try setting up a local one for testing and try to reproduce the problem with it using the same client.

How would I go about doing that (I'm more of a hardware vs. Software guy). I did install WireShark (first time usage) on my Win7 PC yesterday and it only catches a few packets from the Arduino board. I guess this makes sense that it can only see the PC's WiFi traffic  I think there may be a way to have WireShark use the PC's WiFi to snoop on all WiFi traffic?

In the capture below the sender fails early when using port 465 so less packets, and runs to the end with port 587 so more packets, so my assumption is that WireShark is only capturing the initial broadcast from the sender. If I could see the entire sequence, that might provide some insight.

[zilp]

What's in the SMTP server logs?

If you have no access to those, try setting up a local one for testing and try to reproduce the problem with it using the same client.

How would I go about doing that (I'm more of a hardware vs. Software guy).

First, get yourself a Linux (or other Unix) system. Then install a random MTA, like exim or postfix.

I mean, I guess there are MTAs for Windows, too, but it's not really the environment such systems usually live in, and if you want to get more into building internet connected stuff, it's probably valuable to understand such environments anyway.

I did install WireShark (first time usage) on my Win7 PC yesterday and it only catches a few packets from the Arduino board. I guess this makes sense that it can only see the PC's WiFi traffic  I think there may be a way to have WireShark use the PC's WiFi to snoop on all WiFi traffic?

In the capture below the sender fails early when using port 465 so less packets, and runs to the end with port 587 so more packets, so my assumption is that WireShark is only capturing the initial broadcast from the sender. If I could see the entire sequence, that might provide some insight.

What you are seeing in that screenshot there has nothing to do with the SMTP connection itself, that's just your device discovering its IPv4 address (DHCP) and the MAC address of the IPv4 default gateway (ARP).

But your analysis is correct that if the interface that you are sniffing on is not on the path between the device and the target, then you'll generally only see broadcast packets and no actual payload traffic between the two.

On Linux, I would just configure the machine as a router or switch that sits between the two sides so that I could sniff the traffic. I have no idea whether that's a thing that Windows can do. If you have a switch that can do port mirroring, that could also be a way to get a look at the traffic, but your standard unmanaged office switch won't be able to do that.

Another approach that might work in this particular case could be to set up a simple TCP proxy, i.e., a thing that listens on some TCP port, and then establishes a second connection to a configurable target for every incoming connection and forwards all data between the two. On Linux, xinetd would be a piece of software that could do that. You could also write a simple verison of that yourself, of course. Then, you could have your device connect to that proxy, and that proxy connect to the target server. Then you wouldn't see the exact packets between the two, but still the same payload, so, if it's a payload syntax problem, that might be sufficient to diagnose the problem.

[Jester]

From https://datatracker.ietf.org/doc/html/rfc5321#section-4.1.1.4:

   The mail data are terminated by a line containing only a period, that
   is, the character sequence "<CRLF>.<CRLF>", where the first <CRLF> is
   actually the terminator of the previous line (see Section 4.5.2).
   This is the end of mail data indication.  The first <CRLF> of this
   terminating sequence is also the <CRLF> that ends the final line of
   the data (message text) or, if there was no mail data, ends the DATA
   command itself (the "no mail data" case does not conform to this
   specification since it would require that neither the trace header
   fields required by this specification nor the message header section
   required by RFC 5322 [4] be transmitted).  An extra <CRLF> MUST NOT
   be added, as that would cause an empty line to be added to the
   message.  The only exception to this rule would arise if the message
   body were passed to the originating SMTP-sender with a final "line"
   that did not end in <CRLF>; in that case, the originating SMTP system
   MUST either reject the message as invalid or add <CRLF> in order to
   have the receiving SMTP server recognize the "end of data" condition.

Thanks and great explanation

In the 2023 version the text was terminated with a simple “.” On a new line, it appears that the 2024 version wants a <CR><LF>.<CR><LF>

The termination has always been "a dot on a line by itself", where "lines" are delimited by CRLF (both of them.  "NVT ASCII" per the RFCs.)
using Arduino or similar, do ESPCLient.print("\r\n.\r\n");

Thanks for that. I think the incorrect termination is now resolved. Ian.M brought to my attention that the version I tried with

Code: [Select]

espClient.println(F(".")); actually allowed the sender to progress to the 550 message so it appears that the entire email did make it to the email server.  When I used the incorrect termination

Code: [Select]

espClient.println(F("<.>")); it appeared to be progressing further (took longer to provide the error message), however the last error message in that case before timeout is 354 indicating it was just waiting for the termination and never received it.

Using

Code: [Select]

]ESPCLient.print("\r\n.\r\n")  yields the same result as

Code: [Select]

espClient.println(F(".")); , both of these appear to correctly convey end of message. 

550 5.7.1 normally means message blocked for security or content policy reasons (e.g. Suspected spam).

Try using a normal mail client on the same account to send the same message. If that doesn't work contact your mail providers tech support.

So I think I have at least progressed to what hopefully will be the last issue to resolve, why is the email provider not letting the email in?
10:01:13.055 -> 550 5.7.1 Message Rejected (104) [F3EC52000E/1721829661-832712]

I spoke with email provider tech. support and they seem a bit lost. they asked me to send the identical email from Thunderbird on my PC as well as via their webmail and both of those work.

If anyone has further suggestions I'm all ears.

Thanks for helping me sort out the termination issue.

[WatchfulEye]

Try using wireshark to capture exactly what thunderbird sends and then program the arduino to exactly duplicate it.

[agehall]

Don’t ever rely on println()-like functions to do network related stuff. Always use the non-ln() versions and add your own line termination. This way you don’t get bitten by extra <cr><lf> being inserted or the wrong (just <cr> or <lf>) being sent.

[shapirus]

Another approach that might work in this particular case could be to set up a simple TCP proxy, i.e., a thing that listens on some TCP port, and then establishes a second connection to a configurable target for every incoming connection and forwards all data between the two. On Linux, xinetd would be a piece of software that could do that. You could also write a simple verison of that yourself, of course.

xinetd has an unnecessary learning curve for this case.

I would advise simpleproxy instead: https://packages.debian.org/bookworm/simpleproxy

Its name represents exactly what it is.

[Nominal Animal]

No, the sending code does not look right.  The EHLO domain is not right, the code does not check for DATA response, and there should be an empty line between message headers and the message body.

If it has worked at some point, it was only because the server was lax and allowed slightly malformed and/or unsafe messages through.

The emailResp() function is completely bogus. 

There are two types of responses from the SMTP server:
    NNN DASH TEXTSTRING CR LF
    NNN SPACE TEXTSTRING CR LF
where NNN is the three decimal digit status code, TEXTSTRING consists of zero or more ASCII codes 9 or 32-126 inclusive, DASH is ASCII code 45, SPACE is ASCII code 32, CR is ASCII code 13, and LF is ASCII code 10.

For each response (with the exact same NNN), the final line will be of the latter (space) form.  If there is more than that final line, the preceding lines will use the former (dash) form (so you can simply ignore the dash form lines).  Each emailResp() call should receive exactly one response, consisting of zero or more dash form lines, and exactly one space form.  It should also return the decimal value of NNN from that last line.  (You can use negative codes and codes 0-99 for errors, so that 100 and above are SMPT response codes.)

If you are sending a message to recipient@example.com from account@domain, the client should:

• Connect to the SMTP server using the appropriate port, and wait for a response.  The response should be 220.
• Send "EHLO domain\r\n", and wait for a response.  The response should be 220.
• Send "AUTH LOGIN\r\n", and wait for a response.  The response should be 334.
• Send "username\r\n" (where username is Base64-encoded form of the username), and wait for a response.  The response should be 334.
• Send "password\r\n" (where password is Base64-encoded form of the password), and wait for a response.  The response should be 235.
• Send "MAIL FROM:<account@domain>\r\n", and wait for a response.  The response should be 250.
• Send "RCPT TO:<recipient@example.com>\r\n", and wait for a response.  The response should be 250.
• Send "DATA\r\n", and wait for a response.  The response should be 354.
• Send each header line, for example "Subject: This is the message subject\r\n".  (Every header line must end with "\r\n".)
• Send empty line between headers and message body, "\r\n".
• Send message body.  If the body is not empty, it must end with "\r\n".
• Send ".\r\n", and wait for a response.  The response should be 250.
• Send "QUIT\r\n", and wait for a response.  The response should be 221.

Also note that each line must not exceed 998 characters (plus CR LF), but limiting to 78 characters (plus CR LF) would be even better.  Servers will check domain against the list of domains the specified username is allowed to send, and will reject the MAIL FROM command if they do not match.  In the case where the server knows about domain users, it may even check that username and account match.  So, you cannot just use whatever domain in a EHLO greeting!

[NorthGuy]

I spoke with email provider tech. support and they seem a bit lost. they asked me to send the identical email from Thunderbird on my PC as well as via their webmail and both of those work.

This is most likely about the message body, not about the SMTP protocol. May be you're missing some fields, or your "from" address does not match the user being authenticated. Or your headers are malformed in some way. Or they may use spam filters. Their spam filters might be very complex, they definitely can tell if the mail is composed by a well-known mail client or not.

The error code they show most likely encodes the cause of failure or contain a reference to a log record, but nowadays you can't expect tech support personnel to be able to decipher that.

[westfw]

Using Code:

Code: [Select]

ESPCLient.print("\r\n.\r\n")yields the same result as

Code: [Select]

espClient.println(F("."));both of these appear to correctly convey end of message.

Lucky.
Some of those println() type functions will produce alternate line endings.
(actually, "\n" isn't very safe either, in "print", since it will sometimes be expanded to a "full eol as appropriate for the device" or something.  Using espClient.write() would be better.  (write() should send raw bytes.  print() sometimes does interpretation.))

[HwAoRrDk]

I agree with others, if the server is responding with "550 5.7.1 Message Rejected", then the SMTP transaction is completing, but the message is being rejected because the server doesn't like the recipient or contents.

The "5.7.1" part after the "550" is what's known as an 'Enhanced Status Code' (that the server says it supports by responding with "250-ENHANCEDSTATUSCODES" in response to "EHLO") that decodes into a more specific meaning. In this case, it refers to a permanent "delivery not authorized, message refused" status ('permanent' meaning not likely to be resolved by resending the message in the current form), meaning "the sender is not authorized to send to the destination". See RFC3463 for details.

It may mean the user you're logging in with isn't allowed to send to the given destination address (unlikely, given it appears you're sending to yourself), or there's something incorrect about the recipient address. But it could also mean that there's something about the message contents itself that is incorrect or missing that means the server can't figure out how to deliver it.

I note from the posted code you are only sending a single header in the message: "Subject". You aren't including the usual "From" and "To", nor others like "Date" or "Content-Type". I would advise trying to add "From" and "To" and see if that helps. Make sure they match the "MAIL FROM" and "RCPT TO" being given to the server. For example:

Code: [Select]

  espClient.println(F("DATA"));
  espClient.println(F("To: <info@*****.com>"));
  espClient.println(F("From: <info@*****.com>"));
  espClient.println(F("Content-Type: text/plain; charset=UTF-8"));
  espClient.println(F("Subject: 8266 sender message 2024"));
  espClient.println();
  espClient.println(F("8266 sender message 2024"));

Obviously, you can't include a "Date" header if you have no clock or calendar facility on the Arduino, so that can be omitted. And of course, as others have mentioned, you need a blank line between headers and body of the message. (BTW, if memory serves, I think println() can be used without arguments to emit just line break characters - no need to pass an empty string.) I won't go into the mess that is other content types and MIME message formatting.

[westfw]

it could also mean that there's something about the message contents itself that is incorrect

I'll point out that SMTP mailers (even ESMTP) have been getting more and more paranoid careful over time, due to the widespread forgery of email for nefarious purposes.  So "something incorrect" could mean "The From address in your message doesn't exactly match the "Mail From" command in the SMTP dialog, or "the hostname used in one of those addresses doesn't exactly match the name lookup of your IP address."

[zilp]

Code: [Select]

  espClient.println(F("DATA"));
  espClient.println(F("To: <info@*****.com>"));
  espClient.println(F("From: <info@*****.com>"));
  espClient.println(F("Content-Type: text/plain; charset=UTF-8"));
  espClient.println(F("Subject: 8266 sender message 2024"));
  espClient.println();
  espClient.println(F("8266 sender message 2024"));

Obviously, you can't include a "Date" header if you have no clock or calendar facility on the Arduino, so that can be omitted. And of course, as others have mentioned, you need a blank line between headers and body of the message. (BTW, if memory serves, I think println() can be used without arguments to emit just line break characters - no need to pass an empty string.) I won't go into the mess that is other content types and MIME message formatting.

Note that the Content-Type header in the syntax used here is a MIME-Header, and as such, if you use it, you also have to add a MIME-Version header:

https://datatracker.ietf.org/doc/html/rfc2045#section-4

Also, you have to include a Date header, it is mandatory in RFC5322:

https://datatracker.ietf.org/doc/html/rfc5322#section-3.6

[zilp]

So "something incorrect" could mean "The From address in your message doesn't exactly match the "Mail From" command in the SMTP dialog, or "the hostname used in one of those addresses doesn't exactly match the name lookup of your IP address."

The mail domain never has to match the host name of the server (or else, no email provider could operate), but also, and more relevant here, checks of host names or IP addresses only apply to server-to-server connections, not to submission connections, where the client authenticates to the outbound server of their own domain in order to submit an email for delivery, as it is expected that email client will connect from just about anywhere, and also, it would serve no purpose, as the SMTP AUTH credentials do already reliably establish the identity of the client.

For server-to-server, you might also want to setup SPF, DKIM, and DMARC, but none of that is relevant for submission connections.

[Jester]

I agree with others, if the server is responding with "550 5.7.1 Message Rejected", then the SMTP transaction is completing, but the message is being rejected because the server doesn't like the recipient or contents.

The "5.7.1" part after the "550" is what's known as an 'Enhanced Status Code' (that the server says it supports by responding with "250-ENHANCEDSTATUSCODES" in response to "EHLO") that decodes into a more specific meaning. In this case, it refers to a permanent "delivery not authorized, message refused" status ('permanent' meaning not likely to be resolved by resending the message in the current form), meaning "the sender is not authorized to send to the destination". See RFC3463 for details.

It may mean the user you're logging in with isn't allowed to send to the given destination address (unlikely, given it appears you're sending to yourself), or there's something incorrect about the recipient address. But it could also mean that there's something about the message contents itself that is incorrect or missing that means the server can't figure out how to deliver it.

I note from the posted code you are only sending a single header in the message: "Subject". You aren't including the usual "From" and "To", nor others like "Date" or "Content-Type". I would advise trying to add "From" and "To" and see if that helps. Make sure they match the "MAIL FROM" and "RCPT TO" being given to the server. For example:

Code: [Select]

  espClient.println(F("DATA"));
  espClient.println(F("To: <info@*****.com>"));
  espClient.println(F("From: <info@*****.com>"));
  espClient.println(F("Content-Type: text/plain; charset=UTF-8"));
  espClient.println(F("Subject: 8266 sender message 2024"));
  espClient.println();
  espClient.println(F("8266 sender message 2024"));

Obviously, you can't include a "Date" header if you have no clock or calendar facility on the Arduino, so that can be omitted. And of course, as others have mentioned, you need a blank line between headers and body of the message. (BTW, if memory serves, I think println() can be used without arguments to emit just line break characters - no need to pass an empty string.) I won't go into the mess that is other content types and MIME message formatting.

YES.... that was it


The missing to from was the problem.

Super Duper big thank you to everyone that helped out.