The death of Xon/Xoff flow control...

[IanB]

Most terminal emulators are now able to "cut & paste" text directly into the window.  So it cannot be strictly called a file upload situation, but it is actually the equivalent of someone typing extremely fast into the terminal buffer.

Good point. I honestly didn't think of that scenario 

[AndyC_772]

The problem is not the board but the PC with the terminal emulator. The PC sends a large amount of text to the board. But the board can't keep up due to writing the text to flash.

I've come up against a similar issue when using a serial interface to download firmware updates to a target.

To address the issue, my preferred way to handle serial Rx on the target is to use DMA. All received characters go into a circular buffer in RAM, which happens entirely in hardware.

This means no characters are dropped, even if the CPU stalls while a Flash write is in progress. As soon as the write completes and the CPU resumes operating, the code can process whatever characters were written into the buffer in the meantime.

[voltsandjolts]

nctnico is writing to flash, presumably not enough RAM available to buffer, hence the bottleneck.

Maybe use 300 baud connection! 

[AndyC_772]

Writing to Flash isn't that slow; I'd be amazed if the serial interface is faster than the mean write throughput. The problem is usually that if the CPU is running code from Flash, it stalls completely while writes are taking place, which means it can't run a UART Rx interrupt or poll the small FIFO in the UART.

DMA, however, can still work even when the core is stalled.

[iMo]

After some experiments above (TeraTerm+2xFT232) I wonder how to actually use the xon/xoff for the text transfer (==plain text).

When I send a file as a text the 0Ah 0Dh are not transferred. It works when sending the file as "binary". But that would not work with xon/xoff in duplex mode, imho.

Btw., in above setup in simplex transfer I see C8h and 88h bytes (on o'scope) coming from the receiving part to sender, no idea what they are for..
  THose are the xon/xoff bytes in reverse bit order, it seems..

[Andy Chee]

Something out of left field:

You haven't told us your broader application.  Perhaps it's your coding approach that could be improved upon, in order to execute fast enough within the time it takes to receive a serial byte at 115200 baud, for 8,N,1 format would be about 87uS.

For example, are you attempting to parse a text file for a particular string pattern or character?

Have you tried slowing the baud rate down to see if the problem goes away?  If the problem remains with slower baud rates, then it highly suggests a coding issue.

[mikeselectricstuff]

Writing to Flash isn't that slow; I'd be amazed if the serial interface is faster than the mean write throughput.

If it's SPI flash, erase times are a killer.

[Nominal Animal]

Apologies for the long post, but it should be worth the read for anyone dealing with fully-featured OS and microcontroller communications using USB Serial (CDC-ACM).

USB Serial (aka USB CDC-ACM) supports a few useful PSTN requests that are supported by generic OS drivers:

• GET_LINE_CODING (0x20) and SET_LINE_CODING (0x21) transfer a zero value with a 7-byte data record with
  • 32-bit baud rate (bits per second), in little-endian byte order
  • 8-bit stop bit count (0=1 stop bit, 1=1.5 stop bits, 2=2 stop bits)
  • 8-bit parity type (0=none, 1=odd, 2=even, 3=mark, 4=space)
  • 8-bit data bit count (5, 6, 7, 8, or 16)
  , allowing e.g. Linux drivers/usb/class/cdc-acm.c driver to maintain the corresponding termios fields.  See include/uapi/linux/usb/cdc.h:struct usb_cdc_line_coding for a C definition of this structure.
• SET_CONTROL_LINE_STATE (0x22) sends a 16-bit value from host to device, with all but the two lowest bits zero.
  • Bit 1: This corresponds to RS-232 RTS (Ready-To-Send) state.
  • Bit 0: This corresponds to RS-232 DTR (Data Terminal Ready) state.
  The generic Linux CDC-ACM driver sends this message automatically with bit 0 set when the tty device is first opened, and cleared when last open descriptor closes, allowing the MCU to check if there is an application on the host computer accessing the tty device or not.
  (In Teensyduino, the USB (Serial) object evaluates to a boolean corresponding to the bit 0, and is thus true if the tty device is open on a Linux host.)
• SEND_BREAK (0x23) with value corresponding to termios tcsendbreak() with the duration specified in milliseconds.

Similarly, there is one useful notification:

• SERIAL_STATE (0x20), with zero value but a 16-bit little-endian data, with the lowest seven bits having useful information:
  • Bit 6, bOverRun: Set if device discarded data due to buffer overrun
  • Bit 5, bParity: Set if device detected a parity error
  • Bit 4, bFraming: Set if device detected a framing error
  • Bit 3, bRingSignal: State of ring signal detection on device
  • Bit 2, bBreak: State of break detection on device
  • Bit 1, bTxCarrier: Corresponds to RS-232 DSR (Data Set Ready)
  • Bit 0, bRxCarrier: Corresponds to RS-232 DCD (Data Carrier Detect)

The generic Linux CDC-ACM driver maps SERIAL_STATE to TIOCMSET/TIOCMBIC/TIOCMBIS ioctl()s on the open tty device, and maintains the termios structure for the tty device via SET_LINE_CODING; see drivers/usb/class/cdc-acm.c:acm_tty_ioctl() and acm_tty_set_termios().

As you can see, there is no CTS (Clear To Send) support at all!

This is because the USB device is supposed to NAK the bulk data packets when it is not ready to receive more data yet.  Thus, a suitable USB Serial converter should respect the handshaking protocol using the abovementioned USB properties.  When internal buffers are close to full, host USB data packets are NAK'ed, and the downstream device is told to stop sending more data (via XOFF if software flow contro, via CTS=0 if hardware flow control).  That is, all flow control should be implemented in the USB-UART bridge, because USB CDC-ACM just cannot do it fully (not without vendor extensions).

As it happens, Stefan Wagner (wagiminator, I like his work) has a WCH CH551/2/4-based OLED project (sources, hardware whose terminal-mode usb_cdc.c (sdcc, so C99/C11/C23 and not C++ code) looks straightforward to modify for a software-or-hardware flow control USB-UART converter.  This way, the software side treats the USB connection as a normal bulk bidirectional data stream without any flow control; the CDC-ACM driver manages the USB flow control; and the USB-UART converter handles the serial flow control.  The part count is ridiculously low and cheap.

The Silicon Labs AN758 application node contains most of the useful USB info needed, but note it specifically does not implement or describe implementing any flow control.

For USB CDC-ACM protocols, one should note that the native FS packet size is 8, 16, 32, or 64 bytes depending on the endpoint configuration, and 512 bytes for HS.  USB 2.0 is inherently half-duplex, and while you can easily reach about 950,000 bytes/second in one direction, requiring a response packet for each one will decimate the bandwidth.  XMODEM is unfortunate in that it uses 132-byte packets, which requires an extra FS packet.  A custom protocol would yield somewhat better bandwidth, but because of the data ack/nak response packets, would not reach anywhere near what USB-UART converter handling the flow control can (because no response data packets are needed for flow control).

[DiTBho]

ttys* still has auto-clean on open, so if you open /dev/ttys0 the driver issues a CTS pulse, which may reset the target

Useless and annoying...

[nctnico]

XMODEM is unfortunate in that it uses 132-byte packets, which requires an extra FS packet.  A custom protocol would yield somewhat better bandwidth, but because of the data ack/nak response packets, would not reach anywhere near what USB-UART converter handling the flow control can (because no response data packets are needed for flow control).

Thanks for the investigation. Fortunately, there is also Xmodem 1k which uses 1024 byte packets which is widely supported and needs (ballpark) 3 lines of extra code to implement.  All in all the throughput is better using 1024 byte packets.

A dedicated tool is also not ideal because somebody has to pay for developing it and realistically, a terminal emulator already has this functionality. In addition I want to be able to give this in the hands of people not so tech savvy and they may run whatever OS which then ends up in another software product which needs maintenance, support and so on.

[m k]

Nowadays input has at least three wheels rolling on their own.
First UART, its job is to change serial to parallel.
Then memory, to remember those parallel characters.
And then USB, that does the move on part.

UART and memory must be hard wired, but USB can't be.
So the memory must be so big that it can hold an USB glitch.
Or handshake must be controlled by the hard wired part.

Back in the day DTR and DSR pins were for information of operational state of a device, so steady style.
Actual handshaking was meant to go through RTS and CTS.

DTE is a Data Terminal Equipment, practically everything we use now.
DCE is a Data Carrier Equipment, like old style modem and very obsolete.

DTE -- DCE  DTE -- DTE
DTR > DTR  DTR > DSR
DSR < DSR  DSR < DTR
RTS > RTS   RTS > CTS
CTS < CTS   CTS < RTS

But PC changed that and practically all handshaking combinations are possible.

Terminal emulations.
VT100 is 7 bit machine.
VT102 is 8 bit machine.

(if memory serves)
101 is 102 without a printer port.
125 is 100 based graphical terminal.
132 is 102 based editing terminal.
131 is 132 without a printer port.

Remember also that DEC machines are ANSI terminals by nature.
(DEC was the source, ANSI made standards)
So tapping a key can produce quite many characters.

Old Esc[ is later replaced with CSI, ESC with 8th bit set.
Characters 0xC8 and 0x88 have no special data flow meanings in early DEC machines, so perhaps some TeraTerm internals.

Binary full duplex xon/xoff mode is a misunderstanding, xon/xoff is not binary communication anything.
When binary file is sent the duplex is limited, second file can't go to the other direction simultaneously.
It's the same with all binary communications, there smallest communication part is more than smallest part of communication.
In UART's case the smallest is a bit, it can't be blocked inside a character.
Single signal level is not a binary communication.

A side note.
DEC LA34 and LA120 console terminals, matrix printers with keyboard.
Printing process is iron pin to color ribbon to tractor paper on iron plate.
Early model is slow, but it doesn't really soften its sound.
Today hearing protection would most likely be mandatory pretty fast.

[iMo]

FYI- a decade back a group of people around the retrobsd project spent a lot of effort with an attempt to run ZMODEM on the pic32MX based 2.11bsd unix (for an easy transfer of sources and bins to the unix). There are several various source codes available for zmodem, we tried perhaps with all (we build for the unix), with all possible parameters (there are many). Also there are many zmodem apps for the PC side. The best working one I found at that time is below (provided somebody is experimenting with zmodem). But generally, it never was 100% reliable. We used USB_UART dongles at that time (with various pic32MX boards).

Code: [Select]

Executing C:\Users\User\Desktop\Utility\winkermit-3.0-test-21-nov-2013-vc6\k95.ini for WIN32...
?SET SPEED has no effect without prior SET LINE
Good Morning, smile!
Kermit 3.0.0 Dev.-, Nov 21 2013, for 32-bit Windows
 Copyright (C) 1985, 2013,
  Trustees of Columbia University in the City of New York.
Type ? or HELP for help.
[C:\Users\User\Desktop\] K-95>

[m k]

I tried to understand those cdc-acm.c and cdc.c files.

Line 290 of cdc-acm has acm_process_notification.
From line 326 it's
if (difference & USB_CDC_SERIAL_STATE_DSR) acm->iocount.dsr++;
if (difference & USB_CDC_SERIAL_STATE_DCD) acm->iocount.dcd++;

DCD is Data Carrier Detect, it's a modem signal indicating that phone line is live.
So this section is for DTE.

From line 346 start status bits for UART, so everything is checked, but CTS is missing.
The thing is also adding counters, clearly uncertain that it can catch all changes.
But it's a wake up, possibly not so high priority.

Line 505 has acm_read_bulk_callback
From line 326 it's
case -EPIPE: set_bit(EVENT_RX_STALL, &acm->flags); stalled = true;

That is much like a flow control situation of receive buffer filling.

Line 579 has acm_write_bulk
It doesn't have anything about stalling transmit.
But it's from different level, so transmit buffer check must be elsewhere.

Line 665 has acm_port_dtr_rts
It has a comment that overall it's not very operational.
Indicating that DTR or RTS is not supported way too regularly.
Combining this with RX_STALL one can think that input side flow control is not very steady thing.

Line 929 has acm_tty_tiocmget
There TIOCM_CTS is external compared to other status signals.
Next is acm_tty_tiocmset
There TIOCM_CTS is nowhere.
So DTR and RTS out and DSR in seems to be supported and CTS not.

Line 998 has wait_serial_change
There DSR, RI and CD are checked.
All are sort of higher level modem signals, so CTS can be elsewhere.

Line 1096 has a speed check.
If speed is rational DTR is set.
Since RTS is missing one can think that DTR has a higher level, and lower level RTS is for flow control, if available.

cdc.c defines CTRL_DTR and CTRL_RTS first and then SERIAL_STATE_DSR among others.

8250 UART has
Modem Control Register bits 0 and 1 for DTR and RTS, and
Modem Status Register bits 5 and 4 for DSR and CTS.
It was also a standard for decades.

No idea when or why CTS has changed its availability.

So it's pretty clear that RTS to CTS cabling may have some problems.
Alternative is RTS to DSR, if DTR to DSR is not accepted.
Finally there are no difference between different status pins.
From UART's point of view they are just its internal register bit sources.
Its operational bits are elsewhere.

One other thing, a practical certainty, at least used to be.
One must be lucky if 8 data + any parity will work.

E,
Maybe flow control part wasn't clear.

One pin is sending and one receiving.
If you can control both ends it doesn't matter what is what.

Normally DTR and DSR are a pair, like are RTS and CTS.
But nothing prevents mixing them.
So one end can have DTR/DSR flow control when other has RTS/CTS.
It's just transmitter pin to receiver pin.

One disconnecting problem there was, but can't remember exactly what.
Early PC and its application software stuff anyway.
Maybe RTS/CTS flow control and DTR looped back to DSR and CD.
Then DTR blinks for what ever reason and used program disconnects because CD is lost.