A proposal to improve Maple Firmware (Bootloader)

LeafLabs Garden » Community » Help Us Help You

A proposal to improve Maple Firmware (Bootloader)

(67 posts) (13 voices)

Started 5 years ago by gbulmer
Latest reply from feurig

« Previous 123…5 Next »

dreamcat4
Member

Are you certain?
I think todbots port opening program:...

Sorry gbulmer but you're absolutely mistaken. I wish it were the case but have experimented myself with this recently sometime in the last 1 or 2 months. The serial baud rate consifuration parity bits, etc is set by the OS level and API through which it can be accessed is typically passing flags or struct to ioctrl() / similar. After finding (that same example program you mentioned) did not work to change the DTR I searched all through the OSX BSD header files for any option that reffered (or possibly could refer) to DTR.

There are a few - I tried them in the same C program. They do not affect either way the sending of DTR signal. Its always sent. And elsewhere theres absolutely no evidence that this feature can be switched off on this platform. Perhaps you may be getting confused with Linux or other BSD OS which may be known to have a working control for switching off the DTR signal.

However your suggestion about using out of band USB signal (commonly known as USB io call) instead of serial DTR seems pretty reasonable. Isnt that where you specify an address (in the USB space) to have a certain meaning for the device? Then it seems the right way of addressing the issue. Good work.

As for whats the difference between hard and soft reset. There are 2 kinds of answers to your question:

1)
Hard reset is understood to mean an electrical triggering by changing the voltage level on the reset pin. Therefore its "hard". Wheras "soft" reset must mean a reset at the software level. Typically in the case of most MCU device a soft reset is the triggered by calling an ASM / machine code instruction from somewhere in the loaded program code. That isnt specific and can occur in any place from the running machine code. In the bootloader code, the program code, the OS kernel loop (scheduler) or watchdog timer. An OS environment may mask / remap the reset instruction to block it (if kernel privelidge insufficient). Or to intercept the call and allow OS cleanup actions to take place beforehand (for example spin down the hard disk).

2)
Behaviour is very device dependant. Generally speaking in a soft reset a device can assume many pre-requisites eg power condition (vdd voltage level etc) are already met. So its not necessary for the device to go through every single initialization checks or to worry about waiting for those external conditions stabilize. A soft reset may save other additional time by skipping certain other initialization steps which either arent yet initialized or could not be taken for granted in a hard reset. In some cases it might be assumed that the circuit to which the device is connected was already initialized. Which again may save other time if those connected components may take a long time to initialize. Some MCU devices may have multiple levels of "soft" reset. Which can be as simple as setting the program counter to address 0x00. Or as comprehensive as a full hard reset. Sometimes a soft reset can even be more comprehensive that a hard reset, including flushing (clearing) IO buffers, closing open connections gracefully, telling the running program to quit (fn callback). Waiting for timers to expire, etc. These aspect all depend very heavily on what the data sheet specifies, and the manufacturer should make as clear as possible.

Posted 4 years ago #
gbulmer
Moderator

dreamcat4 - the struct's go through the ioctl system call not ioctrl (I assume a typo, but it might mislead others)

ioctl acts on an open file descriptor, by which time it is too late, the DTR has been sent by the default device behaviour.

1) Hard reset is understood to mean an electrical triggering by changing the voltage level on the reset pin. Therefore its "hard".

I am glad I asked. That is not my definition. I think my definition is simpler:
A hard reset sets everything to an initial power-on state.
A soft reset does not.
Hence, by this definition, if you want a soft reset, you have to define what it does.

That is one way to trigger reset, but most processors are able to achieve the same behaviour or effect from software running in response to any signal, so this isn't about a difference in behaviour, it is just a different in cause. Clearly, a reset runs a piece of code, and in a process environment where there is no protection, anything could run the same code.

My definition is about what effect the hard or soft reset has, and is not simply what causes it.
Maybe an example will help. Turning a key in a car's ignition, hot wiring the ignition, or communicating with the engine management system might have exactly the same effect. But by your definition, two are hard resets, and one a soft reset.
Lets not muddle the stimulus with the response.

Wheras "soft" reset must mean a reset at the software level. ...

That is about the stimulus, not the behaviour.

.. Typically in the case of most MCU device a soft reset is the triggered by calling ...

I understand all of that and more.
(I used to teach Operating Systems to Computer Science undergraduates, and started programming at the bare metal over 30 years ago.)

Behaviour is very device dependant. ... These aspect all depend very heavily on what the data sheet specifies, and the manufacturer should make as clear as possible.

Yup, understand all of that too.

You introduced the idea that you wanted a 'hard' and 'soft' reset.

I want to understand what difference you want between a 'hard' and 'soft' reset.
So far, the requirements seem to be:
1. a reset using a button push
2. a reset in response to a default USB-transported signal.
3. it should be possible to prevent the USB transported signal from being automatically sent.
Is that it?
So far, I haven't understood any difference in the reset response, but only in how and when it is triggered.

Once you have defined the requirements, we can try to help.

If there is no difference between what you want from a hard reset, or a soft reset, other than one is triggered by an external 'button push' and the other is triggered by a set of bytes sent from the host over USB, then that is fine.

It sounds like the only problem that you are stymied on is preventing the Maple reset when opening its USB device on Mac OS X. Yes?

Would it be accurate to say your only requirement is to prevent the USB triggered reset?

That is an obstacle for you because you haven't found a way to either:
a. prevent a DTR being sent over USB automatically, or
b. prevent the Maple USB firmware from interpreting the DTR as a reset (it is always a soft reset if I have understood your definition).
If one (or both) of those were available, is that it?

(full disclosure: I am not a member of the LeafLabs staff)

Posted 4 years ago #
dreamcat4
Member

I hate to continue this. But acually (gbulmer) I think you've completely missed my point for trying to contribute to these discussions. Its pretty simple really: DTR is only defined as a message for passing information about flow control. Its not intended for these purpose. I've just been quietly trying to steer people here towards thinking about other options. Because what the Arduino people did was actually only done there due to limitations for that hardware. Their use of the DTR feature was a risk and therefore shouldnt be encouraged, copied. There's simply no arguing about this point. If you doubt or don't my opinion, just check the definition of "DTR" in the context of serial comms.

I have little interest in arguing over the other related points. Or specifically the details / way I have discovered this. Buy thats just my own general attitude. It doesnt mean they're not important for others to discuss here. You're also entitled to a different opinion about the value / merits of DTR - I personally don't.

Posted 4 years ago #
trevden
Member

dreamcat4, have you tried holding the port open with another process (a la http://unix.derkeiler.com/Newsgroups/comp.unix.bsd.freebsd.misc/2006-05/msg00475.html) or using the cu.* device?

In other news, the Hayes '302 patent has expired, so I propose the following in-band sequence to reset a Maple: <pause>+++<pause>ATZ<cr>

:)

Posted 4 years ago #
gbulmer
Moderator

dreamcat4 - I apologise. I missed your point.
I thought you wanted two different reset behaviours, and I was trying to understand what they were.

I agree DTR was used by the Arduino developers to force a 'hard reset'. As you say, that is an interpretation of DTR, and not its conventional serial comms. meaning, so using it for a different interpretation creates potential problems.

I'll try to state my view in a separate post.

Would you please share some of your experiments with Mac OS X and DTR.

In your experiments were the bits you investigated the DTR control bits in /usr/include/ttycom.h which start with TIOCM, and termios.h CDTR_IFLOW?
My comms guru, Nick, suggested doing an open with O_NDELAY, which should block until a DCD, which *might* allow a small window to get at the DTR, but equally it might not. Did you try that?
Any other lines of investigation which you have already exhausted?

trevden - openeing with another process - nifty! But naughty :-)
EDIT: I should explain my comment - it is a nifty idea, bit has two disadvantages: some of the existing Arduino software opens the USB port for exclusive use, and I want to be able to unplug from the host completely, and plug into a (different) host later.

I have never found a successful way to raise a defect report on Mac OS X short of paying to raise an incident.

Is anyone a registered Apple developer who could afford to raise an incident, if it comes to that?
I am willing to pay the $99 Apple developer membership (I let my lapse) to raise two incidents, if necessary.

(full disclosure: I am not a member of the LeafLabs staff)

Posted 4 years ago #
gbulmer
Moderator
I'll try to state my requirements for an improved Maple bootloader.

I want to use existing host-OS-supplied USB drivers to avoid a lot of work, and simplify installation for the user.
Almost all of the problems I have when I run Arduino workshops were caused by the need to install the FTDI drivers, and get them to work. I want to avoid having to install *any* device drivers, or run any host programs which can only be run by users with administrator privilege for day-to-day Maple use.

I think out-of-band signalling is an appropriate methodology for triggering non-data behaviour when a goal is to allow the 'remote' microcontroller end to be naive. I believe that is part of the goal of both Arduino and Maple. Arduino was constrained by its FTDI USB to serial interface, but Maple is not as constrained.

I think I can justify that view.

Out-of-band signalling allows the users Maple program to be largely ignorant of some of the communication mechanisms available. The users program can implement a minimum subset of communication, and still achieve useful behaviour. I think that is in the spirit of Arduino, and I feel it is appropriate for Maple.

I think it is a requirement to have a mechanism which can 'grab the attention' of the Maple MCU, and force it into a 'recoverable state'. For me, the 'recoverable state' could be ready to upload a new program, but it could be into a "monitor" which is able to continue the previously running program, upload a program, or a bunch of other useful stuff. Lets stick with the current behaviour where the recoverable state is a bootloader which will trash the previously running program, and either upload a new one, or restart the currently loaded program from the start.

I think the 'recoverable state' has to be reachable by out-of-band communication so that the users naive program can normally be recovered, and not get itself into a difficult to recover state by blindly consuming the in-band data. (Let me continue to call this 'recoverable state' to minimise misunderstanding.)

A user program like:
```
void loop() {
      char c = USBSerial.read();
}
```
Should not lock up the Maple MCU.

A further benefit of using out-of-band signalling to reach the 'recoverable state', is it ensures every in-band communication sequence is legal, so the host end of any communication can be naive too. A naive host program can send any data to Maple without having to inspect and deal with a magic in-band sequence of bytes. With the popularity of tools like Processing I think that host program simplicity is relevant and useful.

This doesn't mean the host program can be completely ignorant of the out-of-band mechanism if it needs to do something outside of the default.

IMHO it is also useful that a user can easily write a program which survives and continues to run during multiple USB connect/disconnect to the host, and multiple sessions of host programs opening and closing a logical connection to the Maple.

A simple example is a simple data logger. The use case is upload the user program, leave it to run to collect some data, open a USB connection and retrieve the data; maybe it runs and puts data into RAM. The user opens a host connection to microcontroller, and maybe 'chats' with their program. The same running program is able to run throughout those repeated open USB connection/close USB connection.

If one agrees with out-of-band signalling, then the simple data-logging case might be achieved on a Maple by either preventing the trigger to 'recoverable state' from being sent by the host, or by giving the Maple program an API which allows it to request the 'recoverable state' trigger to be ignored.

I would strongly prefer the first approach, preventing the host from sending the trigger to 'recoverable state'. That allows a Maple program to be simple, and it allows it to be more easily debugged because it always allows itself to be forced into a 'recoverable state'. This matters to me because I work with school children and non-programmers, and I would like them to be able to write a simple data-logger-style program with ease.

Summary: If existing out-of-band signalling can be made to work properly, there are several benefits:
1. No new device drivers for the host platform (huge win)
2. No in-band data to complicate naive Maple programs.
3. No in-band data to complicate naive host programs.

So far in my investigations, DTR is the only out-of-band signal which is available on all major platforms.
While using DTR as the out-of-band signal to trigger 'recoverable state' is an interpretation of DTR which is beyond the serial communication specifications, we are focused on use-cases built around Maple, so they are not abstract, they are concrete and constrained.

If there are alternatives, which have similar or greater benefits, let us get them on the table.
IMHO, if there is no way to prevent Mac OS X from sending a DTR, then I feel we need to consider alternatives anyway because otherwise we prevent a set of interesting use cases from being developed.

I can imagine using some device specific control information might be fine. I am concerned that using device control data, rather than a DTR, will involve writing, testing and maintaining host-side programs or libraries for every OS, and OS version.

(full disclosure: I am not a member of the LeafLabs staff)
Posted 4 years ago #
gbulmer
Moderator

After a bath, followed by a bacon, brie and tomato toasted sandwich (British bacon, not that streaky stuff) I've got a different proposal.

If we can't find an acceptable way to defeat Mac OS X automatic DTR sending, can we add a facility in the Maple firmware to ignore the DTR?

I am pretty relaxed about how to do this, but my thought is to use the 'But' button.

Proposal:
Maple's USB software ignores DTR when the user is holding down the 'But' button and not the Reset button.
I haven't walked through all the permutations of Reset and But button pressing, so it may have been used already. If not, this seems like a reasonable use.

The Maple user doesn't need to write any extra software.
Host software which doesn't send a DTR won't need to be changed to work (even if we do find a workaround for the Mac's auto DTR).

Comments?

(full disclosure: I am not a member of the LeafLabs staff)

Posted 4 years ago #
trevden
Member

Wait, I'm confused: Maple doesn't do anything on a DTR pulse by itself -- it only resets on DTR + "1EAF", so you can do your disconnectable data logger today without a problem. (I just tried it.)

Is the issue with user programs that kill USB connectivity leaving Maple in a state where it doesn't see the reset command?

And then manually resetting and getting the timing right to do the upload is too confusing for newbies?

(Whether to use DFU or CDC for the uploads is a separate issue, of course.)

(And as far as I can tell, if you can easily kill USB functionality unintentionally, that's a separate bug all its own.)

P.S.: A little birdie told me that USB IAD is implemented in Lion DP2.

Posted 4 years ago #
gbulmer
Moderator

trevden - the current Maple bootloader changes its USB device between a DFU and a USB serial.

Part one of my proposal is to get rid of the USB DFU device because that needs a driver to be installed on Windows.

Most UK schools run Windows in some form, but are unwilling or unable to install new drivers. In a common, and difficult to solve scenario, they use some form of terminal server, with 'thin clients' and only have a very few device drivers. To be precise, it is not yet clear that I can talk to a Maple using those serial devices, though I have been told by an Arduino user, and thin client system administrator that it should be possible, but it will take a lot of time to experiment with configurations. I am not going to start that until it is clear that there is a plausible USB-serial-only Maple.

The USB device changing between DFU and USB serial is also the cause of some problems when it signals the host OS to ask it to re-enumerate. Some OS's change the port, which is inconvenient, some take such a long time, that the users program has either got past 'setup' and any print output is lost, or their is quite a big hold-up, etc.

So, my goal is to have a USB-serial-only Maple. This will combine a USB serial bootloader with user serial I/O over USB serial, but the host OS will not see this difference. All the behaviour will need to be handled on the Maple.

For beginners, it is very useful to have the IDE force the controller into a 'recoverable state'. It makes the first few projects significantly easier than trying to time a reset-button-press correctly. IMHO, the first few projects need to be quite smooth to build confidence.

The bootloader, or the USB library, has to be able to tell the difference between an ordinary communication between the Maple and a host program, and a host program which wants to force it into a 'recoverable state' (the minimum being entering the bootloader ready to upload, but it could be more)

I would prefer to use an out-of-band signal so that simple Maple/host comms is easy to write (as I hope I set out above).

As I've said, my major priority is avoiding needing host side device drivers, because schools often won't or can't install them
I also would like to avoid developing new programs or libraries which have to be supported on all of the different host OS's (several versions of Linux, several versions of Max OS X, which are problematical over DTR, and many versions of Windows with SP's).

It is clear that there must be an upload program running on the host, so that needs to signal 'upload' to the Maple, but I don't think there needs to be any other software which needs to be implemented. Of course Leaflabs might make the source of that program upload available, or publish documentation about the 'recoverable state' mechanism, but I don't feel I should require them to do more.

I suspect no one has a problem in principle with my proposal so far.

Where dreamcat4 identified a problem is my proposal to use DTR for out-of-band signalling.
Unfortunately DTR is the only out-of-band signal that I have discovered which definitely works on all major platforms, and appeared not to require any new software to be developed for the host side.
dreamcat4's experiments indicate that there is no way to prevent it being sent on every open of the USB device attached to the Maple.

I am willing to sacrifice some convenience in order to avoid developing new host-side functionality, but I would prefer the inconvenience to impact the more experienced user rather than the beginner. Hence the suggestion that doing something, i.e. pressing But, will cause Maple to ignore the DTR.

If there is an alternative to DTR, please tell me, because I have failed to find it.

Does that help?

Posted 4 years ago #
trevden
Member
1)

I'm with you on the DFU issue; from a user standpoint, I agree that it would be easier to work with a device that reliably presents itself as only one well-supported thing. ;)

2)

Maple today uses DTR + "1EAF" to force the unit into a recoverable state. Yes, the force is not 100% absolute; you can write a user program that *intentionally* disables USB -- as you should be able to do.

That said, if a program like:
```
void loop() {
      char c = USBSerial.read();
}
```
is unintentionally making it unrecoverable, this is a separate bug somewhere, not an inherent flaw in the DTR + "1EAF" method.

3)

The DTR method is also used by Arduino, and is reliable across OSes and versions.

The observation that Mac OS X sometimes sends *extra* DTR pulses is not a problem for Maple; they are simply ignored unless immediately followed by the magic string.
Posted 4 years ago #
gbulmer
Moderator

trevden - this might seem a bit crazy, but I much prefer to avoid in-band signalling. I think in-band-signalling makes software more complex. I much prefer a control instruction like a reset to be out-of-band.

I am willing to accept that it is a misuse of DTR, and would accept any alternatives which work on most existing, deployed, platforms, and don't need any new host software.

One source of problems with in-band-signalling can come from implementing an existing protocol.
For example, lets pretend a connection is opened, hence a DTR is sent on Mac O X, the host sends a '1' (that is the existing protocol), and it waits for an acknowledgment. I think this just sticks.
Or, the stream of data may be purely binary, and very, very occasionally it happens to start with '1EAF'. That can be a nightmare to debug.

I don't know enough to be sure, but mixing in-band and out-of-band signalling might also be quite awkward to test and debug. For example, I don't know for sure whether the host USB device driver guarantees that bulk transfers (which might contain the '1AEF' bytes) and ioctl's, which presumably become USB control transfers, will have their temporal order guaranteed. Debugging that could drive one nuts.

Hence I like one type of signalling, and preferably out-of-band-signalling.

Yes, DTR is used by Arduino, and it is reliable across OSs and version.
Until this thread, I was unequivocally in favour of using it for Maple.
I had believed in my experiments I'd suppressed the DTR on file open on Mac OS X, but based on dreamcat4's assertion I am either mistaken, or that was only possible using the FTDI driver :-(

I still would prefer purely out-of-band signalling.
I'd willingly accept two DTRs in rapid succession (10's of milliseconds with no data in between) in preference to in-band-signalling for reset, if it is reliable.

Posted 4 years ago #
dreamcat4
Member

I had believed in my experiments I'd suppressed the DTR on file open on Mac OS X, but based on dreamcat4's assertion I am either mistaken, or that was only possible using the FTDI driver :-(

I used/use a stock Arduino UNO, which is 8u2 running LUFA firmware + DTR. Which autodetects on Mac OSX as /dev/tty.usbmodem3d11 "USB serial device". So unfortunately I can't go back to check if FTDI devices behave any differently. The official Arduino wiki suggests al the only sure-fire solution is to install 100uF capacitor (approx) between the Reset pin and GND. Which entirely prevents all resets whilst the modification is in place. The capacitor must be removed again for the board to be re-flashed with new firmware.

Can't comment as much on Maple, since I don't own a Maple yet. I agree here that 2 bytes magic number seems pretty crummy. It probably works well enough in practice most of the time. However something like an MD5 or SHA1 hash would also meet the uniqueness criteria. In this day an age, 16 bytes / 160bits doesnt hardly cost much and give a lot more confidence of the solution.

I must say I don't know very much about USB drivers yet either. But in this day and age find it kind of hard to dismiss that the right kind of out-of-band usb signal cant be sent without needing some specially-installed USB driver. Perhaps i'm mistaken about that. Still, it seems like the only other option we can investigate if DTR + magic number together isn't good enough.

Posted 4 years ago #
dreamcat4
Member

Heres where Maple now listens for DTR + magic number:

https://github.com/leaflabs/libmaple/blob/master/libmaple/usb/usb_callbacks.c#L78-85

when cmpMatch == magic number then ASM executes the fn ptr held in "target" variable

https://github.com/leaflabs/libmaple/blob/master/libmaple/usb/usb_callbacks.c#L108

which in then points to:

https://github.com/leaflabs/libmaple/blob/master/libmaple/usb/usb_hardware.c#L126

On the host-side there are 2 places where its written to send the DTR + magic number:

https://github.com/leaflabs/libmaple/blob/master/support/scripts/reset.py#L70-86

https://github.com/leaflabs/maple-ide/blob/master/app/src/processing/app/debug/DFUUploader.java#L151-170

It seems 1EAF was chosen to be the magic number because its the assigned USB vendor Id of the Maple device. Evidence:

https://github.com/leaflabs/maple-ide/blob/master/build/windows/dist/drivers/mapleDrv/serial/maple_serial.inf#L20

HOWEVER as gbulmer correctly points out here the choice of the magic number is entirely arbitrary. Because its bytes send on the overlying serial stream. It would at least make sense to increase the length of the magic number. Since using such a short (2-bytes) usb vendor id in the serial stream isn't particularly unique :)

So its possible be improved ^^ in those places. However then MapleIDE still must to keep sending the old 1EAF magic number for backwards compatibility on old boards. Perhaps one way to avoid any issue is not to change the magic number, but simply extend it length. So it would always start with 1EAF. Then continue with extra bytes which if not expected would be simply ignored on the current firmware boards.

The problems with other USB solutions are already discussed in the link below. The document is pretty accurate, and worth reading IMHO:

https://github.com/leaflabs/libmaple/blob/master/libmaple/usb/README

Posted 4 years ago #
snigelen
Member
I haven't followed the discussion in detail, but according the first link you give above:
```
/* magic number, {0x31, 0x45, 0x41, 0x46} is "1EAF" */
  char chkBuf[4];
  char cmpBuf[4] = {0x31, 0x45, 0x41, 0x46};
```
it's four bytes with the ascii-values of "1EAF", not two bytes hex like 0x1EAF.
Posted 4 years ago #
dreamcat4
Member

In that case 256^4 = 1 in 4,294,967,296. Which is a pretty unlikely sequence in combination with the way that Maple IDE specifically pulses the associated DTR line(s) beforehand. Compared to the Arduino - this combination is much more sophisticated. Overall I would like to take back my earlier statements in regards as to DTR being inadequate for Maple resets. Because its been implemented here on the Maple with such enhancements - it seems sufficient.

Posted 4 years ago #

RSS feed for this topic

« Previous 123…5 Next »

Reply »

You must log in to post.