I've a simple question to Maple gurus.
Why DFU was selected as bootloader for initial of Maple ?
And not a simple serial protocol (via USB) so not need switch serial-DFU for download flash ?
Also serial protocol easy implements on STM32 without USB support.
And if it compatible with Atmel (stk500) no need special tools - avrdude w/o changes.
Bootloader general question
(15 posts) (4 voices)-
Posted 4 years ago #
-
gbulmer
ModeratorI can't comment for Leaf Labs, but I have a theory, based on my own STM32F thinking before I looked hard at the Maple.
1. I had believed that DFU was supported natively on *all* major Operating Systems (Windows, OS X, Linux), so it looked like there was no need for host-side drivers. That is a significant mistake, because DFU doesn't work properly natively. I don't use Windows usually, so I could have gone down this path.
2. The higher-family STM32F's, e,g STM32F105, have DFUse USB booting, so they should work without a JTAG. I didn't realise that STM32F's DFUse was not identical to DFU.
Once I understood these two errors, I came to the view that DFU was a "bad idea" (TM).
I agree with you that uploading must be an existing, implemented, supported, serial protocol, over USB.
My view was (and is) to use the same existing serial protocol that STM32F's have manufactured in for serial uploading, but implemented over USB.
There are already several Open Source STM32F serial loaders available.
This should allow a Maple to work with existing STM32F toolchains, and Open Source loaders.
AFAIK, avrdude is much more complicated than STM32F needs (a friend read the source, so this is 2nd hand, but I deeply trust his view). So I would NOT want to get connected into something much more complex than needed unless there is a big benefit.
Avrdude is a possible path, but there doesn't seem to be any compelling benefits in using it.
The Python loader (which isn't excellent, but just about works) is only a couple of hundred lines of python, several orders of magnitude simpler than avrdude. The C-based loaders seem okay, and may be more robust (but I haven't looked for a while).
I think that it would be much less effort to make the python loader work very well with the existing STM32F on-board loader than write an Atmel stk500 compatible loader for the STM32F. If I were doing it from scratch, I would probably write it in Java like the rest of the IDE, and use the existing C loaders as guidance.
The result would be something that could upload to any STM32F 'out of the box' using a USB to serial cable. An STM32F bootloader implementing the same protocol over a CDC USB serial device should require no other host-side changes than the name of the device to load to (a command line parameter).
Whereas writing a new STM32F loader which understood the Atmel stk500 protocol would still need to be loaded into a STM32F somehow - IMHO not as useful.
Have I missed your point?
Posted 4 years ago # -
poslathian
LeafLabsgbulmer, those are pretty accurate guesses. The original line of thinking was:
1) DFU is supported on all major OS's and seems to work (ended up not being true thanks to windows needing libusb to provide support)
2) ST already provided a DFU bootloader which we could adapt for our purposes (ended up not being true as ST provides dfuSE which is not DFU at all and is really just a huge lie on the part of ST marketing. We ended up writing an ACTUAL DFU from scratch from stm32.
3) We could run the bootloader and the virtual com port in parallel thanks to USB compound devices - this was true on maple rev1, but it turns out windows and mac and two mutually exclusive ways of dealing with compound devices and so we had to scrap this. This is really too bad because it was cool, you needed not to reset into "bootloader" mode or any junk like that. This was really the key reason for the original choice.
4) By using DFU rather than serial bootloader we could leave ourselves open to a non-serial usb devices for you sketches (like HID, audio/video, mass storage, etc).
5) Even when it became clear DFU was not really the right choice for the single MCU maple, a future design with an additional USB driver chip would still benefit from the DFU approach (like arduinoUno which uses DFU, and mBed which just uses a secondary usb driver chip)
As you can tell, most of these issues are clearly historical. Being no longer the case,
we have since designed and built a serial bootloader to eventually replace or supplement the DFU to solve much of portability issues. We started off essentially implementing a subset of stk500, but an awful lot of stk500 was AVR dependent and so we ended up branching some. This new bootloader is considered experimental, and still in development.Posted 4 years ago # -
gbulmer
Moderatorpslathian/mbolivar - why did you not use the existing ST Micro serial bootloader protocol?
If you had used that, then it looks like there are several benefits:
1. There are already several Open Source software bootloaders which run on host PC's which can upload Cortex-M3 binaries, so even if they are not ideal, those programs would allow a fast implementation and test
2. There is already an implementation of the ST Micro serial bootloader in every STM32F made, so by use of only a USB to USART interface (e.g. an FTDI cable) a 'raw' from manufacturing chip could be booted to be a Maple.
3. More importantly, a single tool chain could be used for initial booting and Maple development. IMHO, it is ALWAYS better to use one set of tools for everything to reduce the testing burden, and reduce support; when the users and the developers use different tool chains, there is significant scope for divergence and bugs.
4. An initial version should have taken very little time to test because most of both ends already exist.
5. It is likely that some other STM32F development environments would work transparently with the New Maple bootloader where they are expecting to do serial loading (I assume the OS hides enough of the difference between real USB to serial cables, and a New Maple bootlader). This might increase Maples market, and may also allow LeafLabs to leverage the efforts of other companies rather than spending effort re-inventing the wheel.
6. It might have shipped months ago in an experimental form that users could try out, knowing that there is very little, new and untried in the mix.Of course "Not Invented Here" may have put you off, but that is where business success and technical 'perfection' part ways - As Voltaire said "Le mieux est l'ennemi du bien." - "The best is the enemy of good."
Posted 4 years ago # -
Better way if ST make a disk interface (such a NXP) so absolutly not need a drivers. copy file to device and be happy. But ST makes a principal error (may be a china new year) so not implement this. and now tons of customer need resolve - how to download code w/o tears. now it's not intererst for me - i make simple serail BL and FOA over RF, GPRS and Ether. So good luck all who still have a problem.
Posted 4 years ago # -
mbolivar
LeafLabsgbulmer,
I'll respond to your points individually.
1. There are already several Open Source software bootloaders which run on host PC's which can upload Cortex-M3 binaries, so even if they are not ideal, those programs would allow a fast implementation and test
Yes, this is a nice perk; in fact, we use one of them internally when we're testing new bootloaders. It's part of libmaple, in support/stm32loader.py.
2. There is already an implementation of the ST Micro serial bootloader in every STM32F made, so by use of only a USB to USART interface (e.g. an FTDI cable) a 'raw' from manufacturing chip could be booted to be a Maple.
This is the sticking point. For us, it's a non-negotiable design requirement that the user be able to flash new code without any exra hardware (besides the USB cable). Additionally, using the baked-in bootloader requires manual intervention on the part of the user to set BOOT0 and BOOT1 correctly. While this is easy (just make sure BUT is held down when RESET is released), we don't want people to have to press buttons to reprogram their boards -- a single click should suffice.
3. More importantly, a single tool chain could be used for initial booting and Maple development. IMHO, it is ALWAYS better to use one set of tools for everything to reduce the testing burden, and reduce support; when the users and the developers use different tool chains, there is significant scope for divergence and bugs.
We totally agree. The present unfortunate situation (where the developers are using make, but the IDE manually compiles everything) exists only because bundling make for all three platforms (four, counting Linux 64) and getting it to play nice didn't work immediately when the IDE was initially being forked from the Arduino code, and there was time pressure to get it out the door. This unsatisfying situation persists only for historical reasons.
4. An initial version should have taken very little time to test because most of both ends already exist.
I've looked at the stm32loader.py source, and while I haven't tried it yet, I see no reasons why it wouldn't work on Jython (a Python virtual machine written in Java). The PySerial library, which it uses behind the scenes, has a functional Jython backend. Thus, it really doesn't make too much sense not to include a serial bootloader option in the IDE. I've added an issue in our bugtracker.
5. It is likely that some other STM32F development environments would work transparently with the New Maple bootloader where they are expecting to do serial loading (I assume the OS hides enough of the difference between real USB to serial cables, and a New Maple bootlader). This might increase Maples market, and may also allow LeafLabs to leverage the efforts of other companies rather than spending effort re-inventing the wheel.
6. It might have shipped months ago in an experimental form that users could try out, knowing that there is very little, new and untried in the mix.
There's nothing stopping anyone from using serial upload with the Maple and external tools. As stated, just hold down RESET and BUT, then release RESET before releasing BUT. The STM32 on the board is now in serial bootloader mode, and will accept uploads on USART1.
It seems that better documentation about this is warranted. I'll make sure to write some.
Of course "Not Invented Here" may have put you off, but that is where business success and technical 'perfection' part ways - As Voltaire said "Le mieux est l'ennemi du bien." - "The best is the enemy of good."
Indeed. Thank you as usual for your insightful comments.
Posted 4 years ago # -
gbulmer
Moderatorposlathian/mbolivar - sorry, I don't appear to have been clear enough. I'll try again.
You appear to be saying that you have implemented your own bootloader protocol:
We started off essentially implementing a subset of stk500, but an awful lot of stk500 was AVR dependent and so we ended up branching some. This new bootloader is considered experimental, and still in development.
I interpret this as saying the protocol used by the new (not yet available) LeafLabs bootloader (which works over a USB transport) is not compatible with the protocol implemented by the ST bootloader (which loads over a USART transport) which is already in all STM32F103 chips.
My previous question was not about Maple loading over serial instead of USB. My question is about the protocol layered over that transport. (I believe in communications using layered approaches, and the transport should be factored out as much as practical) Okay?
It is essential that a Maple user can load a program over USB. I am not suggesting otherwise.
Starting with the ST USART-based bootloader, let us take the transport details, like the baud rate, and use of RTS/CTS/DTR, and ignore them.
Let us call the remaining exchanges between the host and MCU the "STM32F103 Boot Protocol". This is a protocol designed by ST, and is implemented in the STM32F103, and in several Open Source programs, and by tool vendors for uploading, using serial/USART as a transport.
[Clearly the whole protocol works over a USB to serial/USART cable, we have both loaded to 'raw' STM32F's that way.]
My question is simply 'why did LeafLabs not implement the existing, documented "STM32F103 Boot Protocol" over USB in the new Maple bootloader?'
So the STM32F103 Boot Protocol protocol would be layered over a USB transport (preferably USB CDC device).I have listed the benefits of such a strategy in my question above. So far, I can only see benefits. I am trying to understand if there are any downsides.
I am assuming, to get an existing Open Source software to load to the new Maple USB serial bootloader over USB (not over a USART transport) the Maple would need to be put into USB bootload mode, and the host program run using the USB CDC device/file name. [Assuming it is anything like the FTDI serial, it might "just work" (TM)]
So, all I am asking is what is the reason that LeafLabs ...
... started off essentially implementing a subset of stk500, but an awful lot of stk500 was AVR dependent and so we ended up branching some.
Rather than use the existing, documented protocol that ST, the tool vendors, and the Open Source developers have already implemented?
(One final time; I am NOT asking anything whatsoever about using the USART as a transport, I am only asking about loading over a USB transport.)
As I said, I can't think of any reason why anyone would not start with the "STM32F103 Boot Protocol" and layer it over USB serial, so I'd really like to understand the reasoning. You may have discovered something key, which I have not discovered.
Posted 4 years ago # -
poslathian
LeafLabsGBulmer, sorry for the misunderstanding, I understand your question more clearly now. Regardless of the bootloader interface (USART,USB,Magick, whatever), Why did we choose to branch off of the STK500 command set for our experimental bootloader rather than branch of the command set provided by ST in this document (http://www.st.com/stonline/products/literature/an/13801.pdf), which is implemented embedded SRAM serial bootloader (that comes on every stm32, including maple, but only runs over USART). This is the command protocol implemented by the stm32loader.py script which is in libmaple/support (original author ivan@tuxotronic.org).
First let me clarify, the serial bootloader we made was and still is an experiment. From a functionality standpoint, the key issue we wanted to fix was that in windows Maple boots as a USB DFU device, then pulls down the DFU and pulls up a USB Virtual COM port. This is annoying for two reasons. First is that the installation procedure on windows requires pointing windows at two inf files. Second the USB device detaches and re-attaches on reset, which makes some annoying noises and generally confuses windows (pops up new device window sometimes etc. etc.).
That was the problem. Period. The potential solutions to the problem were:
1) compound USB devices - this was what we did in Maple Rev1, but incompatibilities between Mac and Windows meant this solution had to be thrown out
2) Writing a custom windows driver - this is not officially off the table, but its a high cost option that will also require us to get custom drivers signed by windows down the line. This is the route MBed took in order to simultaneously present a usb-mass-storage device for programming, and a serial port for communication. As far as I know they purchased this custom driver from a consultancy, something we could not afford at the time.
3) Present a single serial interface to the host, and multiplex both the bootloader and the communication interface onto it - this is the solution that arduino has adopted for both its UNO and original models. In the original models, this was made easy by a secondary FTDI chip providing the USB interface, in the UNO a secondary avr-USB chip provides the interface. Either way, they connected the main MCU via a USART port to either of these chips, and presented the host OS with a single point of communication (virtual COM). Because they have TWO chips onboard, handling the multiplexing is relatively easy, since you can keep one chip permanently booted (ftdi or avrusb) and still talk to it.We opted to EXPERIMENT with option 3, via the serial bootloader - but without adding an extra chip. The difficulty is, without adding a secondary chip provide the USB interface, we have no means of rebooting the MCU without also tearing down the USB interface (without having a fairly sophisticated bootloader like the original rev1 bootloader, which did this)
The reason I relay the story is to give you a sense of the difficulty of the problem. The decision had not been made to blast full speed ahead and build a serial bootloader. We were and are still exploring our options.
Besides solving the windows problem, we had two additional metrics we wanted to optimize. 1) how quickly we could implement the minimum experiment. 2) if we are going to build a new bootloader, how might we leave room for future integration with Oak (FPGA related functions) or perhaps embedded debugging in the IDE. We didnt want to build these features yet, but its in the back of our minds. and 3) how can we keep our new serial bootloader as similar to the AVR tools as possible (NOT the ST tools, which is a mostly closed source windows only ecosystem).
We looked at both STK500 and the STM32 SRAM embedded bootloader interfaces. Generally the command sets are roughly the same in terms of features. STK500 had a nicer packet structure, which made for easier parsing and easier implementation of the host side bootloading program (our version of an avrdude). There was no opportunity for any code reuse in either case since the ST bootloader is hardcoded on the chip with no source, and the ST loading utility is window closed source. the open source stm32loader.py adequately does its job, but we did not want to build off of that. Obviously the stk500 approach didnt offer reuse opportunities since its pretty AVR centric. Although, our initial approach attempted to upload programs directly with avrdude (this failed for various technical reasons).
So, given that we had little love for brunt of the ST ecosystem, found its protocol inferior to STK500, and saw that there was a chance that we could just use avrdude for uploading if we implemented things right, we built an experimental bootloader ontop of stk500 (same protocol, different commands). For technical reasons regarding a bug in pyserial and a flakey virtual com implementation on Maple (the latter has been fixed, the former is being investigated) our bootloader wasnt working right, and we switched gears into fixing up the virtual com implementation.
Now that this is done we have returned to the issue of bootloaders, and resumed the discussion.
I dont know if this answers your questions or if its the answers you wanted. I meant only to tell the story of whats going on. People get very excited about changes to the core codebase - like a bootloader - and by providing a thorough account of the situation I hoped only to inform you all and solicit feedback.
Sorry for the long post - this discussions timing was good as we returned to this issue this week, and ex-positing on the issue was valuable for me to get back into the subject.
Posted 4 years ago # -
gbulmer
ModeratorPoslathian
Regardless of the bootloader interface (USART,USB,Magick, whatever), Why did we choose to branch off of the STK500 command set for our experimental bootloader rather than branch of the command set provided by ST in this document ..?
Yes, that is exactly my question!
Thank you for the thorough, clear answer.
As you say, it might not be the answer I'd have wanted, but the rationale is clear.
Key differences between your criteria and mine are:
a) LeafLabs consideration for expansion, e.g. to support Oak,
b) I'd like to make it easy to bring up 'raw' STM32F's using a common toolchain (so reflashing to 'raw' would only need a USB to USART cable and a jumper setting if the loader used the same protocol as the built-in USART loader)
c) Your point about debugging is a very good one.
I had resigned myself to using JTAG for debugging, and hadn't given alternatives much thought.Thank you for taking the time to explain, GB-)
(Very interesting point about mbed, I had not realised it was bought in. That explains why it wasn't/isn't available)
Posted 4 years ago # -
gbulmer
Moderatorx893 -
... USB (and other) support can be place in bootloader section and not duplicate in user flash.
that was what I had hoped too.
Use a 'trap' (like a real operating system) to 'call' the USB code, then the user application wouldn't need to know the address of the I/O entry points.
Posted 4 years ago # -
May be not for this thread - but i work now on .NET for STM32 (not CBT but start from RET) and see more PCB on AT91SAM7X and LPC - it's incredible. Last brilliant as i see - top layer = VCC (3.3V) :), i resolve problem with serial bootloader (only need one maple serial driver in windows - w/o separate DFU driver. Protocol stk500 and i use avrdude to program. Flash as disk still in dev stage but not think that it complete anytime (not need for mee in a real). As for me i use only one time upload software and later update code over GPRS or RF link. With a STM32 only one not a helpfull - no way to define MCU iinside code (may be it's a new feature from ST :)).
Posted 4 years ago #
Reply
You must log in to post.