http://forums.leaflabs.com/topic.php?id=615 A place to share, learn, and grow... en-US Fri, 22 Jan 2016 00:05:26 +0000 http://bbpress.org/?v=1.0.2 q http://forums.leaflabs.com/search.php http://forums.leaflabs.com/topic.php?id=615&page=2#post-3482 Thu, 27 Jan 2011 23:09:54 +0000 crenn 3482@http://forums.leaflabs.com/ <p>I guess this isn't something that's needed. </p> http://forums.leaflabs.com/topic.php?id=615&page=2#post-3478 Thu, 27 Jan 2011 18:02:15 +0000 poslathian 3478@http://forums.leaflabs.com/ <p>x893 im with you here...</p> <p>incidentally forums user sinneb recently got chibiOS working on the Maple hardware, so if you need it.... </p> http://forums.leaflabs.com/topic.php?id=615&page=2#post-3448 Mon, 24 Jan 2011 18:30:43 +0000 gbulmer 3448@http://forums.leaflabs.com/ <p>MessiahAndrw - Yes, one could write a byte code interpreter. Yes, it could divide RAM into fixed size 2K pages.<br /> That would be a slow simulation of virtual addressing. Access to a 2KB segment of SD is likely to be, approximately, 300,000x slower than access to RAM, I guess, if it were via SPI. SPI would take about 4ms to read or write 2K at 4Mbit/second</p> <p>I'd recommend avoiding swapping variables to SD.<br /> If a page contains any variable that has been changed, the page must be written out to SD if the interpreter wants to replace the variables page with some other page.<br /> Code never changes, so it can be discarded and read in again when it is needed.<br /> So replacing a code page is 2x faster than replacing a variables page.<br /> With a 2K page size, it would only do 250 2K 'page replacements' /second.<br /> If variables had to be paged out, that rate would be halved to 128 'page replacements'/second.</p> <p>I'd strongly recommend a much smaller page size.<br /> I have not looked hard at SD, but normal flash memories support 128 byte pages, which would give 16x more page replacements/second.</p> <p>The mechanism I've suggested avoids the need to write a byte code interpreter.</p> <p>Just use the existing C/C++ compiler (or any other language compiler which supports shared libraries/dynamic libraries).<br /> By having the compiler generate position independent code (a command line flag), it can be loaded into RAM or Flash (on the Maples ARM Cortex-M3). It isn't constrained to page boundaries, just instruction boundaries (4 bytes boundaries are okay)</p> <p>The OP could put a chunk of code, preferably stuff that is needed across the entire run of the program, into flash and call it as ordinary functions, because that's all it needs to be. </p> http://forums.leaflabs.com/topic.php?id=615&page=2#post-3444 Mon, 24 Jan 2011 05:16:05 +0000 MessiahAndrw 3444@http://forums.leaflabs.com/ <p>Well you could use a byte code interpreter, and divide your RAM up into say 10x 2KB pages that can store bytecode and variables. When your bytecode interpreter jumps to or references an address not in memory you can dump the least use page (to disk if it's marked as dirty) and replace it with the page you're referencing.</p> <p>Routines that need to run fast (or are used in many programs) could be loaded into flash memory as a runtime extension, and each extension would register a custom opcode into the interpreter. </p> http://forums.leaflabs.com/topic.php?id=615&page=2#post-3428 Fri, 21 Jan 2011 14:38:24 +0000 x893 3428@http://forums.leaflabs.com/ <p>Yes - i agree. RT-11 with more sophisticated software run on 56K memory. And RSX-11 with 256K was a high top.Dynamic manager was a very simple and this method can be implemented easy. But in a real i can't imagine apllication on STM32 which need more than 512K flash. Also can use CircleOS with full support SD, file operation and software slot mechanism. </p> http://forums.leaflabs.com/topic.php?id=615&page=2#post-3426 Fri, 21 Jan 2011 13:34:32 +0000 gbulmer 3426@http://forums.leaflabs.com/ <p>x893 - I think if you look back in the history of computing, you will find techniques for dynamic loading of code at every stage of computing. </p> <p>Prior to virtual memory, it was usually visible in the programming language. For example FORTRAN COMMON allows for overlaying data, and I vaguely remember mechanisms in a proprietary version of FORTRAN (1980's) to load a code module. A 'mini-computer' system I programmed in the late 70's (bought as Singer Computers by ICL) allowed programs to chain programs and keep data resident. Most Basic's did, but they were interpreted, so it's significantly easier to do the fixup. I wrote a Bill-Of-Material system in the early 80's on a Commodore Business Computer (a PET in a nice case, with floppy disks), which had 32K RAM 'fixed', and 32K window for overlays (shared with the peripherals, so file reads and writes were a bit weird). I could write statements to pull in 'overlays'.</p> <p>I used RSX-11 on a normal PDP-11, and I honestly can't remember how dynamic loading worked. Most things were set up for assembler though, so I can imagine it would be pretty easy with some assembler macro's.</p> <p>My proposition is to make it 'invisible' to the running program by hijacking the normal shared library mechanism and use it in a slightly bizarre. AFAIK, the DEC PDP-11-era software (including RT-11) didn't try to do that.</p> <p>The OP's question is to load into RAM, and run in their. A multi-user, multi-tasking Edition 7 UNIX ran fine in 256K (supported 5+ users), so that is my mental model :-) </p> http://forums.leaflabs.com/topic.php?id=615&page=2#post-3425 Fri, 21 Jan 2011 08:34:47 +0000 x893 3425@http://forums.leaflabs.com/ <p>I've been laughing reading this thread. Around 1983, I happened to run into dynamic library in RT-11 for LSI-11/20. Nothing has changed since then except for memory size and processor speed. If someone does not have enough 1MB flash memory - let him spend a week of their lives for the implementation of the manager and no brains hovers around the globe. </p> http://forums.leaflabs.com/topic.php?id=615&page=2#post-3424 Fri, 21 Jan 2011 08:18:44 +0000 gbulmer 3424@http://forums.leaflabs.com/ <p>The paper which explains how Dynamically Loaded Libraries work (for Linux) is called:<br /> "How To Write Shared Libraries"<br /> by Ulrich Drepper<br /> and can be retrieved from <a href="http://www.akkadia.org/drepper/dsohowto.pdf" rel="nofollow">http://www.akkadia.org/drepper/dsohowto.pdf</a></p> <p>The edition there is dated December 16, 2010<br /> The one I read was 2006, so I don't know what else has been added, but I thought it covered most of the mechanisms. It has a lot about optimisation for Linux, and the security problems caused by shared libraries/DLL's, which aren't needed but are interesting. I think the 'meat' of the paper, for this purpose, is in the first 8 pages.</p> <p>The mechanism I am suggesting is to userp the indirect jump table to trigger code load and eviction, by substituting your own 'fix up' 'thunk'. </p> http://forums.leaflabs.com/topic.php?id=615&page=2#post-3417 Thu, 20 Jan 2011 15:00:59 +0000 gbulmer 3417@http://forums.leaflabs.com/ <p>Poslathian</p> <blockquote><p>As Bill Gates fictionally said - no one would ever need more than 640K of memory anyway</p></blockquote> <p>Maybe I am cynical, but I can't imagine Gates owning up to that one after, about 1984. Further, if the reporter was dead before 1997, and it was an old audio piece, it was probably a pretty safe position to take. Once he stepped down from MS, he probably got a bit more relaxed, but it's a bit late to change history again.</p> <p>AFAIK, it was only recently that Gates himself admitted he'd underestimated the impact and expansion rate of the Web, and was forced to rush out and buy Spry to get a web browser (became IE) because the need became so urgent. </p> <p>I watched a TV 'documentary' (in 2009?), where he seemed quite happy to dissemble; though, to be fair, the TV producers can selectively edit interviews, and even Gates might ignore inaccuracies which make him look smarter.</p> <p>I am more suspicious of its accuracy, because it is so like the quote by IBM's Thomas Watson Jnr about the number of computers needed by the world. </p> http://forums.leaflabs.com/topic.php?id=615#post-3416 Thu, 20 Jan 2011 14:57:52 +0000 poslathian 3416@http://forums.leaflabs.com/ <p>Yea, I think I've were doing something this sophisticated Id want to move away from C and get some sort of runtime environment - like python,lisp,lua or whatever. </p> http://forums.leaflabs.com/topic.php?id=615#post-3414 Thu, 20 Jan 2011 14:51:26 +0000 gbulmer 3414@http://forums.leaflabs.com/ <p>poslathian - I strongly agree that one would not want to do this if one could avoid it.<br /> As I wrote, I believe solving resource constraint problems get asymptoticaly hard.</p> <p>BUT, it is an interesting question!<br /> AND if all of the machinery for dynamically loaded and linked libraries can be used, incluing generating Position Independent Code, it might not be too hard (hard, but not too hard :-)</p> <p>I have thought of a solution to evicting code safely. So I'm confident I could design a solution, burning CPU time for space. One could even develop a protocol and host side server to do this over USB, rather than SD (if you really wanted to do something weird :-)</p> <p>If you have ever used FORTRAN, and COMMON, you will understand ways to overlay variables in the same memory. Translate that into C union's, yeuk :-)</p> <p>I would not want to do this for a living :-) </p> http://forums.leaflabs.com/topic.php?id=615#post-3412 Thu, 20 Jan 2011 14:34:16 +0000 poslathian 3412@http://forums.leaflabs.com/ <p>oh boy - I suppose Ill just wait around on this thread until you guys implement an entire operating system...</p> <p>storing programs that are larger than the available onboard flash/ram in and SD card and then loading needed code snippets and subroutines in on the fly sounds - well it sounds like you should avoid it if you can ;). What you would really need to get that to work in a native way is hardware support for virtual memory (an MMU) which doesnt exist on cortex-m3 (hence no real linux support except some of the MMU-free dists). Of course, thats only if you want to just write code and compile it whilst staying agnostic of the fact that programs arnt necessarily all loaded into memory at the same time.</p> <p>You could achieve this effect on your own by writing sketches that are explicitly aware of the fact that they need to grab routines from SD flash and cache them in local memory. Some added constraints would make your life a lot easier, for example if dynamically loaded routines included no static or global variables, and never called any other dynamically loaded routines (they only depend on core of your program that you keep in memory all the time). </p> <p>Maple Native and Maple Pro (stm32f103ret6) will be here soon enough with 512KB of program flash and quite a bit of RAM on the former, so if hopefully there isnt much of a reason one would NEED to this - except *massive* amounts of street cred...</p> <p>As Bill Gates fictionally said - no one would ever need more than 640K of memory anyway... (<a href="http://www.wired.com/politics/law/news/1997/01/1484" rel="nofollow">http://www.wired.com/politics/law/news/1997/01/1484</a>) </p> http://forums.leaflabs.com/topic.php?id=615#post-3410 Thu, 20 Jan 2011 14:01:13 +0000 gbulmer 3410@http://forums.leaflabs.com/ <p>crenn - I have a paper describing these things in quite a lot of detail, somewhere.<br /> Let me check around to find it. If I don't give you an answer by Monday, please bump this topic.</p> <p>Essentials:<br /> Every piece of binary, linkable, code has (conceptually) two symbols tables:<br /> a. all of the symbols (i.e. function names and global variables) defined in this file of code, and the offset of every global symbol within this file<br /> b. all of the symbols (i.e. functions or global variables) this file of binary wants to use.</p> <p>You can statically link (once and forever) all the files of binary code into a single executable binary, which has all the symbol users connected to all of the symbol definers. That is pretty much what normally happens with an embedded system.</p> <p>BUT, on normal operating systems, the pieces of code can be loaded dynamically as needed, at run time. This was primarily invented as a technique to share libraries between several programs on a multi-tasking or multi-user operating system, to save memory space, and reduce the size of programs on disk. (Google for "Rob Pike" and "shared libraries" for his opinion. Quite interesting reading).</p> <p>Anyway, the paper explains how this all happens on Linux, amd hence explains mechanisms which you can get at.<br /> I think once you understand how shared libraries work, you will probably see how the same technology could be made to do what you were asking about.</p> <p>So, I will have a dig around. I think it came from ACM, so it shouldn't be too hard to find.</p> <p>(BTW - I have a technique,. and a few thoughts, for evicting functions safely) </p> http://forums.leaflabs.com/topic.php?id=615#post-3408 Thu, 20 Jan 2011 09:43:56 +0000 crenn 3408@http://forums.leaflabs.com/ <p>Would you believe I've never ever used DLLs? Would you have a handy link to a tutorial?</p> <p>I don't doubt this isn't going to be a hard task, and could make me bald! But it also means learning new things (including DLLs, which isn't going to be a bad thing since my project could use some custom software to display the current data from the UAV). </p> http://forums.leaflabs.com/topic.php?id=615#post-3407 Thu, 20 Jan 2011 09:23:50 +0000 gbulmer 3407@http://forums.leaflabs.com/ <p>cren - if your aim is to make it near-invisible, you might experiment with dynamicaly linked libraries (DLL's).</p> <p>If every independent piece of code (all the way down to a function) is in a seperate library, then calls from one function to another go indirectly, via a 'jump table'.</p> <p>Now, for every function which is not in memory, you can have the entry in every corresponding 'slot' in the jump table point at your code. Your special jump-table function will get called when a function calls another, but the target function isn't loaded. Your code loads the missing function, and fixes the jump-table accordingly. If your code has to evict a function, it wires itself into the call slot for the evicted function. A trick here is to know that a function is not in the middle of a call, and going to be called when the stack unwinds.</p> <p>The basic mechanism exists in the compiler, linker and dynamic loaders, so you'd have a pretty slick, near invisible technique to handle program code loading. </p> <p>But be aware stack unwind isn't an issue for the existing tools and technology, so you'd need to come up with an approach. You could just search down the stack for a number which looks like the start address of the function you've chosen to evict, and if it isn't there, go ahead, and throw it away. The stack is word aligned (or 8-byte aligned) so looking isn't too arduous. There is a small chance that you keep a function unnecessarily because there is a number on the stack which corresponds to the functions address (4,294,967,296 to one?-) but that is safe.</p> <p>This won't be as easy as I might have made it sound, but it could be interesting.</p> <p>EDIT: I made a significant mistake.<br /> The return address on the stack points to various points within the function (which is a candidate for eviction), not just at its start. So a search for a return address on the stack would look for a range of values, and not a single value. This makes the odds of a mistake higher, and is a slower search. </p>