Processing of Maple data using a DLL

LeafLabs Garden » Support » Maple IDE Support

Processing of Maple data using a DLL

(13 posts) (4 voices)

Started 5 years ago by StephenFromNYC
Latest reply from gbulmer

StephenFromNYC
Moderator

Thursday September 30, 2010

Hello,

Does anyone have experience with using DLL to process data generated with a Maple?

This might be accomplished using Processing or another app as a "bridge" (assuming the DLL is running on the system processor, and not the microcontroller).

I am just curious, because DLLs were mentioned during the Open Hardware Summit last week (I believe you can still view the "legal" session moderated by David Mellis). I do not have an immediate need for accessing DLLs, but I may have one soon.

Thanks!

Stephen from NYC

Posted 5 years ago #
gbulmer
Moderator

Stephen - By DLL do you mean a "Dynamic-link library" or something else?

(I probably can't answer your question for procssing. though I am curious, but I'm not sure if you mean 'Dynamic-link library', so I thought I should seek clarification).

There are a bunch of technologies for getting at "Dynamic-link libraries", which platform/OS do you need to use?

Posted 5 years ago #
StephenFromNYC
Moderator

Hello,

Yes, by DLL I mean "dynamic link library".

I am glad to hear there are several methods for accessing DLL code from the Maple/Arduino.

Accessing DLLs might be another "stress test" for the Maple.

Primarily Windows, but possibly Mac OS. A few years ago I was dual booting Windows/Linux and perhaps I should do it again. Linux would be a far third.

I am most familiar with DLLs on the windows platform. Can DLLs created on Windows be used on other platforms?

Thanks! Looking forward to your answers.

Stephen from NYC

Posted 5 years ago #
perry
Administrator

Hi stephen,

I'm not really sure what you mean by using a DLL with Maple. DLLs are shared libraries which are loaded by programs at run-time, hence the "dynamic" moniker. See http://en.wikipedia.org/wiki/Library_(computing)#Dynamic_linking

It's not clear why you would want a dynamically loaded library on the maple. You'd have to have some way of dynamically loading the code over some communication interface, placing it in RAM or flash, and then run a bunch of dynamic loader code to parse the binaries, fix up relocations, and then jump to the code.

It would help if you told us what you wanted this for. I could see this being useful if for some reason you wanted to update the code on a maple at runtime while it was still running and performing other functions, but otherwise I don't see much use for it.

Posted 5 years ago #
StephenFromNYC
Moderator

Thursday September 30, 2010

Hello,

Let me try to flesh out my idea some more. As I said in my first post I do not have an immediate need. I am just looking for more ways to benchmark and debug the Maple software.

Let me clear about what I am NOT proposing. I was NOT hoping the DLL could run on the Maple (too complicated and not enough memory). It is not clear how a DLL written for an X86 processor could run easily on a RISC processor such as the STM32. I was NOT hoping to reprogram, update, or patch a sketch running on the Maple (but that would be an interesting trick if it could be done).

When I named the initial post "Processing of Maple data using a DLL" I was focusing on the potential ability of the Maple to acquire DATA (via ADC) faster (ie. more samples/sec) and at a greater resolution (ie. 12-bit vs 10-bit) than the Arduino.

So I was imagining this sequence of events:

1) the Maple acquires data by reading analog pins (at a high rate & 12-bit resolution)
2) the Maple sends the analog data to the system (via some method). I will be honest and say I still do not fully appreciate the full abilities/roles and limitations of DMA, I2C, SPI, and the serial communications options available to the Maple/Arduino. The analog data might be stored temporarily in the Maple SRAM or FLASH if the data can be more sent more efficiently (eg faster) in large bundles to the system (instead of sending each individual analog read one at a time; which is what I am doing now with my LabVIEW/Arduino code).
3) a separate program running on the system (eg. Processing) simply reads the data sent from the Maple and relays the data to a DLL (eg. a graphing program). This program might reformat the data a little, but not much is done.
4) the DLL running on the system processes or interprets the data (eg. graphing the data and saving the interpreted results in a file)

So the Maple is just acquiring the data and sending it somehow to the system.

Assuming this can be done already with the slower Arduino, the "added value" is knowing if the program running on the system (list #3) can read the data at the faster rate of the Maple and if the DLL (list #4) can also keep up.

Of course, in the ideal world the original code for the DLL would be available and could be easily modified to interpret the increased 12-bit range (0-4095) of data values. Perhaps the person who wrote the DLL thought ahead and created a flexible method for informing the DLL of the resolution of the incoming data (eg. 10-bit vs 12-bit).

Where does an FPGA enter this conversation? How is an FPGA used? I know one will be on the Oak, but I have never used an FPGA before.

I hope this makes more sense.

Thanks!

Stephen from NYC

Posted 5 years ago #
josheeg
Member

Well Xilinx as far as I know is the only place with linux closed source software for their FPGA's and their software you can make schematics of digital logic or code digital logic if you needed something that is like a lot of that.

It can be modules like a adder is put into a module and used so it is expandable like that.

That is a good way to share thowse on something like google sites file cabinet with versioning.

Also schematics and digital symbols are easier than the fpga code ...
I switched from using a fpga to a maple and ft2232h for high speed usb to pc logging. the maple controls the transfer but the data goes streight to the ft2232h

Posted 5 years ago #
perry
Administrator

hey stephen,

I think you're misunderstanding the term DLL. A dynamically linked library is just a piece of code that can be shared with other programs running on a system. This is a useful idea when many programs need to share the same bit of code (eg. a drawing library).

For the use case that you listed, the external bit of code that operates on the captured maple data doesn't need to be a dll at all. You can read the data from the maple with any program that can read a serial port, and operate on that data straight in that program. Alternatively, you can use other forms of inter-process communication (IPC) such as sockets (for communication over the internet), or shared memory. Both of these forms are likely to be fast enough to handle your data from the maple.

i hope this clears things up

Posted 5 years ago #
StephenFromNYC
Moderator

Hello,

Yes, it is completely clear that the external code which operates on the captured maple data does not need to be a DLL. Yes, any program which can read a serial port can then manipulate the captured maple data.

However, if a DLL already exists which does everything a user needs (and perhaps some things better than the user can code) why not use the DLL?

Another way to say this is why reinvent the wheel if the code (in the format of a DLL) has already been written, has been tested, and is available.

Thanks!

Stephen from NYC

Posted 5 years ago #
gbulmer
Moderator

Stephen - Just to get a bit better understanding, I'd like to ask a few more questions.

Are you hoping to use handy dynamic link libraries:
a. From Processing, or are there other acceptable languages?
Processing is Java underneath, so Java mechanisms like JNI might be enough. But there are other technologies for creating "foreign" interfaces like SWIG (http://www.swig.org/) which may be relevant.

b. On a single platform, or are you likely to need portability across Linux, Mac and Windows?
Clearly the DLL needs to be available on the platforms you want to use, but some of the mechanisms on each platform may be sufficiently different that it is worth looking at a technology, maybe like SWIG to get some cross-platform support. I realise you said Windows or Mac, with Linux a distant 3rd, but is their a DLL you have in mind which exists on more than one? (The DLL will be compiled specifically for each platform, and won't itself be portable, but it may contain the same functions).

Are you likely to have access to the source code?
It may be much harder to use the DLL in a robust way, from some languages, otherwise.

Can you write C?
A lot of ways at getting at low-level stuff use C or C++.

(Health Warning - I am not a SWIG user, though I did consider using it some years ago)

Posted 5 years ago #
StephenFromNYC
Moderator

Hello,

In my first post I said:

Does anyone have experience with using DLL to process data generated with a Maple?

This might be accomplished using Processing or another app as a "bridge" (assuming the DLL is running on the system processor, and not the microcontroller).

To be more clear what I should have said is that I was curious about possibly using a DLL running on the system (not on the Maple) to process data (on the system) AFTER the data is sent from the Maple to the system. All the Maple is doing is sending some information to the system. It was more of a "test-of-concept" question prompted by recent discussions of porting Arduino/MATLAB and Arduino/LabVIEW code to work with the Maple. Often not a trivial problem.

A) The "bridge" app (which transfers the data from the Maple to the DLL) could be written in C/C++, java, or another language. I just mentioned Processing as a possibility because it seems to be a popular language for manipulating data sent from the Arduino via the Serial port. Yes, I have seen projects which use Java to call DLLs.

B) These day when I program I use Windows (because I have LabVIEW on my Windows system). Most DLLs which I know are for Windows. Do other OS use DLLs? Honestly, I never liked the concept.

C) I am definitely now an elegant programmer (which is obvious from my use of braces when I code). I taught myself C a long time ago, but I never did any system programming. I once used some low level bit-manipulations for a graphics programming project.

A few years ago I was dual-booting Windows and Fedora. In order to get a wireless card to work I had to compile some code on my system. I think I used a Windows OS DLL to get wireless working on Fedora!

Today, if I do any non-trivial programming my choice is java. This makes it easier for me to use the code on computers running a different OS.

I like using XEmacs as my editor, but I will switch to Vim for some substitions.

I am curious about learning low-level programming using the Maple, because I want to modify the interrupts to improve ADC speeds.

Hope this summary helps.

Stephen from NYC

Posted 5 years ago #
gbulmer
Moderator

Stephen - I did understand that your were writing about using DLLs on the host, and not the Maple :-)

Yes, other OS's have dynamically linked libraries, some *NIX call them "shared libraries".

Several people believe shared libraries/DLLs are a bad idea.
For example, Rob Pike of UNIX, Plan 9, Squeak, and now Go lang. Go lang has no dynamic library support, partly because Go lang is designed for servers, and DLL's can be a security nightmare.

My favourite quote is

When Sun reported on their first implementation of shared libraries, the paper they presented (I think it was at Usenix) concluded that shared libraries made things bigger and slower, that they were a net loss, and in fact that they didn't save much disk space either. The test case was Xlib, the benefit negative. But the customer expects us to do them so we'll ship it.

Anyway Java has pretty good support for accessing the underlying system, so I'll wait until you have a concrete example before making anymore suggestions.

I am interested in fast ADC sampling, but I'm waiting to see what 0.0.7 brings.

For a regular signal, or one with a good external trigger, DMA is the way to sample as fast as possible.
Maybe even drop the processor speed to get the ADCs to run at 1Msample/second !-)

Posted 5 years ago #
StephenFromNYC
Moderator

Hello,

Yes, for purposes of increased ADC sampling speed (1 megasamples/sec) I think the lower processor speed is a reasonable tradeoff. It has been a few weeks since I last looked at the specifications sheet, but I think the STM32F103RB has to be slowed from 72 MHz to 48 Mhz for this to work.

I think I understand why DMA is useful to fast ADC (data movement around the microcontroller chip without using clock cycles), but I do not know how to get data from the Maple to the system at a high rate. What do people use?

Honestly, when I first read that the Maple board supports USB 2.0 I assumed this meant the very useful 480 mbps rate. However, this is the "high speed" rate. The "full speed" USB 2.0 rate is only 12 megaBITs per second (roughly 1 megaBYTEs per second).

On the Maple how fast can SerialN be pushed? 115200 baud is faster than 9600, but not fast enough if data is being collected at 1 megasamples/sec. Is high speed SerialN reliable at the highest speeds? Is high speed SerialN reliable for continuous periods of time or only in short spurts (think marathon running vs 100 meter sprinting).

Or do we need to use ethernet? On the Maple is the maximum speed 100 megabits/sec or 1000 megabits/sec? Which Arduino ethernet shields are most reliable for the highest rates of data transfer?

I think I will try pushing the Maple to 1 megasamples/sec after 0.0.7 comes out. It will be easier to benchmark the ADC using micros().

Hopefully where we go with our ADC projects will be more clear after 0.0.7 is released.

Thanks for your ideas and help! I am still a work in progress!

Stephen from NYC

Posted 5 years ago #
gbulmer
Moderator

The STM32F103 needs to have the processor clock slowed to 56MHz, so that the ADC clock is 14MHz. So better than you had expected !-)

Yes, I'd guess around 1Mbytes/second from Maple's on-board USB, until I have good tests.
That's approximately 100x faster than an Arduino, so I am still pretty happy :-)

The fastest STM32F103 serials are 4.5Mbit/second, so even slower than USB. I'd assume the STM32F USARTs are reliable enough that the host PC might be a problem.

I have only looked at a couple of Arduino Ethernet shields, and they were 10Mbit/second, but didn't get that fast as the SPI maxed out before then. The SPI on the Maple will go much faster, so they may still be worth investigating.

Other possible higher speed interfaces include the FTDI FT2232 module that Josheeg is using.
FTDI FT2232H datasheet claims
USB to parallel FIFO transfer data rate up to 8 Mbyte/Sec.
Single channel synchronous FIFO mode for transfers > 25 Mbytes/Sec.
USB Bulk data transfer mode (512 byte packets in High Speed mode).

Yes, once 0.0.7 is out, I'll have a better understanding if I want to hack on it or use it as it comes.

Posted 5 years ago #

RSS feed for this topic

Reply

You must log in to post.