Sending data over USB « LeafLabs Garden

LeafLabs Garden » Support » Maple Hardware Support

Sending data over USB

(26 posts) (8 voices)

Started 4 years ago by Silntknight
Latest reply from gbulmer

« Previous 12

StephenFromNYC
Moderator

Hello,

I think I have confused other users and myself, so let me step back and try to summarize what I understand from recent posts about how to optimize SerialUSB() data transfers. Then at the end of this post I will try to refocus my questions.

In my post:

http://forums.leaflabs.com/topic.php?id=639#post-3629

I was wrong to focus on how to send just five 12-bit analog reads, because I incorrectly thought that sending five 12-bit values was the most one could fit into a single USB 64-byte packet. In my hurried and faulty logic I got my bits and bytes confused. Obviously, 60-bits is a lot smaller than 64-bytes. Forty (40) 12-bit values acquired using analogRead() should fit into a single USB 64-bit packet.

For the moment, let's just stick with the goal to efficiently send five 12-bit values using SerialUSB().

I feel like I am missing something really fundamental in my reading of gbulmers Maple code (http://forums.leaflabs.com/topic.php?id=639#post-3660) which he said was compiled but not tested.

I believe the five 12-bit values are first combined into a single variable (res). res is then broken down into pieces and each char is sent to the host using SerialUSB().

I understand the line from gbulmer's code:

long long res = (a[4]<<48) | (a[3]<<36) | (a[2]<<24) | (a[1]<<12) | a[0];

This uses bitshifting and ORs to combine the five 12-bit values into a single variable of type long long.

However, I am confused about gbulmer's for loop:

for (int i=0; i<8; i=i+1) {
unsigned char c = res & 0xff;
res = res >> 8;
SerialUSB.print(c, BYTE);
}

This code uses a mask to strip the higher bits. Then to prepare the variable res for the next cycle of the loop bitshifting (res = res >> 8) is used before the value of c is sent via SerialUSB.print(c, BYTE).

The loop is executed eight times for a total of 64 bytes transferred.

Why are multiple SerialUSB.print() statements used if the goal is to send the five 12-bit values in a single USB 64-byte packet?

My assumption is that each time (for a total of eight) SerialUSB.print() is called the data is transferred over USB even if the 64 byte packet is mostly empty (not filled with data).

If this assumption is wrong, does the USB system wait until the eighth SerialUSB.print(c, BYTE) command before a single, completely filled, 64-byte packet is sent?

Why is it not possible to send the entire variable res using a single SerialUSB.print() command? For example, SerialUSB.print(ser, LONG LONG).
In the host routine, the process is reversed. Eight SerialUSB.read()s are used to fetch each 8-bit value, the variable res is shifted by eight bits to prepare for the use of OR. This combines eight 8-bit reads into a single long long.

In the host code, should the line:

a[j] = res & 0x0fff;

instead be:

a[j] = res & 0xff;

Here are my refocused questions:

Goal (present): I want to use my Maple to acquire 12-bit data via analogRead() as quickly as possible and to send the data to the host via USB as quickly as possible (without any data being dropped). A higher data transfer rate will let me apply software filters to the data set (for example, to remove 60 Hz noise from the data). I do not know if I will use the Maple to do the filtering or use the host. I will be happy to shift the filtering to the host if this maximizes the data transfer rate.

A transfer rate of 20,000 data points a second (where each data point is a 12-bit analogRead() value) should be sufficient for my long term goals. However, a rate which is 10X slower will make me happy in the short term and will keep me plugging away.

Goal (future): DMA will speed up the data transfer between the Maple and a host system, but I think for my purposes DMA is not needed.

I may be wrong to minimize this task, but at the moment, the exact rate of analog reads is not that important to me. That can be a problem for another day.

I appreciate that two bytes are wasted using SerialUSB().println(), but I was using SerialUSB.println(), to see the boundaries between analogRead() values when I debugging.

mbolivar, in your comment from:

http://code.google.com/p/leaflabs/issues/detail?id=10

Can you please share the code you used to determine the current SerialUSB() rate to be about 150KB/second (using the February 2011 snapshot)?

I understand that at most five 12-bit values fit into one long long (using the line from gbulmer's bitshifting code). A long long is eight bytes, so at most forty (40) 12-bit numbers fit into 64 bytes.

Therefore, in the absence of data compression/expansion a single USB 64-byte packet can transfer forty (40) 12-bit values (acquired using analogRead()).

My next question is: "How do I combine eight long long values into a buffer which I can efficiently send in a single 64-byte USB packet using SerialUSB.write(buffer, 64)"?

I assume it will be necessary to flush before calling SerialUSB.write(buffer, 64):

http://forums.leaflabs.com/topic.php?id=575

Thanks and sorry for any confusion caused!

Stephen from NYC

Posted 4 years ago #
gbulmer
Moderator
StephenFromNYC - yes my code may be a bit confusing. I was focusing on sending the data so that it could be accurately reconstituted. I try to ensure the sequence of bytes is defined by my code rather than assuming both sides are little endian or big endian.

The loop is executed eight times for a total of 64 bytes transferred.

The loop transfers 64 bits, 8 bytes, not 64 bytes.

Why is it not possible to send the entire variable res using a single SerialUSB.print() command? For example, SerialUSB.print(ser, LONG LONG).

This doesn't work because:
a. there is no such function as SerialUSB.print(ser, LONG LONG), (and I'm not sure what LONG LONG signifies, a parameter to print, or the data type long long with capitalisation for emphasis?) and
b. all of the numerical print functions convert binary values into strings of ASCII digits. For example a 4 byte uint32 could be anything from 1 ASCII digit to 10 ASCII digits long. So even if it did exist, it is likely that it would convert binary to ASCII.

The way to transfer a 64 byte block in a single USB packet is to do a SerialUSB.write(void *, uint32); with a 64 byte buffer.

If you only need 20,000 12-bit data points/second, then I think the Maple can do that. The inefficiency is sending near-empty packets, not the peak throughput.
I'd likely just send 32 values/64 byte packet, and avoid any concerns about packing for now.

You have several choices about how to pack stuff into a 64 byte buffer.

If there is no bit-level packing, I'd do something like this:
```
unsigned char buffer[64];
int buffindex = 0;

void writebuff(uint16 value) {
  if (buffindex >= sizeof(buffer)-1) {
    SerialUSB.write(&buffer[0], sizeof(buffer));
    buffindex = 0;
  }
  buffer[buffindex++] = value & 0x00ff;
  buffer[buffindex++] = (value>>8) & 0x00ff;
}

void loop() {
  writebuff(analogRead(0));
  writebuff(analogRead(1));
  writebuff(analogRead(2));
  writebuff(analogRead(3));
  writebuff(analogRead(4));
  // other stuff
}
```
(WARNING code not checked)

This handles the write when the buffer gets full, without any packing.

Take things from there.

If packing becomes important, handle it inside writebuff.

PS - the
```
buffer[buffindex++] = value & 0x00ff;
  buffer[buffindex++] = (value>>8) & 0x00ff;
```
maybe easier to read as
```
buffer[buffindex] = value & 0x00ff;
  buffindex = buffindex + 1;
  buffer[buffindex] = (value>>8) & 0x00ff
  buffindex = buffindex + 1;;
```
(sorry that the indentation is broken, I can't get leading spaces into the first line using backtick)
Posted 4 years ago #
StephenFromNYC
Moderator

Hello,

Yes, I understand that buffer[buffindex++] increments the value of buffindex AFTER the array assignment is made. I understand that the lines buffindex = buffindex + 1; can both be replaced by either buffindex++ or ++buffindex.

I have a question about expressions and compiler optimizations used by the Maple.

In the line SerialUSB.write(&buffer[0], sizeof(buffer)); is there any difference between &buffer[0] (the address of the zeroth element of the array named buffer; in other words the beginning of the array) versus just using just "buffer".

In other words are the next two lines equivalent (in terms of machine code created by the compiler)?

SerialUSB.write(&buffer[0], sizeof(buffer));
SerialUSB.write(buffer, sizeof(buffer));

Thanks!

Stephen from NYC

Posted 4 years ago #
gbulmer
Moderator

StephenFromNYC

In other words are the next two lines equivalent (in terms of machine code created by the compiler)?

SerialUSB.write(&buffer[0], sizeof(buffer));
SerialUSB.write(buffer, sizeof(buffer));

Yes, in this case the machine code generated by the compiler for &buffer[0] and buffer should be identical, and should be identical in general.

I did the &buffer[0] deliberately for a couple of reasons; partly to ensure the difference between a pointer (unsigned char *) and an array is clear (the number of bugs I've seen involving unallocated space!-), and partly it makes it more obvious (to me) that there may be a bigger buffer (than 64 bytes).

For example, if the ADC values were being saved into a buffer using DMA, then the DMA 'end event' might trigger after 64 bytes worth of results. Then the DMA event handler might reset the DMA to another buffer, or the other half of the same buffer, and trigger sending the USB data to the host.

That sort of approach might give a more uniform sampling rate for the set of ADC samples (I don't know how long the SerialUSB.write(...) might block for, so this is an unknown concern).

(I could have defined something like MAX_USB_PACKET, and used that instead of sizeof(), but it started to clutter the code.)

(Technically it was the case that buffer is a const pointer, so is only a right-hand-side expression because it can't be assigned to, but I haven't worried about that level of C language precision for years, and I'd need to check Harbison & Steele http://www.amazon.com/Reference-Manual-Samuel-P-Harbison/dp/013089592X/ref=dp_ob_title_bk but I can't find my copy of the 5th edition :-(

Posted 4 years ago #
mbolivar
LeafLabs

Stephen,

mbolivar, in your comment from:

http://code.google.com/p/leaflabs/issues/detail?id=10

Can you please share the code you used to determine the current SerialUSB() rate to be about 150KB/second (using the February 2011 snapshot)?

It seems like you're referring to this comment, which was not left by me:
http://code.google.com/p/leaflabs/issues/detail?id=10#c4

I am personally unable to reproduce those results, but I will ask the comment's author to do so :).

Posted 4 years ago #
StephenFromNYC
Moderator

March 16, 2011

Hello mbolivar-

Any feedback from the author ("ajme...") who claimed a rate of 150 kilobytes/sec using SerialUSB() and the February 2011 snapshot?

I appear to be getting a fast data transfer rate, but the rate does not appear to be consistent (depending on how the rate is tested the transfer rate bounces up and down a lot).

It would help me to see the code and test method used by "ajme..."

Thanks!

Stephen from NYC

Posted 4 years ago #
mbolivar
LeafLabs

StephenFromNYC,

"ajme..." is poslathian on the LeafLabs forum.

I asked how he obtained his figure, and (as far as I recall) his method was (on the host PC side) to use the Unix "cat" utility to pipe the output of the virtual serial port's device to a file in Linux (e.g. $ cat /dev/ttyACM0 > some-file). Meanwhile, the Maple was sending data in a tight loop using the low-level usbSendBytes() function. He then timed how long it took for the file to reach a certain size.

I'd like to emphasize that this is only my recollection of his methods. Poslathian, perhaps you'd care to weigh in?

I personally have been unable to achieve transfer rates greater than ~30KB/s using the current code.

Posted 4 years ago #
gcam
Member

hi (google traslator),
I send the data on the USB with the following instructions

void writebuff(uint16 value) {
if (buffindex >= sizeof(buffer)-1) {
SerialUSB.write(&buffer[0], sizeof(buffer));
buffindex = 0;
}
buffer[buffindex++] = value & 0x00ff;
buffer[buffindex++] = (value>>8) & 0x00ff;
}

void loop() {
writebuff(analogRead(0)); // x axis (value +- 32000)
writebuff(analogRead(1)); // y axis (value +- 32000)
writebuff(analogRead(2)); // z axis (value +- 32000)
...
}

On the PC (visual basic 2010) I read the values are between 0 and 255.

How do I go back to the values obtained from the accelerometer.

Posted 4 years ago #
gbulmer
Moderator

I have no idea if there is bit shifting in vb2010, but it isn't needed.
A left shift of 8 (which reverses the right shift of 8 in writebuff) is exactly he same as multiplying by 2^8, which is 256.

So, if you read the values as 0 to 255, then to rebuild a 16 bit value an expression would be something like
wholeValue = read0 + (read1 * 256)
where read0 is the first byte and read1 is the second byte read.

Posted 4 years ago #
gagol
Member

I'm a little late, but if you still needs to convert ascii to int, here is a little function that works for me :

char buffer[] = "00535";
int value = atoint(buffer[0], 5);

The code requires leading 0.

uint32 atoint(uint8 *t, uint8 _digits) // CONVERT ASCII NUMBERS TO INT, PASS POINTER TO *CHAR LOCATION OF THE BEGINNING OF
{
uint32 _val = 0;
xb = 1; // FACTOR FOR EACH DIGIT PROCESSING
while(_digits--) // FOR EACH DIGIT STARTING FROM RIGHT END GOING LEFTWARD
{
_val += (t[_digits] - '0') * xb; // ADD CURRENT DIGIT * FACTOR POWER TO VALUE
xb *= 10; // AUGMENT FACTOR POWER BY 10
}

return _val; // RETURN INT
}

Posted 4 years ago #
gbulmer
Moderator
gagol - It might be helpful to know that Maple comes with the Open Source Newlib implementation of most of the Standard C library:
http://sourceware.org/newlib/

The standard atoi has already been implemented.
Use it as:
```
#include <stdlib.h>

//...

    char buffer[] = "00535";
    int value = atoi(buffer[0]);
```
All strings, e.g. char buffer[] = "00535"; are terminated with a '\O' (end of string) character, so there is no need to pass the length of the string to atoi(). The atoi function converts until it reaches the end of string, or a non digit character.

Newlib includes a further set of conversions functions including strtol, strtoul, etc. which handle number bases other than 10.

It also includes sscanf() which can extract pieces of string and convert them into any basic C types.

While is is only a convention, functions which convert to unsigned numbers, like that atoint, might be better named atouint. The comments might also mention the restriction that it only works for unsigned integer strings, so that people don't misunderstand. It might also be worth mentioning in the comment that it has 'unpredictable' results if the string contains characters which are not ASCII digits.

The usual way of doing the conversion from a string of ASCII digits to integers starts with something like this:
```
#include <ctype.h>

uint32 atouint(char* t) {
    uint32 val = 0;
    while (*t != '\O' && isdigit(*t)) {
        val = val*10 + (*t - '0');
        ++t;
    }
    return val;
}
```
This checks that the characters in the input string are the ASCII digits. It dispenses with the need to track the number of characters in the string, by relying on the end of string, and doesn't separately track powers of 10 because it converts left to right.

A small additional point if you are aiming to produce code for other people to read. Through hundreds of years of research and experience, we believe that in general, mixed case text is easier to read than UPPER CASE text. We also believe that lower case text is more readable than upper case text, hence the reason that C is a lower case programming language. Mixed upper and lower case text, or only lower case, has a lot more 'shape variation' than upper case only, and so we read it more quickly and more reliably.

HTH

(WARNING: I haven't compiled and tested the code, it is just an illustration)
(PS - because of the abysmal forum software, which strips '\' followed by '0', I have used the letter O to create '\O' to stand for the character with the value 0. Yes, ugly)
Posted 4 years ago #

RSS feed for this topic

« Previous 12

Reply

You must log in to post.