Did I kill a maple? « LeafLabs Garden

LeafLabs Garden » Support » Maple Hardware Support

Did I kill a maple?

(14 posts) (4 voices)

Started 3 years ago by jake1981
Latest reply from gbulmer

jake1981
Member

Hi.

I have been working on a project with TTL jpeg camera + rs485 + bunch of other stuff.. And been busy writing some libraries for my project.. Skipping now to the interesting part..

I wrote a small protocol library for rs485 with 3 modules: command module (ascii commands), something like the serial command from arduino, receipt module for that (simple answering module), and a simple file transfer module ( and actually xmodem as well, but didn't use it with this).. Every module has a checksum system, so data can be validated. It's very simple but causes some extra data. But not a problem.

Well, I put all the pieces together, hardware and software and wrote a simple "server" software to my computer which I also connected to my rs485 bus.. And started a.. Small stress test.. My computer request the maple to take a photo with camera and then send it over rs485.. Loop on maple is a simple rs485 listener.. And loop on my computer is one that sends requests for image file to maple..
Been doing some tests during this day and lots of data has passed through my rs485 during the whole project.. But never so much as today.. My usual tests today were about 200 pictures, but finally I decided to stick with my code.. And started a final test, which ended with 370 images sent. These images are about 12000-16000 bytes in size.. (320x240)..

Transfers are done in 248byte blocks, because data must be readed from camera at same time, camera should support 115200, but the truth is that it doesn't- data loss is huge with anything else than 38400..
And camera also doesn't seem to support larger that 248 byte blocks to be downloaded.. So file transfer reads 248 bytes from camera with 38400 baudrate, makes a header for it and sends it to my computer along with header with 115200 baudrate- all this takes about 6 seconds.

I guess amount of pictures send today is nearly 2000 in complete, but those last 370 images were the last nail in the coffin so to speak.. I first suspected it's my code and started to make changes.. The problems were that my maple was sending false information.. I had bool variables for somethings, like: is camera frozen, was camera initialization okay, is TVout enabled, is internal motion detecting system enabled.. And if the camera has not been frozen to frame, maple still returned that yes, it is frozen because even though I set the boolean variable to false (trust me, I multiplied setting of variable to false and still..), it resulted the previous value..

There were other strange things too, data errors on the line started to occur a bit too often..

I finally took another maple and tried with that one.. And it works perfectly!

Now the question is; have I fried my maple's RAM? What are the limitations of this device? It is very propable.. That the broken maple cannot be revived, but is there a diagnostic software that I could flash on it that could be used to do some testing that what has gone wrong with it, so I could avoid breaking others as well? Okay, amount of data transmitted must be doubled as it's first readed from camera and then sent, I calculated that it's not huge gigabit like transfer, more like 50 megabytes.. But it's huge amount of data regarding to the project I've been planning here.. So this amount of data normally would never been transmitted through my maples.. But is there any point to keep working on this or am I just breaking all my devices here?

Posted 3 years ago #
ala42
Member

It is highly unlikely that the RAM has a problem. The RAM does not die by using it. You can kill the flash ROM, but not the RAM.
You probably have a bug in your software, e.g. a race condition by using non volatile declared variables in interrupt functions and main program. Also check for buffer overruns.

Posted 3 years ago #
higwoshy
Member

Could it be, if your getting data errors on one maple and not the other, that you've "over-exerted" one of your ports (in past experimentations) that you connect with and damaged it slightly?

It's possible to damage a port and it still function, but not as well as normal. If the damaged port is producing spurious results - resulting in unexpected data, then this could trip up your code at some point. i.e. a buffer overflow, values being set when the shouldn't etc. Have you thoroughly tested all your functions with abnormal data?

Try changing the ports that your using.

Just a guess, but as ala42 said, it's not likely that you've damaged your ram without major smoke appearing.

Posted 3 years ago #
jake1981
Member

I wasn't actually monitoring the test, so I am not sure about smoke, I also had fire in fireplace, so smoke here is pretty normal :)

I was wondering.. I am using 5v tolerant usart.. And Arduino Due, which is similar to Maple- says in it's docs that there are 5v tolerant pins, but driving 5v will eventually damage the device. Maybe this is issue with maple as well and I have transferred so much data over a 5v tolerant pins that something went partially broken?

I changed to another maple, used exactly same ports, and uploaded same code; it works like a it is supposed to. Haven't started yet a new stress test for obvious reasons :) Just don't happen to have any excess maples..

Posted 3 years ago #
gbulmer
Moderator

jake1981 - "I was wondering.. I am using 5v tolerant usart.. And Arduino Due, which is similar to Maple- says in it's docs that there are 5v tolerant pins, but driving 5v will eventually damage the device."

Where did you read that?

I think the Arduino Due page is very clear that the Arduino Due does NOT have 5V tolerant pins
http://arduino.cc/en/Main/ArduinoBoardDue

If you look at the Atmel document on the Arduino Due's SAM3X MCU, doc11057.pdf in section "46.1 Absolute Maximum Ratings":
"Voltage on Input Pins
with Respect to Ground....................................-0.3V to + 4.0V"

So, it is actually different from a Maple's STM32F103.

RAM is not going to "wear out". If the processor on your Maple is working at all, it is extremely unlikely RAM has been damaged in any way.

Each GPIO output has complementary pairs of MOSFETs (N and P channel), and it is possible to damage one of the pair of MOSFETs without destroying the entire GPIO capability. For example by connecting two pins together (maybe of different pieces of electronics), and driving them both at the same time as outputs. It is possible to make this situation safer by putting series resistors in the signal path.

Posted 3 years ago #
jake1981
Member

Okay, it actually seems that I haven't readed that arduino has _any_ 5v tolerant pins. I have mixed some things.. I actually made a google search on that and atleast first page, there weren't any site saying it would have any 5v tolerant pins. I just saw that text somewhere, propably on arduino forum.. Or maybe it was something else. My bad anyway.

I already have a replacement board in order- and I never was feeling myself hopeful to salvage this board. Just hoping that I won't redo my error again, and this posting here, should be a warn for others as well, although, the reason should be found so people could be cautious of what they do. I've made some simple testing- running a small and simple program on that broken board.. Is a success.. For example, doing some simple calculations with variables ( store to int16 a and b a number and store their sum in c and output it to the serialUSB) works just fine..

Should had make a loop that would change the numbers to see long time results.. Well, it's not too late, not going to toss that board as it works atleast somehow..

But with larger and more complex projects, even simple variables stop responding. And it's not that I would be running out of RAM as I found a nice piece of code from this forum that outputted a free ram amount (or atleast an estimate).. And more than half should still be free..

I have a function that sets a boolean to false, boolean which is set in setup to true.. Before it's set to false, it's value is output to SerialUSB and it does exactly what it shoulds; reports false (0).. Then a function is started, actually a void, but anyway.. It does not do anything else, except set it to true.. And after this, value is outputted to serialusb.. Still false.. I'll skip the void, and set it manually between two SerialUSB.print lines.. Still false..

And if anyone is wondering if it's my code, here it is:
SerialUSB.println(testingBool == false ? "False" : "true");
testingBool = true;
SerialUSB.println(testingBool == false ? "False" : "true");

There is _nothing_ that could set it back to false here.. So this is kinda the reason why I am thinking about malfunctioned Ram..
And the same code to another maple.. No problems. Oh, one more symptom.. Strange time-outs.. Sometimes while waiting it to output something to serial.. There might be a 10-15 seconds gap.. Not with this, but when going to filetransfer which is a bit heavier than simple calculus..

5v tolerant or not.. Or capable of handling 5v for longer time.. I am now using a logic level converter to be sure..

p.s. code is not actually a copy paste- but that's how simple it is and that's how it behaves.

Posted 3 years ago #
ala42
Member

Try again with
SerialUSB.println((int)false);
SerialUSB.println((int)true);
bool testingBool;
testingBool = false;
SerialUSB.println((int)testingBool);
SerialUSB.println((testingBool == false) ? "False" : "true");
testingBool = true;
SerialUSB.println((int)testingBool);
SerialUSB.println((testingBool == false) ? "False" : "true");

Posted 3 years ago #
gbulmer
Moderator
jake1981 - when I wrote 'RAM is not going to "wear out". If the processor on your Maple is working at all, it is extremely unlikely RAM has been damaged in any way.' I did mean extremely unlikely.

The probability I was guessing might be less than 1 in 20,000,000 chance (less likely than I could buy a single ticket and win the UK National Lottery). I may be being unreasonably pessimistic about the hardware, and the odds are worse than 1 in 1,000,000,000.

To be able to load a program RAM is working. The Maple bootloader uses RAM to load a program. There is no magic.

It is MUCH less likely that part of RAM is not working, than all of RAM being damaged. So under these circumstances it is MANY times more likely that there is a bug in software or in your electronics, or in libmaple, or on the electronics of the board or in the tool chain, than the RAM of the STM32F is partially damaged.

Science and Engineering work by gathering evidence to support a hypothesis or characterise behaviour in a meaningful, robust and reliable way. With long odds against, it is not enough to do one experiment, using fragile technology (a program) and run it many times. It is more likely that a buggy program will give a consistently wrong answer than be sometimes correct, and sometimes wrong. It is important to reduce the imporance of fragile technology (programs) by devising several different ways to prove the existence of an unlikely fault.

A useful initial approach is to simplify the program to a minimal case that demonstrates the bug. This also removes some of the possible defects that may 'confound' the result. Then try to achieve the same effect using a different program.

I think ala42's approach is good, but in the original program, it would be straightforward to substitute:
```
if (testingBool == false) {
    SerialUSB.println("False");
} else {
    SerialUSB.println("true");
}
```
for SerialUSB.println((testingBool == false) ? "False" : "true");

But I'd recommend replacing the whole thing with SerialUSB.println(testingBool); because all you are trying to see is the value of testingBool changing. There is much less to obscure the error.

To get a second piece of evidence, I might have simplified the test of a single bit of RAM to something like:
```
int flag = LOW;

void setup() {
    // Set up the built-in LED pin as an output:
    pinMode(BOARD_LED_PIN, OUTPUT);
}

void loop() {
    digitalWrite(BOARD_LED_PIN, flag);
    flag = !flag;
    delay(100);
}
```
This has a helpful property that it doesn't need USB to work. That is good because USB relies on software in libmaple and on the host PC both working.

It also has the helpful property that it is easy to change the pin (and add an LED or use some other device to detect the pin state) and so reduce the chances of it being that piece of hardware.

It is also practical to replace the digitalWrite, and go straight to the hardware port to toggle the pin, removing even more of the defect sources that may confuse the result.

Of course, that program, like yours, only test one small part of memory. The chances of the RAM working well enough to load a program, yet a part of it containing the single variable that is used to test it, is probably less than 1 in 20,000 worse than the odds of part of RAM being broken, but enough working to bootload. So it is s poor test; it is not likely that a part of RAM is damaged, yet that program would find it. So to be a meaningful test, I would generalise this to something which writes all bit patterns over most of RAM (a big array), and checks the patterns are written and read correctly.

Are you using this project to learn C? It is a helpful tactic to 'over use parenthesis' and keep expressions simple; this reduces the chance of errors.

C operator precedence is listed at http://en.wikipedia.org/wiki/Operators_in_C_and_C%2B%2B#Operator_precedence

May I suggest you try to answer these questions (I think they are roughly in order of probable to unlikely):
- What is the likely hood that there is a bug in the program, and the defective behaviour is related to that, rather than something else?
- What is the likely hood that there is a bug in my electronics, and the defective behaviour is related to that, rather than something else?
- What is the likely hood that there is a bug in libmaple, and the defective behaviour is related to that, rather than something else?
- What is the likely hood that there is a bug in the Maple board electronics (outside the STM32F), and the defective behaviour is related to that, rather than something else?
- What is the likely hood that there is a bug in the Maple toolchain, and the defective behaviour is related to that, rather than something else?
- What is the likely hood that there is a bug in the host OS that talks to the Maple (not the whole thing), and the defective behaviour is related to that, rather than something else? (answer may depend on OS :-)
- What is the likely hood that there is a powerful enough radio active source near enough to change a bit in RAM, and the defective behaviour is related to that, rather than something else? (may depend on your location and other activities :-)
- What is the likely hood that there is a bug in part of the STM32F which allows it to load a program, but other parts of the core (CPU or RAM, not I/O) are broken, and the defective behaviour is related to that, rather than something else?
(Hint, I think the last one is somewhere between 1 in 10^7 to 1 in 10^9.)

[I am not saying it is impossible that one bit of RAM is not working, just that it is more likely that RAM is being corrupted by alpha particles or radio active decay than a little bit of it has been damaged, but enough is working to load a program.]

Based on this sort of failure mode analysis, you may find you make progress much faster because you focus on finding the more likely sources of error, and remove them, before attempting to find unlikely sources of defects using techniques which are themselves unlikely to locate those harder to find defects.
Posted 3 years ago #
jake1981
Member

1) Likelyhood of a bug in my program? Very propably that there are many bugs, but nothing causing this. And if it's a bug in my code, why does exactly same code work properly on another more or less identical device (can these ever be identical?)..

2) Bug in electronics? Maybe the 5v tolerant serial should be logic level converted to 3.3v just to be sure. Well, actually I made this. But it would be hard to debug this without braking more of maples..

3) I will rule out bug in libmaple with the same answer I gave for question 1.

4) Unknown. Can't really say.

5) Answer to the question 1. Same stuff works for another board.

6) Possible. My iMac sometimes (not often) hangs and only way to get back to business is a cold boot, as computer itself isn't capable of closing applications and to reboot or shutdown. But, I don't think this is it. My web browser, Mail and other apps were working at that moment perfectly, and when hanging occurs-- almost nothing works.. Atleast if it uses network connectivity. Issue that occured after upgrading to new version of OS, something got wrong in upgrade.

7) Okay, there is 6 MQ-2 gas/smoke sensors in range of 30 cm from my Maple. As far as I know, these contain some radio active stuff or something similar. Although, none of these is connected/powered. Can it be this? I don't knowingly have any other radio active material.. Not even microwave material as I don't use microwave oven, so I don't even mind owning one..

8) Oh you wanted answers in numbers :) Now I have to start from the beginning again :) Just kidding.. Yeah, I know it's rare that this happens. But I am pretty sure it can't be anything else. And yes, I agree that RAM is not broken more than partially. But I think it might be this..

---

Okay, this has gone to crazy man hunt-- I am not angry, not to anyone, and not to any device. Ofcourse it's sad that I managed to broke this device and one could get angry as they cost some $$$.. Pretty cheap anyway, and can't be angry to anyone as this propably somehow is all my own fault, effect of my actions, just not sure what it is and was here trying to solve it, but this thread.. Doesn't seem to lead to solutions for me - nor in future to anyone else.. (unless my recent answers changed everything)
so this thread is becoming spam or something useless..

Was just thinking that if there's a beginner's error that I might have caused and that this could work as an warning to others..
But it seems that this can't be solved or we are just agreeing to dis-agree opinions pointed by others that ourselves..
(I hope that sentence makes any sence..)

Oh, but one more thing came to my mind.. I was reading docs.. And there was suggested that I should use new instead of malloc..
I've been using malloc.. Maybe it's because of that? Actually the routine I was using to read the data from camera uses a pointer that has been initialized with malloc, where it's then stored.. So this starts to make sense possibly after all.. So before it broke down, there had just been nearly 2000 mallocs.. Does this ring any bells? Possibly an alarm bell?

Posted 3 years ago #
higwoshy
Member

As I see it, you've solved the problem by swapping to a new maple. Easy solution, but expensive.

Now you are in inquisitive mode - trying to find out why it failed. That's good.

What others are saying is that if you want to find out what failed, then you have to start thinking how it might have failed and then testing this theory to *prove* or *disprove* it. This is how programmers minds (eventually) work.

For example, was it the ram that failed? Probably not, but prove it by writing a ram test program. The docs are available to give you locations to run the test. It's not a multi-tasking OS, so as long as you disable interrupts and only use one or two I/O port to indicate busy/failure/complete it should be ok. Maybe someone has written one already? If only one or two GLOBAL variables are causing problems, then print out their address and then just run a ram test around this location. You do know that local variables are stored on the stack? Perhaps incorrect 'data in' is causing the stack to overflow.

Guessing why something may have failed is half the work, to finish the job you have to prove it. You say your program could not have caused the bug, but you've given no indication of running tests to prove it (to yourself). How does your program deal with massively corrupt data? Does it hang, does it allocate too much memory/stack? Does it make assumptions at any point? Have you tried I/O on other ports? Have you tried testing the ports? Have you used the same wires for each maple? Jiggling wires around causes problems?

These are just a few examples of possible failures. Debugging is an art, and like all art the more you do it the better you become.

BTW malloc will return the same results on whichever Maple it is run, i.e. if in setup you ask for 200 bytes, it will always return the same address (assuming a reset). So, assuming the same data in, the same allocations will occur at the same address on either Maple (I hope that makes sense)

TLDR : Find out the difference between the broken maple and the good maple. What possible errors could a broken maple introduce into your program to cause it to fail? Guessing will have you going round in circles, possibly forever. Only testing will result in answers.

Posted 3 years ago #
jake1981
Member

I swapped it immediately after it happened, it says so in the 1st post of this thread :) And I also ordered one spare.. But the point is that I rather not start broking these daily, trying to find my error :)

Maybe I can fill out ram by creating a nearly all available ram size variable and fill it with incrementing number starting from 0 ending to 255 and after that 0 again and then test it by reading it..

2 maples conclusions:
similarities: with bare eye, they are identical. No exploded components found. Boot speed, to human, seems identical.
differences: broken has usb port bit upwards.. Was always like this. Propably could bend it if this would matter. broken.. Does not work fully and another one has no problems..

When running this program of my on broken one it slows down, takes pauses, which can be noticed by human eye when watching serial traffic from the computer. Not sure if it hangs, it might be in a protocol caused loop (thinking that computer sent that received packet is wrong and waiting for request for retry).. If I remember it correcly, I am not capable of testing at the moment..

Posted 3 years ago #
gbulmer
Moderator
jake1981 - "1) Likelyhood of a bug in my program? Very propably that there are many bugs, but nothing causing this."
Just to be clearer, as ala42 pointed out your program:
```
SerialUSB.println(testingBool == false ? "False" : "true");
testingBool = true;
SerialUSB.println(testingBool == false ? "False" : "true");
```
has several bugs.
They would cause the apparent value of testingBool to be unchanged because it doesn't compare testingBool with anything it will ever equal.

IMHO it is meaningless to continue without correcting that.

It is inevitable that the value printed will not change. I think it is pretty clear why if you consider the corrected statement:
SerialUSB.println((testingBool == false) ? "False" : "true");

As we have tried to explain, you need to prove a theory with evidence derived using something valid. Further, it is worth working from the likely to the unlikely.

As your example program fails at the first step, it contains bugs, and only tests a single int from 20kB, it yield no support for the hypothesis that some RAM is damaged.

I think we are actually trying to help you improve your programming skills, by helping you think about how to debug, then building ways to get evidence.

You seem to resist the observation that your test program is buggy, and that the bugs will cause the incorrect output. I don't understand why.

If anyone is reading this thread, please consider the advice about debugging, and gathering evidence. Try to get positive evidence.
Posted 3 years ago #
jake1981
Member

let's keep it short. If the only problem is my code: _why_ does it work properly on another device?

Posted 3 years ago #
gbulmer
Moderator

1. I'm not saying the *only* problem is your code. It would be unreasonable to assume there is a single defect; another defect (non-5V tolerant Arduino Due pins) was discovered during this thread, so the odds are there are more.

Something working simply because a chunk of electronics is changed doesn't mean the cause was identified, understood and removed ensuring it won't happen again. A metaphor is blaming the engine in a car, then buying a whole new car, when it was just the gas (petrol) in the tank. Use bad petrol (gas) again, and the problem will return.

2. It may look like it is working on another device, but the underlying fault may be being masked or was accidentally removed.
This is how myth and magic were created. A metaphor is blaming buying a car on Friday 13th for a failure, then buying a whole new car to fix it. The fact that the error went away does not prove the proposition that cars bought on a Friday 13th are defective; it may fix the problem, but it might have been the gas (petrol) in the tank, etc. So the cause may become visible, r get reintroduced.

My goal here is to try to encourage folks to think carefully and rationally about causes and tests.

Clearly, you are free to spend your money on buying Maples; "knock yourself out". I don't work for LeafLabs, so it makes little difference to me.

I'll continue to try to identify approaches or conclusions which may not be sound, and so may need treating with scepticism.
I think we have explained enough for the benefit of others.

Posted 3 years ago #

RSS feed for this topic

Reply

You must log in to post.