LeafLabs Garden » Topic: uint64 polarity problem

LeafLabs Garden » Topic: uint64 polarity problem http://forums.leaflabs.com/topic.php?id=1524 A place to share, learn, and grow... en-US Fri, 22 Jan 2016 00:18:17 +0000 http://bbpress.org/?v=1.0.2 <![CDATA[Search]]> q http://forums.leaflabs.com/search.php gbulmer on "uint64 polarity problem" http://forums.leaflabs.com/topic.php?id=1524#post-10119 Fri, 06 Apr 2012 13:07:02 +0000 gbulmer 10119@http://forums.leaflabs.com/ samtal - I would have written the code I posted, and been puzzled too, so I was pleased to learn. Thanks for finding such a good question :-) <blockquote>My understanding is that because I did not specify the number as unsigned, the compiler defaulted to signed int (or long-long). </blockquote> Yes, <code>int</code> which is signed. I would bet money that that wasn't the case in K&R UNIX BSD C. <blockquote>Then, because the high number was assumed to be signed and negative (msb=1), the byte became also signed, and negative (also, msb=1). </blockquote> It is the fact that constant is a <code>signed int</code> that causes the first half of the problem. That is a problem waiting to happen. The <code>int</code> constant is wider than the <code>unsigned char</code>, so the <code>unsigned char</code> needs to be converted (widened), and that is to an <code>int</code>. AFAIK C99 widens to <code>int</code>. I assume it wasn't adequately specified before C99 (or they needed to change C to keep lots of consultants in work - sorry being a bit cynical). The <code>unsigned char</code> is multiplied or shifted (both fail) to an (signed) <code>int</code>. Then the (signed) <code>int</code> (which has its msb=1) is 'widened' to <code>unsigned long long</code>. <blockquote>1. How come two negative numbers gave a negative product? </blockquote> What are the two negative numbers? I think the only one is in_command[[]. <blockquote>2. What happened to the byte? did it lose its msb to the sign bit? </blockquote> when it was multiplied, yes its top bit became the sign bit of an int. samtal on "uint64 polarity problem" http://forums.leaflabs.com/topic.php?id=1524#post-10118 Fri, 06 Apr 2012 09:51:19 +0000 samtal 10118@http://forums.leaflabs.com/ Thanks gbulmer, You had to dig quite deep to get all that info. Chapeau. No doubt that left shift is better than multiplication (and probably significantly faster). Originally, this is what I had, but then when I started to get problems I changes to multiplication, and was left there. I will change it back and see if it will become cleaner. Regarding your comments: My understanding is that because I did not specify the number as unsigned, the compiler defaulted to signed int (or long-long). Then, because the high number was assumed to be signed and negative (msb=1), the byte became also signed, and negative (also, msb=1). Question: 1. How come two negative numbers gave a negative product? 2. What happened to the byte? did it lose its msb to the sign bit? Well, it can become quite complex. I'll juts make sure it works for me, and be more careful next time. I'll keep the securecoding link. It is important. I hope others will gain some knowledge of our work (mainly yours...) samtal gbulmer on "uint64 polarity problem" http://forums.leaflabs.com/topic.php?id=1524#post-10109 Fri, 06 Apr 2012 03:23:37 +0000 gbulmer 10109@http://forums.leaflabs.com/ C/C++ integer operations are always carried out as if the two types were the same. According to these, conversions preserve the bits, but not signed-ness (since C99): <a href="https://www.securecoding.cert.org/confluence/display/seccode/INT02-C.+Understand+integer+conversion+rules" rel="nofollow">https://www.securecoding.cert.org/confluence/display/seccode/INT02-C.+Understand+integer+conversion+rules</a> <a href="http://msdn.microsoft.com/en-us/library/fc9te331%28v=vs.80%29.aspx" rel="nofollow">http://msdn.microsoft.com/en-us/library/fc9te331%28v=vs.80%29.aspx</a> The first one at securecoding.cert.org has an example "Noncompliant Code Example" involving 'port' which has some of similarity with the problem code. when <code>0x1000000</code> is an <code>int</code> then operation <code>in_command[1] * 0x1000000</code> will give <code>int</code>. Then the <code>int</code> is converted to <code>uint64</code> for the <code>datetime + (in_command[1] * 0x1000000);</code> So writing <code>0x1000000</code> as <code>0x1000000u</code> will cause the expression <code>in_command[1] * 0x1000000</code> to be <code>unsigned int</code>, so there will be no sign propagation. Similarly explicity casting will also ensure nothing becomes signed <code>(unsigned long long)in_command[1] * 0x1000000</code>. The most explicit <code>(unsigned long long)in_command[1] * 0x1000000u</code> is better. If you added this code to your program: <pre><code>void show_inputs() { Serial1.println(); datetime = (uint64)in_command[0] << 32; Serial1.print("datetime="); Serial1.println(datetime, HEX); datetime = ((in_command[1]) << 24); Serial1.print("datetime=(uint64)"); Serial1.println(datetime, HEX); datetime = (unsigned int)((in_command[1]) << 24); Serial1.print("datetime=(unsigned int)"); Serial1.println(datetime, HEX); datetime = (uint64)((in_command[1]) << 24); Serial1.print("datetime=(no cast)"); Serial1.println(datetime, HEX); datetime = in_command[2] << 16; Serial1.print("datetime="); Serial1.println(datetime, HEX); datetime = in_command[3] << 8; Serial1.print("datetime="); Serial1.println(datetime, HEX); datetime = in_command[4]; Serial1.print("datetime="); Serial1.println(datetime, HEX); }</code></pre> and ran it, hopefully the problem would be clearer. (There should be a warning from <code>datetime = in_command[0] << 32;</code>) EDIT: This is the output (on my Mac): <pre><code>datetime=8000000000 datetime=(uint64)FFFFFFFF81000000 datetime=(unsigned int)81000000 datetime=(no cast)FFFFFFFF81000000 datetime=820000 datetime=8300 datetime=84</code></pre> Summary: integer literals will be interpreted as signed unless explicitly written as unsigned. Since C99, integer conversions will not lose bits, but might loose signed-ness. samtal on "uint64 polarity problem" http://forums.leaflabs.com/topic.php?id=1524#post-10096 Thu, 05 Apr 2012 15:23:04 +0000 samtal 10096@http://forums.leaflabs.com/ gbulmer, I run your code on IDE12, Maple-r5. Changed the print to HEX. I did not add the ULL, and got good result as follows: datetime=1400000000 datetime=1401000000 datetime=1401020000 datetime=1401020300 datetime=1401020304 When I changed the array inputs to 0x80, 0x81, 0x82, 0x83, 0x84 (msb=1), I got the same error I had: (The second element is smaller than the first due to negative second element) datetime=8000000000 datetime=7F81000000 datetime=7F81820000 datetime=7F81828300 datetime=7F81828384 I then added the ULL to the first two elements, then it was OK: datetime=8000000000 datetime=8081000000 datetime=8081820000 datetime=8081828300 datetime=8081828384 As I wrote, although is seems to work, I do not understand why the array byte makes the difference rather than only the multiplier sign. thanks and let me know what your conclusions are. samtal gbulmer on "uint64 polarity problem" http://forums.leaflabs.com/topic.php?id=1524#post-10095 Thu, 05 Apr 2012 14:11:40 +0000 gbulmer 10095@http://forums.leaflabs.com/ samtal - When you get time, would you run my program with your initialised values in <code>in_command[]</code>, then post that program and output? It might indicate a bug in <code>Serial1.println(unsigned long long)</code> samtal on "uint64 polarity problem" http://forums.leaflabs.com/topic.php?id=1524#post-10094 Thu, 05 Apr 2012 13:45:56 +0000 samtal 10094@http://forums.leaflabs.com/ Thanks gbulmer for a very very thorough and full reply. The problem was solved by adding the ULL. I simply overlooked the fact that the compiler relates to a constant as a signed integer, unless casted. This also explains why I did not get the same problem with the small multipliers. It still does not explain why this depended on the array byte's sign, which is supposed to be unsigned. I also changed the array byte type to uint8 which is 100% unsigned, but still, the outcome depended on the multiplier, unless casted by ULL. I run all my test on the actual Maple5 board. I can now continue my work (hopefully) until the next time... Thanks again. samtal gbulmer on "uint64 polarity problem" http://forums.leaflabs.com/topic.php?id=1524#post-10092 Thu, 05 Apr 2012 12:30:17 +0000 gbulmer 10092@http://forums.leaflabs.com/ samtal - the value in the pastebin code is 0x100000000 which is too big to fit into an unsigned int. By default, unsigned values are unsigned int, so the literal value should be stored as 0. The top 1 is trimmed off. (The compiler should give a warning, but I haven't checked). Try writing it as 0x100000000ULL I am not yet convinced that is enough to explain everything, but it is a start. The example program is significantly bigger than a minimal example. It should initialise everything, so that there is no need to read input data; it is more robust to initialise than type input. There is no example of what is printed, so it is not possible to confirm the same thing is output when run on my machine. Here is something closer to a minimal example. It fully specifies the expected values by throwing away all of the input reading. You should initialise <code>in_command</code> with the actual values used, and republish. Then we can be confident we are testing the same thing. <pre><code>#include "stdlib.h" byte op_code; byte in_command[20] = { 0x14, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0x10, 0x11, 0x12, 0x13 }; byte in_command_length; uint64 datetime; void setup() { Serial1.begin(38400); } void loop() { sort_inputs(); delay(1000); } void sort_inputs() { //Reconstruct the datetime variable by combining the 5 bytes as uint64 Serial1.println(); datetime = in_command[0] * 0x100000000; /* ERROR should be 0x100000000ULL */ Serial1.print("datetime="); Serial1.println(datetime); datetime += in_command[1] * 0x1000000; Serial1.print("datetime="); Serial1.println(datetime); datetime += in_command[2] * 0x10000; Serial1.print("datetime="); Serial1.println(datetime); datetime += in_command[3] * 0x100; Serial1.print("datetime="); Serial1.println(datetime); datetime += in_command[4]; Serial1.print("datetime="); Serial1.println(datetime); }</code></pre> I've run this on my Mac, which also defaults to 32-bit ints, and the results are not negative. It will not compile because 0x100000000 is too big to fit into an int. When that is changed to 0x100000000ULL The output is: <pre><code>datetime=85899345920 datetime=85916123136 datetime=85916254208 datetime=85916254976 datetime=85916254980</code></pre> Which looks plausible. I changed it to output in hex, and it is (still on my Mac): <pre><code>datetime=1400000000 datetime=1401000000 datetime=1401020000 datetime=1401020300 datetime=1401020304</code></pre> Which looks correct to me. There are only 10 hex digits, which is 40-bits, so nothing is negative. Unless the number is printed as 64bits long (which it is not) and the MSB is 1 (which would make the number hex 8 to F), then it just means that bit is set. Please paste your output so that we can inspect it. (Preferably print hex as that is easier to check.) One other possibility: <code>Serial1.println(unsigned long long)</code> has a bug. samtal on "uint64 polarity problem" http://forums.leaflabs.com/topic.php?id=1524#post-10090 Thu, 05 Apr 2012 11:40:18 +0000 samtal 10090@http://forums.leaflabs.com/ Thanks juzza_s, I have built a minimum code that shows the problem. code at: <a href="http://pastebin.com/NF4jCuU8" rel="nofollow">http://pastebin.com/NF4jCuU8</a> What I am doing: I have built a maple real-time clock. To set the clock, I use a windows NT program that sends out via RS232 a reduced time stamp of 5 bytes (preceded by 02, length byte, ending with 03). Example of sent data : 02 05 1E 87 0F AF 22 03 My program reads the bytes, fills in a byte array and then builds a time stamp of one uint64 value. The attached code, when run, shows that if, for example, I will make the second data byte > 0x80, the total value will be reduced because the byte conversion will be negative. As this has occupies too much of my time, I think if there is no solution I will find an easier way to set the RTC time, but I still want to understand what is wrong, thus any help will be appreciated, especially if there is a solution. samtal. juzza_s on "uint64 polarity problem" http://forums.leaflabs.com/topic.php?id=1524#post-10088 Thu, 05 Apr 2012 10:20:49 +0000 juzza_s 10088@http://forums.leaflabs.com/ Can I see some of the code? To have a look at how you're casting and printing? samtal on "uint64 polarity problem" http://forums.leaflabs.com/topic.php?id=1524#post-10087 Thu, 05 Apr 2012 06:50:43 +0000 samtal 10087@http://forums.leaflabs.com/ Thanks, but the problem is not only in printing, it is also in math calculations, where it counts as a negative rather than positive. I have also tried casting to uint64 each appearance of the number, but for no avail. Any idea? juzza_s on "uint64 polarity problem" http://forums.leaflabs.com/topic.php?id=1524#post-10086 Thu, 05 Apr 2012 05:58:44 +0000 juzza_s 10086@http://forums.leaflabs.com/ How are you displaying the value? You could be printing the value out as a signed value where the MSB would produce a negative when printed. samtal on "uint64 polarity problem" http://forums.leaflabs.com/topic.php?id=1524#post-10085 Thu, 05 Apr 2012 04:58:11 +0000 samtal 10085@http://forums.leaflabs.com/ Hi, I have a number defined as type uint64 (or unsigned long long). The number is built out of a byte multiplied by 0x1000000. Yet, whenever the byte > 0x80 which makes the resulting msb equals 1, I get a negative value. uint64 is not supposed to go negative. It is a simple 64 bits positive integer. Did anyone encounter similar situation, or have a reasonable explanation to this? Thanks samtal