Floating Point precision

Hi,

i am reading the GPS Longitude and Latitude data with 7 Digit precision. data will be received as INT32. from INT32 i am trying to convert float. but i can not get 7 Digit precision. it is affecting my accuracy of data. but i am able to achieve this accuracy in the PC with same INT32 data.

please find my sample code.

//****************** SAMPLE CODE ***********************
//Converting 4 bytes INT32 to float
float extractFloat(unsigned char ub)
{
INT32 val = 0;
val = (INT32)ub[0] << 8 * 0;
val |= (INT32)ub[1] << 8 * 1;
val |= (INT32)ub[2] << 8 * 2;
val |= (INT32)ub[3] << 8 * 3;
float fVal=(float)val/1000000;
return (fVal);
}

float rdLat= extractFloat(latByte);
//*******************************

kindly suggest to solve this issue.

Hi @radkrish,

I apologize, the Toradex community we only take Software and Hardware questions about Toradex products themselves, not specifically about application code from our customers.

Maybe other customers can help you with this question on this community, or you can search other forums.

Best Regards,
Hiago.

You are encountering precision issues because the float type in C can only represent a limited range of decimal places. To achieve 7 digit precision, you should consider using a data type that supports higher precision, such as double.

Here’s a revised version of the function that uses double:

double extractDouble(unsigned char ub[4])
{
    INT32 val = 0;
    val = (INT32)ub[0] << 8 * 0;
    val |= (INT32)ub[1] << 8 * 1;
    val |= (INT32)ub[2] << 8 * 2;
    val |= (INT32)ub[3] << 8 * 3;
    double dVal = (double)val / 1000000.0;
    return (dVal);
}

double rdLat = extractDouble(latByte);

Note that you have changed the type of the val variable from float to double, and the constant 1000000 to 1000000.0 to indicate that it’s a floating point constant.