View Issue Details

IDProjectCategoryView StatusLast Update
0001852FreeCADBugpublic2017-02-10 10:50
ReportershoogenAssigned Toshoogen 
PriorityhighSeveritymajorReproducibilityalways
Status closedResolutionfixed 
PlatformOSPOSIXOS Version
Product Version0.15 
Target Version0.15Fixed in Version 
Summary0001852: The python interpreter does not recognize the posix locale settings
DescriptionThis currently breaks the usage of non-ascii characters in filenames for import and export handled in python modules.
Steps To Reproduce$ python -c "import sys;print sys.getfilesystemencoding()"
UTF-8
$ LANG=C python -c "import sys;print sys.getfilesystemencoding()"
ANSI_X3.4-1968
$ ./fc-build/bin/FreeCADCmd
[FreeCAD Console mode <Use Ctrl-D (i.e. EOF) to exit.>]
>>> import sys
>>> sys.getfilesystemencoding()
'ANSI_X3.4-1968'
>>> import locale
>>> locale.getdefaultlocale()
(None, None)
Tagsencoding, locale, sys.getfilesystemencoding, unicode

Relationships

related to 0002891 closedwmayer Sketcher Sketching impossible, Type.Error Exception 
related to 0001545 closed FreeCAD Folders with tick in the name cannot be opened 
related to 0002018 closedwmayer FreeCAD Running Freecad with specific locale causes freeze 
child of 0001027 closedshoogen FreeCAD UTF-8 encoded PyString objects are used instead of unicode objects to pass unicode data to the python interpreter 
child of 0001930 closedshoogen Drawing Export a page to an SVG file and UnicodeEncodeError 

Activities

shoogen

2014-12-08 14:01

developer   ~0005360

Last edited: 2014-12-08 14:02

View 2 revisions

http://www.mevislab.de/docs/current/MeVis/ThirdParty/Documentation/Publish/Python/library/locale.html
> When Python code uses the locale module to change the locale, this also affects the embedding application. If the embedding application doesn’t want this to happen, it should remove the _locale extension module (which does all the work) from the table of built-in modules in the config.c file, and make sure that the _locale module is not accessible as a shared library.

Great idea python ;)

shoogen

2014-12-09 16:21

developer   ~0005361

Last edited: 2014-12-09 16:22

View 2 revisions

https://github.com/5263/FreeCAD/commit/c85ddd166ef80e747719549922bb7e5ddffc58cd
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=559846

shoogen

2014-12-09 16:37

developer   ~0005362

Werner, could you please comment on the following patch

https://github.com/5263/FreeCAD/compare/dev-locale

diff --git a/src/Main/MainCmd.cpp b/src/Main/MainCmd.cpp
index 17cb6c1..b7c1d68 100644
--- a/src/Main/MainCmd.cpp
+++ b/src/Main/MainCmd.cpp
@@ -70,4 +70 @@ int main( int argc, char ** argv )
-#if defined(FC_OS_LINUX)
- putenv("LANG=C");
- putenv("LC_ALL=C");
-#else
+ setlocale(LC_ALL, "");
@@ -75 +71,0 @@ int main( int argc, char ** argv )
-#endif
diff --git a/src/Main/MainGui.cpp b/src/Main/MainGui.cpp
index 300084a..a0050c7 100644
--- a/src/Main/MainGui.cpp
+++ b/src/Main/MainGui.cpp
@@ -194,2 +194 @@ int main( int argc, char ** argv )
- putenv("LANG=C");
- putenv("LC_ALL=C");
+ putenv("LC_NUMERIC=C");
@@ -199,2 +198 @@ int main( int argc, char ** argv )
- putenv("LANG=C");
- putenv("LC_ALL=C");
+ putenv("LC_NUMERIC=C");

shoogen

2014-12-09 19:38

developer   ~0005363

Last edited: 2014-12-09 19:38

View 2 revisions

It seems OCCT fixed the problem in version 6.6.0
http://tracker.dev.opencascade.org/view.php?id=22898

wmayer

2014-12-11 10:54

administrator   ~0005365

I can confirm that "putenv("LC_NUMERIC=C");" is sufficient now to fix the OCC problem -- tested with OCC 6.7.1

shoogen

2014-12-15 10:03

developer   ~0005385

I see two possibilities.
1. Don't touch the locales before the 0.15 release
this will require a different workaround for different platforms.
2. Do tackle this problem before the 0.15 release.
this requires testing on all platforms before the release.

Thank you Werner for testing.
I don't understand why to use putenv (instead of setlocale) on linux. It would make sense on windows, because windows has no fork command.
If putenv is nessesary because of a setlocale used somewhere is (in QT for example) and if other parts of FreeCAD rely on LC_NUMERIC=C (like the xml writer and parser) then, my patch would be a bad idea.
However, the xml output seems to unaffected.

wmayer

2014-12-15 10:48

administrator   ~0005386

I go for possibility 1) as it's too dangerous to do this change for an imminent release.

Thinking further about why putenv works but setlocale fails I think putenv affects the locale settings for C and C++ while setlocale affects it only for C.

The C++ way to change the locale is to do:
std::locale::global(std::locale("C"))

but it should be checked if there is a way to change only numeric input/output

> However, the xml output seems to unaffected.
Here we explicitly change the locale settings of a concrete stream instance. Search for "imbue" in the sources.

wmayer

2014-12-15 11:05

administrator   ~0005387

Maybe just writing
std::locale::global(std::locale("C", std::locale::numeric));
does the trick.

See also:
http://stackoverflow.com/questions/12373341/does-stdlocaleglobal-make-affect-to-printf-function

shoogen

2014-12-16 19:04

developer   ~0005396

Last edited: 2014-12-16 21:05

View 3 revisions

git://github.com/5263/FreeCAD dev-locale-cpp

std::locale::global(std::locale(std::locale(""),std::locale("C"),std::locale::numeric));
seems to set the locale correctly (on linux).

On my system it has the "name"
LC_CTYPE=de_DE.UTF-8;LC_NUMERIC=C;LC_TIME=de_DE.UTF-8;LC_COLLATE=de_DE.UTF-8;LC_MONETARY=de_DE.UTF-8;LC_MESSAGES=de_DE.UTF-8;LC_PAPER=de_DE.UTF-8;LC_NAME=de_DE.UTF-8;LC_ADDRESS=de_DE.UTF-8;LC_TELEPHONE=de_DE.UTF-8;LC_MEASUREMENT=de_DE.UTF-8;LC_IDENTIFICATION=de_DE.UTF-8
and having a name allows to set the C locale as well.

shoogen

2014-12-16 21:08

developer   ~0005399

$ git grep -l strtod
src/3rdParty/CxImage/png/CHANGES
src/3rdParty/CxImage/png/pngrutil.c
src/3rdParty/salomesmesh/src/SMESH/SMESH_Pattern.cpp

$ git grep -l atof
src/3rdParty/CxImage/png/libpng-1.2.24.txt
src/3rdParty/salomesmesh/inc/UNV_Utilities.hxx
src/Base/Parameter.cpp
src/Base/Quantity.cpp
src/Base/Reader.cpp
src/Mod/Cam/App/ConvertDyna.cpp
src/Mod/Fem/App/AppFemPy.cpp
src/Mod/Fem/App/FemMesh.cpp
src/Mod/Mesh/App/Core/MeshIO.cpp
src/Mod/Points/App/PointsAlgos.cpp
src/Mod/Robot/App/Robot6Axis.cpp

$ git grep -l imbue
src/Base/Reader.cpp
src/Base/Writer.cpp

> Search for "imbue" in the sources.
Hm, I'm not amused.

shoogen

2014-12-17 18:17

developer   ~0005401

with the latest change i can't open exiting documents. ;(

shoogen

2014-12-17 19:13

developer   ~0005402

Last edited: 2014-12-17 19:31

View 2 revisions

Qt ;(

wmayer

2014-12-17 19:41

administrator   ~0005403

Which change?

shoogen

2014-12-17 19:51

developer   ~0005404

mine git://github.com/5263/FreeCAD dev-locale-cpp

http://www.qtcentre.org/threads/53127-Override-locale-with-QSystemLocale

shoogen

2014-12-18 07:00

developer   ~0005405

Ok, now i understand the need for the putenv on unixoid OSes.

The damn Qt uses qgetenv to setup the locale.
https://qt.gitorious.org/qt/qtbase/source/e0a8b5ce88bc50440dcec2fe3a86d83e2a7dc7b0:src/corelib/tools/qlocale_unix.cpp#L69

shoogen

2014-12-21 12:08

developer   ~0005414

If you don't want to modify the locale we have to use old method of passing utf8 byte strings in the 0.15 release (at least for unixoid platforms)

git://github.com/5263/FreeCAD ugly_workaround_for_bug_1852

wmayer

2014-12-21 12:28

administrator   ~0005415

I haven't checked yet the branch dev-locale-cpp. Does this fix the issue reliably?

When I said that I would postpone the fix for 0.16 then I meant to leave bug in for 0.15. But since you want to have a solution already for 0.15 then I think it's better to apply the proper solution instead of reverting any commits because this needs a lot of testing anyway and is as your branch name says really "ugly".

shoogen

2014-12-21 16:52

developer   ~0005420

I updated dev-locale-cpp
I looked promissing for normal spinboxes. But I got problems with quantity spinboxes. These persisted when i did an icremental build with the version of the master branch. (I'm waiting for a clean build of master before i continue testing)

And finally I have to use the proper namespace in c++ ;)

wmayer

2014-12-21 17:17

administrator   ~0005422

Write this:
class AppExport DecimalSystemLocale : public QSystemLocale

IMO, in case this class is needed it would make more sense to move this to "Base". In this case the export macro must be "BaseExport".

shoogen

2014-12-21 17:44

developer   ~0005423

To me it seems that the locale works in QT, but now the Quantity Parser is broken.

wmayer

2014-12-21 18:19

administrator   ~0005424

One thing I still do not understand is why something for Qt needs to be fixed.

The story is:
1. All the trouble started because OCC doesn't handle numeric input properly.
2. That's why setting LANG and LC_ALL to C solved this issue.
3. Since this affects Qt "QLocale::system();" is called very first to set up the locale stuff for Qt and QFile::setEncodingFunction/QFile::setDecodingFunction is used to utf-8 handling
4. However, setting LC_ALL and LANG affects Python and since OCC now improved the handling for numeric input setting LC_NUMERIC seems to be sufficient.

Now, does changing LC_NUMERIC affect Qt in a certain way?

Btw, for MacOSX nothing needs to be done because with current master "import sys;print sys.getfilesystemencoding()" returns UTF-8 and applying the dev-locale-cpp branch makes the applicationcrash immediately.

shoogen

2014-12-21 18:50

developer   ~0005425

Qt has the ability to set the deflection Property to 0 when loading an existing FreeCAD file with an unexpected decimal seperator. And OCCT starts mesheshing with 0 deflection. ;)
1. That should be fixed by 6.6.0
2. That did not prevent Jürgen from using atof in the QuantityParser.
3. That made sense for Qt < 4.2. But I would consider this as not beautiful. And using a different locale in Qt, c++ and the python interpreter is messy. When I'm touching the code anyway I try, to make it more beautiful. If that's not possible we have to live with that. But Qt allows to set the decimal seperator independenly.
4. I agree for Linux. But I don't think Qt will respect it on windows.

shoogen

2014-12-21 19:07

developer   ~0005426

It seems that std::locale::global(std::locale(std::locale(""),std::locale("C"), std::locale::numeric));
is not enough for the atof() in src/Base/Quantity.cpp to accept a '.' as decimal seperator on linux with my locale

shoogen

2014-12-21 20:07

developer   ~0005428

neither is a std::setlocale()

wmayer

2014-12-21 20:49

administrator   ~0005429

> Qt has the ability to set the deflection Property to 0 when loading an existing FreeCAD file with an unexpected decimal seperator.
How do you get a FreeCAD project with an unexpected decimal separator (i.e. another one than ".")?

> That did not prevent Jürgen from using atof in the QuantityParser.
If this causes any trouble we can easily use C++ streams instead. There you can set on a single instance (not globally) use the C locale.

> And using a different locale in Qt, c++ and the python interpreter is messy.
I disagree. For low-level handling (e.g file reading, writing, ...) I would like to use a locale that is platform independent, i.e. the "C" locale.

For user input in the GUI (i.e. Qt) I would like to use the system locale because it would be extremely annoying when in another application I use the system locale and suddenly in FreeCAD it would be the C locale.

shoogen

2014-12-21 23:01

developer   ~0005430

I haven't so far i only opened documents when qt expected a comma.

Yes, i have done it for the quantity parser in my branch. But atof is also used in third party libs like CxImage.

I partly agree. It would make sense if we devide the functionalty beetween frontend and backend. But we can't draw the line between c, c++, qt and python. We use qstrings in the properties handling and python to interface fileformats and gui.
> For user input in the GUI (i.e. Qt) I would like to use the system locale because it would be extremely annoying when in another application I use the system locale and suddenly in FreeCAD it would be the C locale.
Thats a good idea. But this means that you would need to use a locale object when handling properties.

shoogen

2015-01-24 18:32

developer   ~0005714

Last edited: 2015-01-24 19:01

View 2 revisions

Ok, its been an interesting but disappointing excursus. Lets just use putenv in the manner it is currently used, but for LC_NUMMERIC. Thus making the minimal possible change.
I filed a pull-request. (without the unit test, because it won't test the behavior correctly on windows)

wmayer

2017-02-10 10:18

administrator   ~0008261

This explains all the trouble we had related to locales: https://forum.freecadweb.org/viewtopic.php?f=3&t=20537#p158962
So, it's caused by Qt by calling setlocale(LC_ALL,"") which then applies the system-wide locale settings. So, on systems which set LC_ALL (e.g. Debian) this leads to problems especially if the decimal separator is a comma.

I played a bit with that and made the observation that the decimal separator of floating point numbers in the user.cfg will become a comma and then it can happen that the next time it cannot handle the conversion from text to float and sets it to 0.0.

Especially for the tessellation of shapes this leads to massive problems because then the deviation is 0.0 instead of a sensible value and causes the tessellation of curved shapes to take forever.

Related Changesets

FreeCAD: master f32caef4

2014-12-09 17:24:34

shoogen

Details Diff
respect locale environment except for LC_NUMERIC

fixes 0001852
Affected Issues
0001545, 0001852
mod - src/Main/MainCmd.cpp Diff File
mod - src/Main/MainGui.cpp Diff File

Issue History

Date Modified Username Field Change
2014-12-08 13:46 shoogen New Issue
2014-12-08 13:48 shoogen Steps to Reproduce Updated View Revisions
2014-12-08 13:49 shoogen Steps to Reproduce Updated View Revisions
2014-12-08 14:01 shoogen Note Added: 0005360
2014-12-08 14:02 shoogen Note Edited: 0005360 View Revisions
2014-12-08 14:04 shoogen OS Linux => POSIX
2014-12-08 14:04 shoogen Summary The python interpreter does not recognize the locale settings => The python interpreter does not recognize the posix locale settings
2014-12-08 14:22 shoogen Tag Attached: sys.getfilesystemencoding
2014-12-08 14:22 shoogen Tag Attached: unicode
2014-12-08 14:23 shoogen Tag Attached: encoding
2014-12-09 16:21 shoogen Note Added: 0005361
2014-12-09 16:22 shoogen Note Edited: 0005361 View Revisions
2014-12-09 16:37 shoogen Note Added: 0005362
2014-12-09 16:37 shoogen Assigned To => wmayer
2014-12-09 16:37 shoogen Status new => assigned
2014-12-09 19:38 shoogen Note Added: 0005363
2014-12-09 19:38 shoogen Note Edited: 0005363 View Revisions
2014-12-11 10:54 wmayer Note Added: 0005365
2014-12-15 10:03 shoogen Note Added: 0005385
2014-12-15 10:48 wmayer Note Added: 0005386
2014-12-15 11:05 wmayer Note Added: 0005387
2014-12-16 13:34 shoogen Relationship added child of 0001027
2014-12-16 19:04 shoogen Note Added: 0005396
2014-12-16 19:04 shoogen Note Edited: 0005396 View Revisions
2014-12-16 21:05 shoogen Note Edited: 0005396 View Revisions
2014-12-16 21:08 shoogen Note Added: 0005399
2014-12-17 18:17 shoogen Note Added: 0005401
2014-12-17 19:13 shoogen Note Added: 0005402
2014-12-17 19:31 shoogen Note Edited: 0005402 View Revisions
2014-12-17 19:41 wmayer Note Added: 0005403
2014-12-17 19:51 shoogen Note Added: 0005404
2014-12-18 07:00 shoogen Note Added: 0005405
2014-12-21 12:08 shoogen Note Added: 0005414
2014-12-21 12:28 wmayer Note Added: 0005415
2014-12-21 16:52 shoogen Note Added: 0005420
2014-12-21 17:17 wmayer Note Added: 0005422
2014-12-21 17:44 shoogen Note Added: 0005423
2014-12-21 18:19 wmayer Note Added: 0005424
2014-12-21 18:50 shoogen Note Added: 0005425
2014-12-21 19:07 shoogen Note Added: 0005426
2014-12-21 20:07 shoogen Note Added: 0005428
2014-12-21 20:49 wmayer Note Added: 0005429
2014-12-21 23:01 shoogen Note Added: 0005430
2015-01-07 11:38 wmayer Relationship added related to 0001545
2015-01-24 10:17 shoogen Relationship added child of 0001930
2015-01-24 18:32 shoogen Note Added: 0005714
2015-01-24 19:01 shoogen Note Edited: 0005714 View Revisions
2015-01-24 22:39 shoogen Changeset attached => FreeCAD Master master f32caef4
2015-01-24 22:39 shoogen Assigned To wmayer => shoogen
2015-01-24 22:39 shoogen Status assigned => closed
2015-01-24 22:39 shoogen Resolution open => fixed
2015-03-21 23:50 shoogen Relationship added related to 0002018
2017-02-10 10:04 wmayer Relationship added related to 0002891
2017-02-10 10:18 wmayer Note Added: 0008261
2017-02-10 10:50 Kunda1 Tag Attached: locale