Category Archives: Common

Intel Threading Building Blocks Scalable Allocator & Valgrind

Hi,

if you ever use the TBB (Intel Threading Building Blocks) allocator to overwrite malloc/free/* and want to use Valgrind for leak checking and fail, here is the simple trick to get it working:

valgrind --soname-synonyms=somalloc=\*tbbmalloc\* <your-application-here>

I missed that hint in the Valgrind documentation for my first tries ;)

Btw., the scalable allocator from TBB is a really BIG improvement over the normal system allocator on current Linux & Windows machines if you allocate mostly fixed size small object, like what happens if you heavily use STL data structures like std::map/set that are implemented as trees and you have other stuff like DOM/AST like data structures (even in the single threaded case, for which it just saves a LOT of memory).

Ramblings about compilers…

In my job I work on binary and source level analysis software running on Linux and Windows. One of my tasks is to maintain the build farm and compile environment, therefore I am responsible for keeping care of the compilers and libraries we use (like the beloved Qt, congratulations for the nice 5.1 release, btw.).

For Linux, we normally use the GNU C/C++ compiler. That works out very well since years. We have hit compiler bugs only a few times and there was always already some patch-release out in the wild that fixed our issues. In addition the GCC developers brought a constant stream of improvements in the area of C and C++ standard compliance. I am more than happy with the GCC quality.

I did some brief experiments with LLVM/Clang two years ago and was not that impressed in the stability of the C++ support (especially of the fresh libc++ STL implementation) but even for a such young project, it was fast as light to get issues fixed, like a crash in the unordered containers (see Bug 10278, only some hours from report to fix, even if it was only a trivial size variable used uninitialized in the libc++ implementation). I guess today LLVM would perform ok to compile our software (including Qt), still I will stick to GCC, given we use some libraries that normally only get love for that compiler on Linux (at least at the moment).

But now, after my experiences with open source compilers, lets take a look at the wonderful world of Windows.

I know for Windows GCC variants are available (like the 32 bit MinGW-builds shipped with some Qt library variant) and some experimental work on some LLVM able to compile applications using Windows SDK headers is ongoing. Still, the most common way on Windows is to use the Visual Studio compiler and given we always have some libraries of 3rd-parties around that we need to use and that are Visual Studio compiled the Microsoft compiler is more or less unavoidable.

Now, lets take a look of this fabulous compiler you get there. At the moment two major Version are of interest, the 2010 and 2012 variant of Visual Studio.

As I started to evaluate Visual Studio 2010 in 2010, I was immediately hit by its nice 64 bit optimizer bug. Not a single Qt application would start up without immediate segfault… More about this here and even in the Qt bugtracker, especially the answers from the support are interesting, it seems to be forbidden to return a object per value from a function ;)

From report to “hotfix” it has taken some months, that means with the release version of 2010, you couldn’t build any optimized x64 application using Qt (and I guess most other evil libraries using object on the stack that got misaligned by the optimizer) for some months.

Afterwards I was that adventurous to install the SP1 to get the included further compiler fixes. Unfortunately that meant I got all my x64 compilers deleted, but no problem, just close to a month later (until which you need to reinstall your Visual Studio/SDK without SP1 and reapply you hotfix) you got again some fix that reinstalls the compilers that SP1 will remove. Really, you will need to install the 2010 + SP1 + then a fix that restores the compilers SP1 removes if you want to have a working VS 2010 Express for x64 (or stay without SP1 and use the above hotfix for the x64 compiler).

For Visual Studio 2012, which I started to eval months ago, you will get the same story, but reversed. Now we have some nice and nasty x86 optimizer bug. That bug is reported since long see here, it leads to problems for software that use libicu, see the matching icu bug. But given the simple code pattern that can make it occur, it may hit other parts, too. Until now, no fix, only some “will be fixed in next version”…

For me that means, I need to stick with 2010 for x86 and with 2012 for x64, given I want to use the pre-build Qt binaries and not to dive into the “compile qt on windows” fun once more. I would like 2012 for x86, too, to benefit from the C++11 feature improvements and to not have to support two different compiler variants, but I guess that will have to wait, seems its not that easy to build some working x86 compiler that can do optimizations.

Therefore, really, +1 to the compiler developers of the GCC and LLVM project. Its amazing what they have archived!
And +1 for the improvements in the C99 and C++ area in Visual Studio, too. Still I really would be more than happy to get a fix for the x86 optimizer issue in VS 2012. Even no report that it is really fixed in 2013 until now, but I am not sure that was really tried, I didn’t try it myself because I am lazy and must blog instead ;=)

A rich python console and more in Kate editor

I have done some improvements in the plugins: python_console_ipython, python_autocomplete, python_utils, js_utils, xml_pretty and django_utils. These plugins I added a month and a half ago (except python_console_ipython) to the kate repository. I have done two improvements and a new feature:

  • Now they work with Python2 and Python3 (except python_autocomplete, this only works with Python2, due pysmell dependence)
  • Now they have a configuration page (except python_autocomplete, this should not have configuration parameters)

Page Config Plugins

The new feature is the integration of the python_autocomplete and python_console_ipython plugins with the project plugin. The behaviour of these plugins depends of the loaded project. That is to say, the python_autocomplete plugin autocompletes with our project modules. Currently the kate community is working to add a new python autocomplete  using jedi (this will work with Python2 and Python3).

Python Autocomplete (with our modules)

And the python_console_ipython plugin has something that we can name the project console. A ipython shell with the python path of our project and with the environments variables that the project plugin needs.

IPython Console (converted in a django shell)

To use this we need to add in the project file (.kateproject) a new attribute named “python”, with this structure:

 {
  "name": "MyProject",
  "files": [ { "git": 1 } ],
  "python": {
        "extraPath": ["/python/path/1",
                      "/python/path/2",
                      "/python/path/n"
        ],
        "environs": {"key": "value"},
        "version": "3.2"
    }
 }

I am a django developer, as we say in Django we can come to have a django shell (python manage.py shell) integrated in our editor. But this feature is a generic feature not tied to Django. This feature can be used by other Python developers. I have added only a specific feature to the django developers. If we set in the project file (.kateproject) the attribute projectType with the value Django, the ipython shell automatically loads the models and the settings like a shell_plus does.

  {
   "name": "MyProject",
   "files": [ { "hg": 1 } ],
   "python": {
        "extraPath": ["/python/path/1",
                      "/python/path/2",
                      "/python/path/n"
        ],
        "environs": {"DJANGO_SETTINGS_MODULE": "myproject.settings"},
        "version": "2.7.3",
        "projectType": "django"
    }
  }

I hope you like it :-)

New Text Folding in kate.git master

In the kate.git master branch the text folding is now new and shiny.

In addition to be faster and less memory hungry (no folding tree is around if you fold nothing, it is only created on-demand exactly for the folded regions), the new code is less complex and smaller (and hopefully better documented + unit tested, it actually has a test for most internal operations).

There is actually now a clean separation, the folding does not mix with highlighting and can be used without it, too.

This allows for a better integration of indentation based foldings like in Python (and already fixes the most ugly miss-placements of folding regions there) and empowers the user to fold arbitrary regions manually.

This can be done at the moment with two kate commands (for the F7 command line):

  • “fold” will create persistent fold for the current selection, this fold will stay until next reload of document (will implement some dfold command later, to delete it)
  • “tfold” will create temporary fold for the current selection, this fold will disappear again on unfolding

This should allow better vi mode integration, too, as vi is able to fold arbitrary regions, too. (There is an internal API to fold any KTextEditor::Range you pass in, which should allow to fold by regex and other ways, too, if somebody steps up to implement it.)

If you find glitches, please report them as bugs and please attach some file to reproduce, I am very interested to get this all in even better shape for KDE 4.11, be folding crashs a thing of the past.

P.S.

For all people not using folding at all, be happy, it no longer will crash you (and me), as it is a nop if no folding ranges are created by the user and some extreme cases got a speedup of 1000% because of less overhead, like VERY large C++ sources ;)