Tag Archives: planet

Kate Vim Progress

Hi all,

Just an eye-glazingly brief blog post about some of the stuff I’ve been working on in the Kate Vim emulation mode since my last blog. Once more, I’ll mostly be dumping them all as bullet points, Gräßlin-Style™, with some longer sections on the more important changes, but this time around I’ve added some nifty animated gifs which you can skip to should the full text prove too harrowing!

In roughly chronological order, I:

    • Fixed regression with gv (“re-select last selection”).
    • Laid the groundwork for improved behaviour when repeating a change that involved an auto-completion, for now just fixing a bug where if auto-completion was automatically invoked and we repeat the change via “.”, only a portion of what we typed gets repeated.
    • As promised, implemented “yank highlighting”, which shows the portion of the text that you just yanked; very handy if you do complex yanks like e.g “y2f)”:

“Yank Highlighting”, 444KB, Click to Play.
 

  • Fixed a bug with keypad handling.
  • Made ctrl-p and ctrl-n wrap around in completion lists.

… and a biggie:

Emulated Vim Command Bar

There are quite a lot of issues with the interaction between Vim mode and Kate’s own Find dialog:

  • The cursor is not positioned at the beginning of the match.
  • The full match is not highlighted.
  • “?” does not search backwards(!)
  • The Find dialog cannot be used in mappings.
  • The Find dialog could not be used to either record or playback macros.
  • Aborts Visual Mode when summoned.

etc. On top of that, we miss out on some nice features of Vim’s search bars, such as ctrl-r for insertion of register contents; ctrl-w for deleting previous word; ctrl-h for delete; ctrl-b/e for jumping to the beginning or end (more conveniently located, IMO, than HOME or END); plus the Qt regex syntax is different from Vim’s. In light of this, I decided to add a new KateViewBarWidget called “KateViEmulatedCommandBar”:

“Emulated Command Bar Search”, 697KB, Click to Play.
This behaves much more like Vim, and has the features and bugfixes mentioned above; it even does a reasonable job (in current master; it missed 4.11) of translating Vim regexes to Qt ones e.g. with Vim, a “+” is a literal “+”, not an “at least one of the preceding” marker; with Qt it is the reverse. The emulated command bar performs this, and other, conversions for you.

The searches carried out can be repeated correctly via “n” and “N”. Search history can be navigated via ctrl-p and ctrl-n, and searches “external” to the emulated command bar (e.g. those carried out by “*” and “#”) are added to this history. “Smart Case” is used for specifying case-sensitivity – if the search term has any capitals, then the search is assumed to be case-sensitive; else it is case-insensitive. More on the emulated command bar later! Back to the micro-fixes:

  • Distinguished recursive vs non-recursive mappings; inspired by this bug report which highlighted a nasty crashing problem if we tried to map j -> gj and k -> gk …
  • … and relatedly, completely reworked gk and gj, fixing a bunch of bugs in the process.
  • Yet more fixes to the “inner block” text object, which is much more complex than you would credit!
  • A bunch of fixes and patches to ctrl-a/ctrl-x (“increment/ decrement number under-or-to-the-right-of cursor”) culminating in me just writing the whole thing from scratch :)
  • Indented paste (]p, ]P etc).

Everything above made it for 4.11, I think. The Emulated Command Bar was disabled by default but can be enabled by setting the hidden config option “Vi Input Mode Emulate Command Bar” to “true”. Continuing:

  • Made “/” and “?”, when using the Emulated Command Bar, usable as (countable) motions: so one can, for example, do d3/foo<enter> to delete up to just before the third occurence of “foo”.
  • Allowed one to set the whole selection to a given text object even after we’ve formed a selection in Visual Mode e.g. you can start visual mode, navigate to inside a pair of brackets, and do “ib” to re-set the current selection to just the inside of that pair of brackets. I think this worked once upon a time and I broke it, so this is more of a regression fix than a new feature :/
  • Made the emulated command bar usable for Commands (like the “F7″ command box in normal Kate) as well as for forwards and backwards searches when summoned via “:”. Note that since the emulated command bar is usable in mappings, we can now incorporate the execution of your Javascript script(s) into the Vim key mapping system, which could end up being a very powerful addition.
  • Allowed mappings for InsertMode and VisualMode as well as for NormalMode. Oddly, this was accomplished mainly as a side-effect of a fix for a tiny bug with mappings. There is no gui for this, yet, but I’ve implemented imap, vmap, inoremap, etc so they can be set from the command line. I’ve also added the <space> special key for use in mappings, so that you can do e.g. imap a<space>b c<space>d to specify a mapping containing spaces from the command line.
  • Implemented “g~” (“toggle case” with motion).
  • A bunch more fixes for t/T/f/F to bring them more in line with Vim’s behaviour, especially when there aren’t enough characters to find.
  • Made ‘r’ (“replace single character”) counted, so it can actually replace several characters.
  • Made “\C” (“case-sensitivity marker”) work in the emulated command-bar search: this marker is used to specify that the current search should be case-sensitive (handy if you want a case-sensitive search for an all-lower-case search term) after which it is stripped out and ignored completely.
  • Made “//” and “??” (“repeat last search forward/ backwards”) work in the emulated command bar, and also allow the “/e” marker (“place the cursor at the end of the match”), although this latter is a little buggy at the moment.
  • Added fix for the usage of Alt-0, Alt-1 etc shortcuts in normal mode; this is for use with KDevelop’s automatic refactoring support which relies on these shorcuts (e.g. “Rename [variable] to [newvariable] Alt-0 | Hide Alt-1″, etc).
  • A bunch of fixes for J (“join lines”), including making it work in Visual Mode when the selection runs from top to bottom; not crashing if all lines are empty; fixing countedness (it would join one too many lines); cursor positioning after the join; etc.
  • Fixed occasional bug where counted inserts e.g. 5ixyz<esc> would start each repetition on a new line.
  • Implemented iW and aW (“inner/ a WORD”) text objects.

… and then on to:

Making the Emulated Command Bar Suitable for Default

The emulated command bar has a lot of advantages over the default Find dialog and command-line for Kate Vim users, but there are drawbacks, too: firstly, it can only perform regex searches and not literal searches, which is awkward when searching for complex expressions pasted from your document that may contain characters that have a special meaning when interpreted as regex’s (e.g. “*”) or which have special meaning in a Vim search bar (e.g. “/” when searching forward, and “?” when searching backwards). The necessity of going through the search term and escaping all such characters is a big strike against its use.

Secondly, Vim’s Find/ Replace dialog has most of the same flaws, when used in Vim mode, as its Find dialog has: in fact, incremental Find & Replace doesn’t work at all with Vim mode(!), and the emulated command bar is of no use here.

Since I’d really like to make the emulated command bar the default (and possibly only) option for use in Vim mode, I really needed to sort out a solution for these two dealbreakers.

For the first, there are basically two options: one is to add some kind of flag that says “look for literal matches for this string; don’t treat it as a regex!”; the other is to make it easy to perform the escaping of special characters. I eventually opted for the second, as it allowed one to treat portions of an expression as a regex if one desired, plus it allowed one to always use the same delimiters in a sed-replace expression (i.e. :s/find/replace/gci) without having to worry about whether the “find” term contained “/”‘s or not. So I added a shortcut (ctrl-g) to the emulated command bar that behaved much like ctrl-r (“insert contents of the following register”) except that it escapes the inserted register contents in such a way that a regex search for the contents would behave the same as a literal search for the contents. The gif below highlights the problem and the solution; the “:) :) :)” bit used ctrl-g to insert the (correctly escaped) yanked expression:

“Escape Expression for Regex”, 483KB, Click to Play.
For the second problem, improving the good old sed-replace seemed the obvious choice, so I added some code in the emulated command bar to translate the regex in the “find” term from Vim style to Qt style. Kate’s own Find/ Replace dialog maintains separate histories for both the “find” and “replace” terms, so I needed to follow suit if I wanted the emulated command bar to be a credible replacement: if the cursor is placed over the “find” or “replace” terms in a sed-replace expression, pressing ctrl-p will summon the relevant history:

“Sed Search/ Replace History”, 259KB, Click to Play.
Kate’s own Find/ Replace dialog makes it easy to focus on the “find” and “replace” fields, so I also added shortcuts “ctrl-d” and “ctrl-f” to the emulated command bar that clears and places the cursor at the “find” and “replace” portions of a sed-replace expression, respectively. As an extra bonus, ctrl-space can be used anywhere in the emulated command bar to complete words from the document, which is handy when filling out your find/ replace terms.

Still missing, though, were interactive search and replace; multi-line search; usage of \U and \L in the “replace” expression to make regex captures upper/ lower cased; etc. Multi-line search and \U and \L codes are available in a class called KateRegExpSearch, so I initially pulled out the guts of the SedReplace code and replaced it with that, and then made the whole thing interactive:

“Sed Multi-Line”, 399KB, Click to Play.
(in the gif above, I constructed my regex with the standard “/” search bar so I could get immediate feedback as I am a giant regex baby who still needs his stabilisers; this final regex was then added to the “find” history, which I then added to the sed-replace search expression with a ctrl-d + ctrl-p).

The emulated command bar now seemed to me to be a decent alternative to Kate’s own Find/ Find & Replace dialogs, so with that out of the way, I then moved on to …

Macro Support!

The lack of macro support is what got me into Kate Vim hacking in the first place, so it seems weird that it ended up being almost the last thing I tackled. The reason, of course, is that macro support would be a bit crippled if we couldn’t run searches or execute commands in our macros, so I really needed to finish the emulated command bar first. It ended up being quite easy in the end; the main difficulty was in interacting with mappings, especially the situation where a mapping triggered a macro which in turn needed to execute another mapping! Anyway, here’s a small, real-life example of a Kate Vim macro in action:

“Macro”, 300KB, Click to Play.
One difficulty with macros comes from recording and replaying auto-completions. In Vim itself, macros are “dumb” sequence of keypresses that are recorded and replayed verbatim, including those that summon, navigate and execute lists of completions. In Kate Vim, though, this approach doesn’t work so well, partly because the available completions at any time are less predictable (e.g. we could have a function appear in the list of completions when we record the macro, but not when we replay it, due to e.g. problems with the C++ parser etc) and also because, in KDevelop at least, the added completions are context-sensitive: for example, a completion of a function will add its own opening bracket if there is not one immediately after the cursor position, but will re-use the existing one otherwise. The “repeat last change” runs into a similar problem and has some measures to mitigate this, but is still far from fool-proof. The gif below illustrates the problem:

“Old Completion”, 184KB, Click to Play.
In what probably represents the worst effort:payoff ratio in all of history, I spent a while re-working the macro recording and playback system to store attributes of completions as they occur, and then attempt to play them back using the same context-sensitive system; the “repeat last change” mechanism was re-worked to also use this system, so I’ll use that to demonstrate:

“New Completion”, 176KB, Click to Play.
 

That’s about it for macro support; now back to the small changes!

  • Fixed regression where opening a view and immediately giving it a selection (for example, by clicking an entry in KDevelop’s grepview) didn’t switch to Visual mode, leaving us with a “stuck” selection that we needed to manually enter & exit Visual Mode to clear.
  • Got rid of the baffling “No more chars for bookmark” message every time we set a breakpoint in KDevelop ;)
  • Fixed crashes with guu & gUU on empty line.
  • Implemented command-mode mappings (cmap etc) for use in the emulated command bar; this was prompted by my preference for using “ctrl-e” instead of “ctrl-g” for the “insert escaped register contents” (‘e’ is right next to ‘r’ on my keyboard) and for “ctrl-m” instead of “ctrl-e” (“jump to the end of the command bar”).
  • Implemented the *unmap (vunmap, iunmap, cunmap etc) family of commands for removing mappings from the command line.

… and that’s about it! I’m going to take a break from Kate stuff for a while to work on some other projects I’ve been neglecting, but will return to finish off some odds and ends later :)

The Kate Vim mode is in pretty good shape nowadays, so if you were thinking of trying it out, now (with the current master) would be a good time :)

Intel Threading Building Blocks Scalable Allocator & Valgrind

Hi,

if you ever use the TBB (Intel Threading Building Blocks) allocator to overwrite malloc/free/* and want to use Valgrind for leak checking and fail, here is the simple trick to get it working:

valgrind --soname-synonyms=somalloc=\*tbbmalloc\* <your-application-here>

I missed that hint in the Valgrind documentation for my first tries ;)

Btw., the scalable allocator from TBB is a really BIG improvement over the normal system allocator on current Linux & Windows machines if you allocate mostly fixed size small object, like what happens if you heavily use STL data structures like std::map/set that are implemented as trees and you have other stuff like DOM/AST like data structures (even in the single threaded case, for which it just saves a LOT of memory).

Ramblings about compilers…

In my job I work on binary and source level analysis software running on Linux and Windows. One of my tasks is to maintain the build farm and compile environment, therefore I am responsible for keeping care of the compilers and libraries we use (like the beloved Qt, congratulations for the nice 5.1 release, btw.).

For Linux, we normally use the GNU C/C++ compiler. That works out very well since years. We have hit compiler bugs only a few times and there was always already some patch-release out in the wild that fixed our issues. In addition the GCC developers brought a constant stream of improvements in the area of C and C++ standard compliance. I am more than happy with the GCC quality.

I did some brief experiments with LLVM/Clang two years ago and was not that impressed in the stability of the C++ support (especially of the fresh libc++ STL implementation) but even for a such young project, it was fast as light to get issues fixed, like a crash in the unordered containers (see Bug 10278, only some hours from report to fix, even if it was only a trivial size variable used uninitialized in the libc++ implementation). I guess today LLVM would perform ok to compile our software (including Qt), still I will stick to GCC, given we use some libraries that normally only get love for that compiler on Linux (at least at the moment).

But now, after my experiences with open source compilers, lets take a look at the wonderful world of Windows.

I know for Windows GCC variants are available (like the 32 bit MinGW-builds shipped with some Qt library variant) and some experimental work on some LLVM able to compile applications using Windows SDK headers is ongoing. Still, the most common way on Windows is to use the Visual Studio compiler and given we always have some libraries of 3rd-parties around that we need to use and that are Visual Studio compiled the Microsoft compiler is more or less unavoidable.

Now, lets take a look of this fabulous compiler you get there. At the moment two major Version are of interest, the 2010 and 2012 variant of Visual Studio.

As I started to evaluate Visual Studio 2010 in 2010, I was immediately hit by its nice 64 bit optimizer bug. Not a single Qt application would start up without immediate segfault… More about this here and even in the Qt bugtracker, especially the answers from the support are interesting, it seems to be forbidden to return a object per value from a function ;)

From report to “hotfix” it has taken some months, that means with the release version of 2010, you couldn’t build any optimized x64 application using Qt (and I guess most other evil libraries using object on the stack that got misaligned by the optimizer) for some months.

Afterwards I was that adventurous to install the SP1 to get the included further compiler fixes. Unfortunately that meant I got all my x64 compilers deleted, but no problem, just close to a month later (until which you need to reinstall your Visual Studio/SDK without SP1 and reapply you hotfix) you got again some fix that reinstalls the compilers that SP1 will remove. Really, you will need to install the 2010 + SP1 + then a fix that restores the compilers SP1 removes if you want to have a working VS 2010 Express for x64 (or stay without SP1 and use the above hotfix for the x64 compiler).

For Visual Studio 2012, which I started to eval months ago, you will get the same story, but reversed. Now we have some nice and nasty x86 optimizer bug. That bug is reported since long see here, it leads to problems for software that use libicu, see the matching icu bug. But given the simple code pattern that can make it occur, it may hit other parts, too. Until now, no fix, only some “will be fixed in next version”…

For me that means, I need to stick with 2010 for x86 and with 2012 for x64, given I want to use the pre-build Qt binaries and not to dive into the “compile qt on windows” fun once more. I would like 2012 for x86, too, to benefit from the C++11 feature improvements and to not have to support two different compiler variants, but I guess that will have to wait, seems its not that easy to build some working x86 compiler that can do optimizations.

Therefore, really, +1 to the compiler developers of the GCC and LLVM project. Its amazing what they have archived!
And +1 for the improvements in the C99 and C++ area in Visual Studio, too. Still I really would be more than happy to get a fix for the x86 optimizer issue in VS 2012. Even no report that it is really fixed in 2013 until now, but I am not sure that was really tried, I didn’t try it myself because I am lazy and must blog instead ;=)

QUndoStack versus Kate’s Undo System

I’m currently using QUndoStack in a project of mine. A QUndoStack contains a list of QUndoCommands which then can be executed with QUndoStack::undo() and QUndoStack:.redo().

Now I thought when deleting an instance of the class Node, I can just hock the creation of the respective UndoDeleteNode class (derived from QUndoCommand) into the destructor, like this:

class Node {
public:
  // ...
  ~Node() {
    undoStack()->push(new UndoDeleteNode(this));
  }
};

class UndoDeleteNode : public QUndoCommand {
public:
  // ...
  void undo() {
    // create node again
  }
  void redo() {
    // delete node again
  }
};

Now the problem with this code is, that executing UndoDeleteNode::redo() will delete the node through the undo/redo system. In this case, the destructor ~Node() will create another UndoDeleteNode item. So what I was looking for a way to only create the undo item when the undo/redo system is not active, i.e.:

class Node {
public:
  // ...
  ~Node() {
    if (!undoStack()->undoOrRedoRunning()) {
      undoStack()->push(new UndoDeleteNode(this));
    } 
  }
};

I found QUndoStack::isActive(), but isActive() does something completely different. In fact, it looks as if there is no way to tell that the QUndoStack is currently operating. This finally also gets clear by reading the API docs of QUndoStack::push(QUndoCommand* cmd):

[...] Executes cmd by calling its redo() function. [...]

In other words, each QUndoCommand you push on the stack is executed immediately. This obviously is a design decision: Following this paradigm, you should not just call “delete node;” Instead, you should simply create a UndoDeleteNode() instance and push it to the stack. The undo command is then immediately executed by calling redo().

This design has the advantage that developers are forced to go this route. Following this paradigm, you very easily get macro support, since all actions are undo/redo items. This is cool.

However, what I dislike about this approach is the fact that it seems you loose the nice API to simply delete the node. You cannot write “delete node;”. Instead, you have to have some helper class such as a Document, that provides a Document::deleteNode(Node* node) function. This function then just creates a new UndoDeleteNode and pushes it to the stack.

Is this how the QUndoStack is supposed to be used? Does it imply that the destructor should be protected, and UndoDeleteNode must be a friend class of Node, to avoid other classes from just calling delete Node?

In Kate, we indeed go the other way: We have a KateUndoManager::isActive(), which indicates whether the edit actions should be tracked by the undo manager or not…

I’m not yet convinced that the approach of QUndoStack is ultimately cool. To me it looks like I’m forced into some design decision I didn’t want to take. Maybe I’m missing something?