Category Archives: Common

A rich python console and more in Kate editor

I have done some improvements in the plugins: python_console_ipython, python_autocomplete, python_utils, js_utils, xml_pretty and django_utils. These plugins I added a month and a half ago (except python_console_ipython) to the kate repository. I have done two improvements and a new feature:

  • Now they work with Python2 and Python3 (except python_autocomplete, this only works with Python2, due pysmell dependence)
  • Now they have a configuration page (except python_autocomplete, this should not have configuration parameters)

Page Config Plugins

The new feature is the integration of the python_autocomplete and python_console_ipython plugins with the project plugin. The behaviour of these plugins depends of the loaded project. That is to say, the python_autocomplete plugin autocompletes with our project modules. Currently the kate community is working to add a new python autocomplete  using jedi (this will work with Python2 and Python3).

Python Autocomplete (with our modules)

And the python_console_ipython plugin has something that we can name the project console. A ipython shell with the python path of our project and with the environments variables that the project plugin needs.

IPython Console (converted in a django shell)

To use this we need to add in the project file (.kateproject) a new attribute named “python”, with this structure:

 {
  "name": "MyProject",
  "files": [ { "git": 1 } ],
  "python": {
        "extraPath": ["/python/path/1",
                      "/python/path/2",
                      "/python/path/n"
        ],
        "environs": {"key": "value"},
        "version": "3.2"
    }
 }

I am a django developer, as we say in Django we can come to have a django shell (python manage.py shell) integrated in our editor. But this feature is a generic feature not tied to Django. This feature can be used by other Python developers. I have added only a specific feature to the django developers. If we set in the project file (.kateproject) the attribute projectType with the value Django, the ipython shell automatically loads the models and the settings like a shell_plus does.

  {
   "name": "MyProject",
   "files": [ { "hg": 1 } ],
   "python": {
        "extraPath": ["/python/path/1",
                      "/python/path/2",
                      "/python/path/n"
        ],
        "environs": {"DJANGO_SETTINGS_MODULE": "myproject.settings"},
        "version": "2.7.3",
        "projectType": "django"
    }
  }

I hope you like it :-)

New Text Folding in kate.git master

In the kate.git master branch the text folding is now new and shiny.

In addition to be faster and less memory hungry (no folding tree is around if you fold nothing, it is only created on-demand exactly for the folded regions), the new code is less complex and smaller (and hopefully better documented + unit tested, it actually has a test for most internal operations).

There is actually now a clean separation, the folding does not mix with highlighting and can be used without it, too.

This allows for a better integration of indentation based foldings like in Python (and already fixes the most ugly miss-placements of folding regions there) and empowers the user to fold arbitrary regions manually.

This can be done at the moment with two kate commands (for the F7 command line):

  • “fold” will create persistent fold for the current selection, this fold will stay until next reload of document (will implement some dfold command later, to delete it)
  • “tfold” will create temporary fold for the current selection, this fold will disappear again on unfolding

This should allow better vi mode integration, too, as vi is able to fold arbitrary regions, too. (There is an internal API to fold any KTextEditor::Range you pass in, which should allow to fold by regex and other ways, too, if somebody steps up to implement it.)

If you find glitches, please report them as bugs and please attach some file to reproduce, I am very interested to get this all in even better shape for KDE 4.11, be folding crashs a thing of the past.

P.S.

For all people not using folding at all, be happy, it no longer will crash you (and me), as it is a nop if no folding ranges are created by the user and some extreme cases got a speedup of 1000% because of less overhead, like VERY large C++ sources ;)

Text Folding Reloaded

If we look at the incoming bug reports, a lot still seem to be located in our code folding implementation.

The problem of the current code folding code, even after the GSoC rewrite of large parts, is that it is inter weaved with the syntax highlighting engine.

During highlighting, we try to build up some tree containing all folding start/ends. To accommodate languages that have indentation based folding like Python, we started to fake such starts/ends which are normally tokens like { or }. As these tree can be unbalanced and contain overlapping regions, the implementation is quiet complex.

Additionally, as the folding relies heavily on the syntax highlighting, which is done per document, it is impossible to have different folding per view.

After looking again at the code to solve some issues, it got more and more clear to me, that we need to restart that from scratch.

Instead of feeding the highlighting info into a tree, I started to implement text folding that just allows you to mark regions as folding ranges manually. Then internally a non-overlapping tree of ranges is created that uses our moving cursors to keep track of changes in the document. Each view will have an instance of such a tree, which allows to have folding per view.

These folds are only created on demand, e.g. if you decide to fold some region via selection + “fold” command. This already works quiet well in my Kate repository clone. The missing part is now the integration with the highlighting, that should use this well defined API to only create a folding region in the moment the user wants to fold it. In addition some API for the highlighters is needed to indicate that for some line there is the fold action. After that is done, I might consider to replace the current implementation with the novel one, which is btw. much easier to unit test (which is already done for most API primitives).

Btw., the work is done in this repository, if somebody wants to contribute, patches are welcome .P

Kate Vim Mode: Papercuts (+ bonus emscripten-qt stuff!)

I was a long time Vim holdout and often wondered to myself, “just why do those nutheads use vim?” until I googled and read the blog I just linked. After trying Vim out for a few days, I was completely hooked, but didn’t want to leave KDevelop with its awesome Intellisense/ code completion/ navigation features. Thankfully, Kate has a Vim mode, so I could get the best of both worlds :)

However, I’d often run into small but jarring inconsistencies with Vim, or sometimes find missing features, and so I set about trying to fix as many as I could. Since these were usually very small and unimportant things, I’ll just go ahead and list them, Graesslin-style :)

Vim mode is one of those rare and happy pieces of code which you can very easily write tests for: just add the starting text; the list of keypresses; and the expected, final text. Kuzmich Svyatoslav wrote a big battery of them a while back, and my first ever patch was an incredibly trivial little one that fixed a hole in the test framework and a bug that the hole obscured.

After that, I:

  • Improved the test framework to enable us to test Insert Mode features;
  • Fixed the inner curly bracket text object to be more like Vim (rather than deleting everything between the curly braces on a diB, Vim usually attempts to leave the closing curly bracket on its own line);
  • Added { and } motions to Visual Mode;
  • Fix a few bugs with Erlend’s very handy “comma text object” extension;
  • Fixed numerous bugs with t/T and , & ; involving them getting “stuck” behind occurrences of the character to search for, and also made them “counted” so that one can do e.g. 3ta, and also mirroring Vim’s behaviour where if we can’t find the required count of characters on the line, we don’t move at all;
  • Stopped the transition from Visual Mode to Normal mode from resetting the cursor position if we exited via ESC or Ctrl-c;
  • Fixed a bug with the nifty “block append” feature;
  • Fixed a nice bug with replaying the last command via . when the last command was an insertion that had a Ctrl-o in the middle of it (all the characters after we exited the Ctrl-o were treated as commands, with predictably zany results!)
  • Ensured that the bracket text object worked in Visual Mode;
  • Made the very handy Key Mapping feature “counted”, so you could perform your chosen Mapping several times in a row, and also made them count as one edit that could be reverted with just one request to Undo;
  • Made p & P move the cursor to the end of the pasted text instead of leaving it at the beginning (whoops!), and similarly for gp/gP which places the cursor one past the end, hopefully respecting Vim’s slightly obscure exceptions and corner cases (e.g. p and P act as gp and gp when in Temporary Normal Mode);
  • When doing Ctrl-o in Insert Mode, move the cursor back one step if we are at the end of the line;
  • Try to keep the length of the number under the cursor unchanged on Ctrl-x and Ctrl-a and also don’t let the cursor stray off the number;
  • Respect Vim’s rules for counting with Ctrl-x and Ctrl-a, especially with repeats;
  • Fix a lovely bug with some of the key parsing where the repeat last change command did not work after a Ctrl-a;
  • Made inserts (i, I, a, A, o, O and block append) counted, so you can do e.g. 100oHello, World!<ESC>, although I now see that the old implementation of counted o and O was “by design”. Oops – I’ll have to sort that out at some point :/;
  • Make Ctrl-c reset the parser so that e.g. d<Ctrl-c> aborts the deletion. This one was cool as it remained undetected for ages by a sheer fluke: the only visible signs were that after doing e.g. d<Ctrl-c>, the cursor would stay at half size, and also another bug that I had been promising myself I’d investigate for a while: typing r<Ctrl-c> would replace the current character with a nonsense character, which turned out to be the representation of the keypress Ctrl-c!;
  • Fixed a few minor flaws where after e.g. osometext<ESC>, we would have to request an Undo *twice* before getting back to where we issued the command;
  • Arranged for a return to Normal Mode if we Ctrl-c or ESC when in Temporary Normal Mode. Vim is weird, here: its behaviour is just as you would expect if you were in Normal Mode, but the status bar still claims we are in Insert Mode!
  • Fixed a bug where issuing y% would actually move the cursor as well as yanking the text;
  • Added the last edit markers, [,] and .;
  • Fixed a few bugs with inner bracket text objects being treated as invalid if there were less than three characters long;
  • Respected Vim’s rules for the [[, ][ etc motions when used with commands: they behave slightly differently than when they are used as cursor motions;

and that’s about it! Kate Vim’s testability make it very pleasant to develop features/ fixes Test First, so all of these are buttressed by a decent amount of automated tests.

The Future

I’m nearing the end of the papercuts stuff, and will shortly move onto slightly larger scale things. This will consist of implementing a Vim-style search and replace bar, with all the usual Vim shortcuts: Ctrl-c to dismiss; Ctrl-r to insert contents of register; Ctrl-rCtrl-w to insert the word under the cursor; etc, and with proper Vim style searching e.g. positioning the cursor on the first letter of the current match; return to where we were if cancelled; interactive search and replace with Y/N/Q/A; etc. I think I’ll use “smartcase” for deciding case sensitivity, but might make this configurable through a hidden config option if there is any demand for it.

There’s currently quite a lot of Vim-specific code that has leaked into Kate’s own Find/ Replace dialog, and I’d really like to be able to delete that and just have Vim use this Vim-specific version. What do people think? I’d have to get it at least as functional as Kate’s own one before I’d consider this, so we’d need e.g. some way of going through the Find and the Replace history separately (I don’t think stock Vim can do this?) and easily toggling whether this is a regex or a literal search, etc. I’d really like some feedback on this, and a suggestion for shortcuts etc!

Done properly, this would give us a powerful, keyboard-friendly Find (and Command Line) dialog where we can easily track the keypresses, which would set us up nicely for the next major task – Vim Macro support!

Before all that, though, I just have a couple of smaller odds and ends to sort out: I need to get auto-completion properly working with . (currently, if auto-completion is automatically invoked, pressing . won’t include the completed text), and it would be nice if Ctrl-p and Ctrl-n could wrap around in the list of completions as they do in Vim (need to sort out a Kate bug with completion first). I also have an another idea which I’d like some feedback on; it would be nice if people would try it out and let me know if they find it helpful, annoying, etc. If only there was a way of letting people test out a patch that hasn’t been committed yet … ! Which leads nicely on to:

emscripten-kate :)

You may have seen my blogs on emscripten-qt, a port of Qt to Emscripten, an awesome technology that allows one to compile C/C++ to Javascript. This caught the eye of Mozilla, who were looking for an example of a “serious” app to go along with the games and utilities in their current list of Emscripten demos, and they kindly offered to sponsor me to work on this. After looking through a list of candidate apps, they eventually suggested porting Kate: not exactly an easy task, as Kate obviously requires not only a big chunk of kdelibs (with its reliance on DBUS and external processes like kded, etc) but also QtScript for its extensibility (the automatic code indentation for various programming languages, for example, is implemented in JS) meaning I’d need to get a C++-based Javascript interpreter ported to Javascript ;)

This ended up being not as hard as I feared, and now we have a decent (though very cut-down; there are no plugins, so it’s more like KWrite than Kate ;)) version of Kate ported to Javascript. I’ve merged in the patch I’d like feedback on as mentioned above so you can easily try it out as long as you have a modern browser, plenty of bandwidth (even after server-side compression, you’re still looking at a tens of MB download :/) and also plenty of free RAM and CPU. There’s an unfortunate bug in Chrome which means you’ll need to use the no icons version, alas. Konqueror and reKonq also do not work, to my knowledge. Let me know what you think, and if anyone has any thoughts on shortcuts for the extended Find/ Command Line bar, do let me know!