All posts by shaheed

Python plugin gets support for Python3

Being vaguely aware that Python3 had some “interesting” differences compared to Python2, I had decided to not think about Python3 for now, but then one of our dear users piped up to say that even building it was broken! That seemed weird, so I started poking around only to find myself falling Alice-like into a Wonderland where strings were not always strings…

Well, I’ve long been interested in i18n and l10n in all their forms, especially as they apply to Indic languages, so I was somewhat aware of the sorts of issues that Unicode can throw up. Luckily, as a KDE developer I’m used to depending on QString handle all the routine grunt work so it was a bit of a rude awakening to discover that, the C API for Python strings takes many forms:

Note only that, but:
  • Ubuntu does not yet have 3.3.
  • The cmake support in KDE before 4.9.4 cannot find the right libraries.
  • The PyKDE4 support for strings was broken-then-fixed.
  • Python3 pickles structures differently than Python2.
Anyway, with some excellent support from Luca Beltrame and Alex Turbov, I’m glad to say that Pate and its plugins should now work with any of Python 2.x, <= 3.2, or 3.3 in the upcoming 4.9.4 and 4.10 releases. Thanks Luca and Alex!

ID + etags > ID || etags

ID files and TAGS files are generated by GNU idutils and etags respectively. They are often used in coding projects to facilitate looking up, for example, the name of a function, and then visiting where it is defined, etc. In large projects, ID files can be hundres of MB in size, and TAGS files several times that. That looking up is obviously nice to have integrated into your favourite editor…

When I started on getting ID file support into Kate, my need was to efficiently navigate source code indexed using an ID file a bit over 100 MB in size. The bad news is that I now have ID files over 300 MB in size, but the good news is even that is not a problem! Not only that, but by combining the ID file with dynamic invocation of etags, I can have all the TAGS functionality I care about without needing to pre-generate a 600 MB TAGS file as well.

It even works reasonably well even when the ID files and the source files are mounted via SSHFS over a WAN link!

But best of all, you can have it too, from today’s Kate 4.10 beta, using a shiny new Python plugin. Here is how…

You may need to restart Kate, but then you should see a new menu bar entry:

Selecting any of them will display a configuration dialog which allows the ID file to be selected:

You are probably thinking “Wow, that’s quite complicated for a file open dialog!”, so let’s go through the features step-by-step…

  • Remember that KDE’s file open dialog (icon on top right) understands environment variables. So if your workspace has a handy environment variable point to its root, getting to an ID file near the root is very fast (I use “$ws” because it is quick to type).
  • The Browse Tokens functionality performs completion on the token, and setting the “Complete Tokens after” prevent the completion kicking in annoyingly before you have finished typing even a first few characters. Values like 3 or 4 work well in many cases.
  • The Transform Filenamessection allows the filenames inside the ID file to be mapped to slightly different filenames in your $ws. That is useful when:
    • The ID file was built somewhere other than your current $ws. That might be another of your workspaces, or possibly an archived workspace for a released build.
    • The ID file was built on your compile server, but you’ve mounted your workspace at a different mountpoint on your laptop (so you have access to the latest version of Kate :-)). Or perhaps you are using Clearcase, and don’t have it on your laptop.
  • Try “What’s This” on the “With this file prefix” to see how you can simply the setting of this value. In the example shown, notice how the first part of the ID filename “/view/myview/vob” is the same as this value? It seems common practice to put the ID file at or near the top of $ws, so including %{idPrefix} helps support this.
    • More important than saving a bit of typing is that if all your workspaces are built using a common recipe, you will never need to touch this value again!
    • The ID file was built somewhere other than your current $ws. That might be another of your workspaces, or possibly an archived workspace for a released build.
  • The Highlight Matches section allows:
    • etags to be run on the matches in the ID file to distinguish the definition(s) of the token from mere declarations and usage with a distinctive icon.
    • Matches in files with the given suffixes will be flagged with another distinctive icon.

Once you’ve chosen a usable ID file, a toolbar will spring into existence at the bottom of your window:

  • As you can see, the Token field allows browsing by token. Selecting a value will cause the table below it to be filled with the matches from the ID file, highlighted using the Settings chosen previously. (They can also be tweaked using the button top right).
  • Click on an entry in the table to open the file and jump to the match. When you do, two entries will also be added to the table on the right which record where you were before the jump, and where you jumped to. That allows you to easily move backwards and forwards though the browsing history.
  • So what is that Filter thingy? Any matches returned from the ID file, can be further filtered using a regular expression. For example, a filter expression of “%{token} =” will find assignments to “malloc_failure” here:
(Note that you have to use Shift-F1 to get to the What’s This since the toolbar is missing the nice little “?” button).
  • Another nice use of the filter is as a poor man’s full text search: ID files will contain tokens including words in strings. So putting a second word of interest in the Filter will return only the matches which contain both words…
  • Finally, in 4.11, auto-completion is enabled (don’t forget to configure it in Kate->Settings->Configure Kate->Editing->Auto-completion)
Inline images 1
I hope you’ll find these useful add ons to the normal experience of using ID files, but feedback is always welcome!

 

Python plugin developer guide, part 1

I’m a Python newbie, so if you are at least that good :-D, you should be able to dive in and write useful Python plugins for your favourite editor too! This is the first in what I hope will be a series of notes and tutorials to help you along the way.

Where can I find examples to steal from inspire me?

Any good developer knows that a good way to launch into a new area is to look at examples, so here is a list of places I’ve encountered.

Remember to respect others’ copyrights, and give credit where it is due.

Where do I put my code?

The Pâté Python plugin looks in several places to find usable plugins, you should be able to use the Python console to find a directory you can use for your developments. Select it, and press “Reload”:

Launch it from the View menu, and then ask KDE where your files should go:

Or, if that’s a bit hard to read, like this:

>>> from PyKDE4.kdecore import *

>>> print KStandardDirs.locateLocal(“appdata”,”pate”)

/home/srhaque/.kde/share/apps/kate/pate

Try the directory

If the given directory does not exist, create it. Press Reload, and the new directory should be visible:

You are ready to create your plugin in the new directory! This plugin directory will be added to the initial value of sys.path used for all plugins.

Plugin structure and naming

Your plugin can be just a single .py file, such as “console.py”, or multiple files contained in a directory. A single .py file is useful for simple things, but if you want to use .ui files or multiple Python files, a directory is recommended. If you use this option, the main plugin file must be named after the directory, so:

  • If the directory is called “console”
  • The main plugin file must be named “console/console.py”

In the directory case, the directory is added to sys.path to load the main plugin file and other Python modules you may have. Since sys.path is searched by Python in order to load modules, having the same plugin in multiple locations would be very confusing. Pâté supports this by refusing to load identically named plugins; instead they are highlighted as being hidden:

This also means you can just copy one of the plugins supplied with Kate to get started!

 

Python plugin user guide

KDE 4.9 has branched, and so the Pâté plugin ”host” for Python plugins has moved from Kate’s playground to be a mainstream plugin. Since a plugin-for-plugins might be a bit confusing, here is a quick tour of how it is managed and works, and especially a few rough edges it still has.

NOTE If you’ve installed this from a package, hopefully your distro will have added kate.so under PyKDE4 as needed, but if you are building from source, see the notes at the end of this post.

Start by enabling the enabling the Pâté plugin:

That should cause the Pâté plugin’s configuration page to pop into existence, but you may need to press the Reload button to see the right hand pane filled in like this:

Enable specific Python plugins of interest, and press the Reload button. The comment will be set to “Loaded”, then close and reopen the dialog (rough edge #1) to see any new configuration pages that the plugin provides:

Go back to the Pâté plugin’s configuration page, and now look at the second tab:

Use the combo box to select one of the built-in support modules (kate and kate.gui) or any of the loaded plugins. The first tab shows HTML documentation for the Python code and you can see any actions it has added into the Kate UI on the second tab (this will be empty for the built-in modules, but not for console):

It looks as though the Python console has added one entry to the View menu, alog with a shortcut. And so it has:

Yaay! See if you can guess what to do next :-)

 

IF YOU HAVE BUILT FROM SOURCE

And then installed the results in say /usr/local, attempting to enable the plugin may result in a crash with something this on the terminal:

>  File "/home/.../kate/plugins/pate/kate/__init__.py", line 38, in <module> from PyKDE4.kate import Kate
> No module named kate
> Could not import kate"

This is because the Pate build should install kate.so as part of the PyKDE4 bindings, but typically, a developer build will install this locally, and not to where the remaining system-supplied PyKDE4 files are installed:

Install the project...
 -- Install configuration: "RelWithDebInfo"
 -- Up-to-date: /usr/local/lib/python2.7/dist-packages/PyKDE4/kate.so
 ...

The system-supplied PyKDE4 binding modules should be in Python’s sys.path:

srhaque> python
 Python 2.7.3 (default, Apr 20 2012, 22:39:59)
 [GCC 4.6.3] on linux2
 Type "help", "copyright", "credits" or "license" for more information.
 >>> import sys
 >>> print sys.path
 ['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7']

For me, Ubuntu puts PyKDE4 at /usr/lib/python2.7/dist-packages/PyKDE4, so we need a link from there to the actual location:

srhaque> ls -l /usr/lib/python2.7/dist-packages/PyKDE4/kate.so
lrwxrwxrwx 1 root root 53 May 20 09:51 /usr/lib/python2.7/dist-packages/PyKDE4/kate.so -> /usr/local/lib/python2.7/dist-packages/PyKDE4/kate.so

Note that just having this location in PYTHONPATH and/or in sys.path does not work because the PyKDE4 needs an __init__.py to be seen (see the one in the system directory to understand why).

I’ve not thought of a good way to simplify this for developers, and I assume end-users won’t see it because we’ll install in the default path. Suggestions/fixes welcome.