Category Archives: Developers

Syntax Highlighting Checker

The KTextEditor Framework uses the syntax highlighting files provided by the KSyntaxHighlighting Framework since the  KDE Frameworks release 5.28.

The KSyntaxHighlighting Framework implements Kate’s highlighting system and meanwhile is used in quite some applications (e.g. LabPlot, KDE PIM). What is quite nice is that the KSyntaxHighlighting framework is nicely unit tested. And while we do not have tests for all highlighting files, we still provide some quality assurance through a compile time checker.

How does it work? Well – in former times, Kate loaded all highlighting .xml files from disk (through the KTextEditor framework). This lead to a slow startup over time, since there are >250 .xml files that needed a stat system call at startup.

With the KSyntaxHighlighting Framework, all these xml files are compiled into a Qt resource (qrc file), that then is included into the KSyntaxHighlighting library.

In order to create the Qt resource file, we need to iterate over all available xml files anyways. So what happens is that we take this opportunity and also scan the highlighting files for common mistakes.

As of today, we are checking the following:

  1. RegExpr: A warning is raised, if a regular expression has syntax errors.
  2. DetectChars: A warning is raised, if the char=”x” attribute contains more or less than one character, e.g. when char=”xyz”, or char=”\\” (no escaping required), or similar.
  3. Detect2Chars: Same as DetectChars, just for char=”x” and char1=”y”.
  4. Keyword lists: A warning is raised, if a keyword entry contains leading or trailing spaces. Additional trimming just takes time.
  5. Keyword lists: A warning is raised if a keyword list is unused.
  6. Keyword lists: A warning is raised if multiple keyword lists use the same me (=identifier).
  7. Keyword lists: A warning is raised if a non-existing keyword list is used.
  8. Contexts: A warning is raised, if a non-existing context is referenced.
  9. Contexts: A warning is raised, if a context is unused.
  10. Contexts: A warning is raised, if multiple contexts have the same name (identifier clash).
  11. Attributes: A warning is raised, if non-existing itemData is used.
  12. Attributes: A warning is raised, if multiple itemDatas use the same name (identifier clash).
  13. Attributes: A warning is raised, if an itemData is unused.

This list helps us nicely to catch many mistakes at compile time even before running unit tests.

Update (2017-12-17): All above issues are fixed for all highlighting files starting with the KSyntaxHighlighting 5.42 framework, to be released in January 2018.

KTextEditor depends on KSyntaxHighlighting

Recently, the KSyntaxHighlighting framework was added to the KDE Frameworks 5.29 release. And starting with KDE Frameworks 5.29, KTextEditor depends on KSyntaxHighlighting. This also means that KTextEditor now queries KSyntaxHighlighting for available xml highlighting files. As such, the location for syntax highlighting files changed from $HOME/.local/share/katepart5/syntax to

$HOME/.local/share/org.kde.syntax-highlighting/syntax

So if you want to add your own syntax highlighting files to Kate/KDevelop, then you have to use the new location.

By the way, in former times, all syntax highlighting files were located somewhere in /usr/share/. However, since some time, there are no xml highlighting files anymore, since all xml files are compiled into the KSyntaxHighlighting library by default. This leads to much faster startup times for KTextEditor-based applications.

Running Unit Tests

If you build Kate (or KTextEditor, or KSyntaxHighlighting) from sources and run the unit tests (`make test`), then the location typically is /$HOME/.qttest/share/org.kde.syntax-highlighting/syntax.

FirstAid – PDF Help Viewer

Hi,

in the recent months, I didn’t find much time to spend on Kate/KTextEditor development. But at least I was now able to spend a bit more time on OpenSource & Qt things even during work time in our company. Normally I am stuck there with low level binary or source analysis work.

For our products, we were in the need of some online help. As our documentation is delivered as PDFs generated by the tools of the TeX Live distro, a natural idea was to use some PDF viewer and integrate it more tightly in our software than just “open the manual at page 1”.

We did review PDF viewers out there, but most (like Okular) have too many dependencies to be just bundled with our product (or a license not permitting that).

Without bundling, we can’t ensure that the tight coupling is working, without starting to test the integration with X different viewers which more or less all need other kinds of command line arguments to open the right page or even lack that feature or will not reuse an already running instance, ….

Therefore, as our GUIs are developed with Qt anyways, we did take a look at libpoppler (and its Qt 5 bindings), which is the base of Okular, too.

Easy enough, taking the small demo program shipped with the library and adding a small stdin based interface to tell it “goto <named reference>” we arrived at some small PDF viewer that is fit enough for our use case.

We named the thing “FirstAid”, the sources can be grabbed at github.com/AbsInt/FirstAid. Like libpoppler and the demo, its licensed as GPLv2+.

As already the README states, the aim of this small project is not to replace some full fledged viewer like Okular, the design goal is to have a small viewer that is auto-started by some host application and will jump to the requested labels for a tightly coupled online help. It can be used as a pure standalone PDF viewer, too, but that is more intended for testing it e.g. on the documents that should later be shown as online help.

FirstAid

I already annoyed Albert with some small issue I had with libpoppler, perhaps I will provide more useful fixes in the future if more things come up during FirstAid development. In any case, already THANKS A LOT for the Qt 5 bindings around libpoppler, they work nicely for us!

I really think this small project shows the benefit of OpenSource: We needed a PDF viewer, we were able to create a small one in less than a month based on OpenSource libraries and we can give back the results to the community (if it is useful for others is a different story, but perhaps other people have the same itch to scratch, if not, ignore it). I hope more possibilities for such things come up at work in the future.

For building: It should build out of the box if you have some recent Qt and libpoppler-qt5-dev installed, at least the Travis CI is able to build it out of the box with the given config. For me, it shows some small bugs if used with Qt 5.6/7 compared to the Qt 5.8 Beta I used here for testing.

KSyntaxHighlighting – A new Syntax Highlighting Framework

Today, KDE Frameworks 5.28 was released with the brand new KSyntaxHighlighting framework. The announcement says:

New framework: syntax-highlighting
Syntax highlighting engine for Kate syntax definitions

This is a stand-alone implementation of the Kate syntax highlighting engine. It’s meant as a building block for text editors as well as for simple highlighted text rendering (e.g. as HTML), supporting both integration with a custom editor as well as a ready-to-use QSyntaxHighlighter sub-class.

This year, on March 31st, KDE’s advanced text editor Kate had its 15th birthday. 15 years are a long time in the software world, and during this time Kate won the hearts of many users and developers. As text editing component, Kate uses the KTextEditor framework, which is used also by applications such as KDevelop or Kile.

The KTextEditor framework essentially is an embeddable text editing component. It ships everything from painting the line numbers, the background color, the text lines with syntax highlighting, the blinking cursor to code completion and many more features. One major feature is its very powerful syntax highlighting engine, enabling us to properly highlight around 275 languages.

Each syntax highlighting is defined in terms of an xml file (many examples), as described in Kate’s documentation. These xml files are read by KTextEditor and the context based highlighting rules in these files are then used to highlight the file contents.

For the last 15 years, this syntax highlighting engine was tightly coupled with the rest of the KTextEditor code. As such, it was not possible to simply reuse the highlighting engine in other projects without using KTextEditor. This lead to the unfortunate situation, where e.g. the Qt Creator developers partly reimplemented Kate’s syntax highlighting engine in order to support other languages next to C/C++.

This changed as of today: The KSyntaxHighlighting framework is a tier 1 functional framework that solely depends on Qt (no dependency on Qt Widgets or QML), is very well unit tested, and licensed under the LGPLv2+. As mentioned in the announcement and in the API documentation, it is a stand-alone implementation of the Kate syntax highlighting engine. It’s meant as a building block for text editors as well as for simple highlighted text rendering (e.g. as HTML), supporting both integration with a custom editor as well as a ready-to-use QSyntaxHighlighter sub-class. This also implies that you can reuse this framework to add syntax highlighting to e.g. QML applications.

We hope that other applications such as Qt Creator will start to use the KSyntaxHighlighting framework, since it allows us to cleanly share one single implementation of the syntax highlighting engine.

In the next KDE Frameworks releases, we will remove KTextEditors syntax highlighting engine in favor of just using KSyntaxHighlighting. This will happen step by step. For instance, we already have a pending patch that removes all xml files from KTextEditor.git in favor of using the ones shipped by the KSyntaxHighlighting framework. That means, with the KDE Frameworks 5.29 release, Kate’s and KTextEditors dependency (and other application’s dependencies) will look as follows:

KTextEditor and KSyntaxHighlighting

This is quite an interesting change, especially since moving the syntax highlighting engine out of KTextEditor was already planned since Akademy 2013 in Bilbao:

Another idea was raised at this year’s Akademy in Bilbao: Split Kate Part’s highlighting into a separate library. This way, other applications could use the Kate Part’s highlighting system. Think of a command line tool to create highlighted html pages, or a syntax highlighter for QTextEdits. The highlighting engine right now is mostly internal to Kate Part, so such a split could happen also later after the initial release of KTextEditor on 5.

This goal is now reached – thanks to Volker Krause who did most of the work. Pretty cool!

If you are interested in using the KSyntaxHighlighting framework, feel free to contact us on our mailing list. Further, we welcome all contributions, so please send patches to our mailing list, or post them on phabricator. (You can also find the KSyntaxHighlighting framework on github for convenience, but it’s not our primary platform).

You can also support Kate and the KDE Frameworks by donating to the KDE e.V., KDE’s non-profit organization.

Update: Changed XML Syntax Definition File Location

Starting with KDE Frameworks 5.29, the KTextEditor framework now uses the syntax highlighting files from KSyntaxHighlighting. These files are located in $HOME/.local/share/org.kde.syntax-highlighting/syntax. If this folder does not exist, just create it. Futher, you will not find any syntax highlighting files in your system installation, since the syntax highlighting files shipped with KSyntaxHighlighting are compiled into the executable.