Loading lots of images asynchronously

In AppWhirr we have to load a lot of icon image for store list pages. The network communication is absolutely asynchronous (even the response parsing) since we started the project but still the listing pages blocked the UI thread.

We identified the problem and it’s the icon image resource loading. We store the icon images as QPixmap as QPixmap is optimized for image drawing and we don’t have to modify these images.

Our first attempt was to load and create QPixmap in a background thread (with QRunnable and QThreadPool). It worked on Windows but didn’t boosted the performance as much as we hoped. 

Then we built the Mac version and it didn’t even loaded the images. We checked the logs and we found the problem: “QPixmap: It is not safe to use pixmaps outside the GUI thread”

We searched for a solution and found that the problem is QPixmap performs some platform and painting system dependent operations. On Windows it just hangs and performed on the main-thread but on Mac it doesn’t even allow to create or change the QPixmap’s data in a background thread. Obviously none of these is ideal.

We had to find another solution and of course Qt is here to rescue. Although QPixmap cannot be created or modified in a background thread QImage can. And QPixmap can be created directly from QImage.

“QImage on the other hand is a pure platform-independent data structure optimised for direct pixel access, so it is safe to paint into a QImage. You only manipulate data managed by Qt itself - it does not use hardware acceleration and stuff. But you can’t paint into a QPixmap (or even create one) in a non-GUI thread, which is optimised for fast painting, using the underlying paint subsystem.”

(source: http://lists.trolltech.com/qt-interest/2008-11/msg00534.html )

To sum up the solution:

If you have lots of images as QByteArray resource retrieved from the server and want to present it as a QPixmap you can transform the QByteArray to QImage in a background thread (with QImage::fromData) and in the main thread you can transform the QImage to QPixmap (with QPixmap::fromImage).

Don’t optimize - think differently!

I wrote a post which included this principle, but I think it’s important enough to highlight it here, in a dedicated post.

So the story from our AppWhirr developer blog (http://bit.ly/yqnBjA):

“We store most of our internal information in SQLite databases. At a point one of these databases got too busy and slow and we had to rewrite that part (the database was used by a lot of objects and from a lot of threads simultaneously). Finally we ended up with a dedicated SQLite database server thread which is managed by a gateway class and works with an event-queue. The basics are similar to this approach:  http://www.linuxjournal.com/article/9602 . Of course this required  a lot of recoding - the whole communication is rewritten in a fully asynchronous manner. It took almost 7 days to create the separate test project, test our solution and measure it then recode the previous solution in the client. The result: with our test database and with previous solution the initial application loading and synchronization took about 10-15 seconds on our test machine. After the recoding it takes less than 2 seconds! And the totally asynchronous communication improved other parts of the application responsiveness as well.”

Also this is not just about performance. Currently we had to extend the app installation process and I decided I will rewrite it before I extend the process with the new required steps, and separate into distinct sub-part delegates which communicate via a thin and clearly defined interface with each other.

The result: 1) the code is much easier to digest 2) it resolved bugs! 3) I could even make it perform better because I identified some potentially time consuming operations running on the main (UI) thread.

So my advice: don’t try to optimize an existing solution but try to think differently and rewrite it. It will benefit much more both for you and for the project.

Previously the installation code was basically just 1 giant black-box nobody wanted to touch. Now it’s separated into 4 distinct easy to understand parts which can be extended or modified more easily.

Qt 4.8 with MSVC 2010: ‘cl’ is not recognized as an internal or external command

We have some issues with the Qt 4.7.4 SDK so I figured it out maybe the Qt 4.8 (currently RC) will solve these (mostly Flash related) problems.

The new Qt SDK Maintenance Tool is very handy, some clicks and you’re good to go with any Qt version.

So I downloaded the 4.8 RC and downloaded the Windows SDK 7.1 which (should) contain the Visual Studio 2010 C++ compiler.

I tried a full recompile, even deleted build directories by hand, but I got this annoying error every time I tried to compile the project: ‘cl’ is not recognized as an internal or external command.

I double-checked everything and it seemed to be just fine, but still I got this error. I opened the SDK’s Command Prompt which should set everything to build from command line, but even the Prompt gave me this error. So what’s wrong? What did I miss?

Well, I guess nothing. After hours of trying, recompiling, reinstalling still got the error. Then I found a blog post which made me suspicious: if you have the Visual Studio XXXX and install the Windows SDK after then it won’t install the build tools because it’s already there for the Visual Studio.

I installed Visual Studio 2010 a few month ago for some testing but I uninstalled it right after I finished with the testing. But what if Windows still detects it somehow to be installed…

So as a last chance I installed the Visual C++ 2010 Express, then tried to recompile again… And it worked!

I still cannot believe it. But here it is, for anybody experiencing the same issue.

Status report

I’m one of the developers and founders of a new upcoming startup project called AppWhirr (official site: appwhirr.com). We are really close to the launch of the first public Beta version, so I don’t have enough time to blog here.

The good news are: 1) I will continue posting to this blog when things go smoother, and 2) we have a development related blog as well. Currently that one is more important, so most of my development posts will land there (if related to AppWhirr).

Link: http://appwhirr-dev.tumblr.com/

Happy developing!

Custom Layout in Qt - what you have to do to ensure widget sizes

I have to write custom layouts for our application. It was a bit hard to understand a what a layout has to do and where it has to do, but in most cases it worked.

There was an exception which caused visual artifacts. This was a bug with a lower priority, so I skipped and started to fix it yesterday.

The issue was: in one of our custom layouts the widgets doesn’t expand horizontally and the added widgets kept their initial width after the first layouting.

I logged every related event and it seemed that the layout works perfectly and calls setGeometry() on every added widget with correct sizes, but somehow the widgets’ resizeEvent() methods doesn’t get called. I tried to set every relevant policies and properties but somehow the resizeEvent() remained silent and called with the initial sizes.

What was the solution? I did this for every widget in the layout when the inner _performLayout() method called (which is a bit optimized layout-ing performer, but basically it’s called every time it has to do some layouting).

for: every widget in layout:
currWidget->setGeometry(itemGeom);

And the itemGeom was correct. Then I added these right before the setGeometry() is called, just because I was curious:
currWidget->setMinimumSize(itemGeom.size());
currWidget->setMaximumSize(itemGeom.size()); 

And the layout started to work perfectly! So as a lesson: although the layout is the master (or it should be) in layouting and resizing the widgets, the widgets can override the given setGeometry() value if their minimum/maximum size does not allow it to exceed.

Lesson learned :)

Qt: mkdir or mkpath

I just vanished a bug: for some reason the uncompression algorithm we use worked in most cases, but not in the most important part: at self-updating.

I debugged, changed, debugged, … And nothing.

Finally I found it: at 1 place in the code I used QDir’s mkdir() method instead of mkpath() and in some cases it tried to create a directory which’s parent directory was not yet created.

It took me hours to figure it out. But now, everything works perfectly, and can continue to do more productive tasks :)

So remember: use mkdir() if you know for sure the parent directory does exists and mkpath() if the parent may not yet created.

QWidget and css style-sheet

I tried and tried for hours to style part of the application to conform with the planned design without any luck. I’ve dropped out parts-by-parts of the layout and at a point I couldn’t achieve the semi-transparent effect required.

Then I searched Qt’s css documentations and suddenly it was there:

Supports only the background, background-clip and background-origin properties.

If you subclass from QWidget, you need to provide a paintEvent for your custom QWidget as below:

 void CustomWidget::paintEvent(QPaintEvent *)
 {
     QStyleOption opt;
     opt.init(this);
     QPainter p(this);
     style()->drawPrimitive(QStyle::PE_Widget, &opt, &p, this);
 }

The above code is a no-operation if there is no stylesheet set.

Warning: Make sure you define the Q_OBJECT macro for your custom widget.

I forgot that I switched a widget with another, and that the new one is inheriting directly from QWidget. In this case this custom paintEvent is required. Dropped in the code and it finally works.

MSVC faster than MinGW?

So far it seems that MSVC 2008 compiler can compile about 2 or 3 times faster our medium complexity (Qt based) project in QtCreator. Probably because MSVC 2008 is pretty good in compiling simultaneously the multiple projects (we have 3 sub-projects, or as QtCreator refers to them “subdirs”).

Anyway it’s great. We have to recompile the whole thing from time to time. With MinGW it took about 8-10 minutes approximately, and with MSVC 2008 it’s done in 3-4 minutes maximum.

Also the compiled exe seems to be faster. So win-win :)

gcc, the permissive

We’re working on a cross-platform startup, and until now I compiled the project with gcc and with mingw. I just had to switch to Visual Studio (2008, particularly).

With mingw the Win APIs only need windows.h to be included. With Visual Studio it also requires user32.lib to compile.

Ended up using 
#pragma comment(lib,”user32.lib”); 
with win specific #ifdef s. Strange that mingw doesn’t require to link it. Or Qt does it for mingw automatically, but not for VS? 

round() in Visual Studio

Why round() is not part of Visual Studio 2008’s math.h ? There’s floor() and ceil() but no round() ?

It’s easy to fix with an inline function like this:

inline float round2(float x) { return floor(x + 0.5f); }

but why should I do? It’s 2011..