A few questions regarding porting from Qt to CS

Discuss anything related to product development
Post Reply
charlyw64
Posts: 2
Joined: Sat Mar 06 2021 5:54 pm

A few questions regarding porting from Qt to CS

Post by charlyw64 »

Hi,

I have stumbled across the CS fork of Qt while reading some comments about the current Qt development direction. I am a software developer who in a team of 3 is developing and maintaining commercial buisiness applications that once were multiplatform and developed using Qt. We have migrated our software to newer Qt versions several times in the past (started off in 1999) but some design decisions in Qt 5 prevented this effectively because these things broke our ability to interface with 3rd party libraries and internal API of the Windows system (which now is our only platform we develop for).

As CS was derived from a Qt version that precedes those design decisions to me it seems that CS is a better foundation to migrate to than Qt-5 (yes, we will never consider Qt-6 in it’s current licensing state - so even the move to Qt-5 would be a dead end) would be. I have a few questions that I would need some help with in my evaluation of the feasibility of our port from Qt-4 to CS.

- In Qt-5 sources were supposed to be only acceptable to be UTF-8 encoded. All our attempts, to make our programs work, failed which we tracked down to this design decision as we are working with Latin-1 encoded sources - changing to Utf-8 broke the compile process as both the compiler and the Qt-translation routines can’t deal correctly with string literals that contain any character that is encoded in more than 1 byte. Funnily there are only two source files in all of Qt-5 that require Utf-8 encoding and they are only used on one of he many mobile platforms). I read through the documentation of CS and just want to be reassured that it didn’t follow the lead of Qt-5 and still allows Latin-1 (or locale) encoded sources.
- In later Qt-4 and Qt-5 versions access to pixmap resources was removed from programs which have no GUI (QCoreApplication derived, not QApplication). Part of our software operates as Windows services which does not allow for a GUI event loop but still needs access to QImage and QPixmap resources as well as the image format plugins. In that area we use a patch for Qt to allow the access to these kind of resources which works nicely. If the same restrictions apply to CS would I be correct to assume that to comply with the LGPL we only would need to make the patches we applied available to out customers (if they are not accepted into the CS code base)?
- The QString implementation of CS seems to store Utf-8, we rely heavily on having no conversion penalty (both speed and memory fragmentation issues would make life hard for us) when interfacing with external Utf-16 requiring libraries. Are we able to use QString16 throughout our development somehow? Or do we have to convert from QString16 to QString whenever using GUI elements (our software deals with high volume text that is coming from outside sources)?
- We made some essential fixes to some of the collection classes of Qt-4 which are not fully 64-bit code compatible on the Windows platform (integers of the compiler are 32 bit whereas the Qt code incorrectly assumes 64 bit integers). We made the patches available to the Qt developers but to my knowledge we still must apply these patches to the current Qt-4 although our patches were made available to them well before Qt-4 was EOL’ed by Digia/Trolltech. Should these errors have made their way into CS what would be your stance regarding possibly accepting our patches into the main code base?
- Lastly, looking through the sources it seems that CS still allows for access to platform specific native window and bitmap handles - this access is mostly preveted by the new rendering pipeline in Qt-5 which was one of the major roadblocks we faced in porting to Qt-5. Am I correct in this regard?

Thanks for bearing with me - but if my initial impressions are correct then CS may be our best way forward now. We may need some help porting our build system from Qmake to CMake but first things first, viability of a port regarding the above issues is the main concern at the moment...
barbara
Posts: 447
Joined: Sat Apr 04 2015 2:32 am
Contact:

Re: A few questions regarding porting from Qt to CS

Post by barbara »

Thank you for your interest in the CopperSpice project. We understand some of your concerns and the team will do our very best to support the
migration from Qt4 to CS. We encourage you to read the following documentation.

https://www.copperspice.com/docs/cs_overview/cs-migration.html

Qt5 source were supposed to be only acceptable to be UTF-8 encoded.
I believe what you might be referring to is the default encoding for a string literal. Since UTF-8 is the "de facto" standard for storing text in almost all software and there needs to be some default, CS will assume all string literals are encoded as UTF-8.

Our focus is on CS so let's proceed in that direction. If you have source code which does not compile when linked with the CopperSpice libraries and you have what you believe is a string error simply create a small sample which repeats the problem. You can post a fragment here or let us know where we can find the code.

For reference, our DoxyPress application links with the CS libraries and it has been tested using many language outputs. If you have a use case that shows some area with an issue the CS team will will work with you to figure out the best solution.

access to pixmap resources was removed from programs which have no GUI . . . we use a patch for Qt to allow the access to these kind of resources which works nicely
We are not aware of any restrictions in this area and would need to see a small sample of your code to determine if you need your existing patch. This is something you should try with CS before applying any patches as it may not be a problem. If it turns out that you need to modify CS then all you need to do is distribute the CS library changes with your application.

If you believe these might benefit CopperSpice we are happy to review any pull request you wish to upstream.

QString implementation of CS seems to store Utf-8
In CopperSpice we made a major improvement by adding QString8 for UTF-8 and QString16 for UTF-16 support. For clarity the QString class in Qt is not UTF-16 but rather UCS-2.

You are free to create any QString16 you need and then pass this to a 3rdParty library. We strongly suggest you only use QString16 when needed and not all the time. Typically UTF-8 will be much faster since it uses half the memory. If you encounter issues we can go over specific sections and see if any improvements are necessary.

not fully 64-bit code compatible on the Windows platform
The CS container classes were fully redesigned and leverage the container classes provided by the C++ standard library. We have maintained the legacy API and also support the STL API. Our containers use "size_type" for all indexes. If your program is run on a 64-bit machine this data type will be 64-bits.

it seems that CS still allows for access to platform specific native window and bitmap handles
CopperSpice uses platform specific code as part of the GUI system. We suggest you create a small sample which shows where you need access to a native window. Without a specific use case we are not sure what problem you want to solve.

We may need some help porting our build system from Qmake to CMake
We have a sample project in our CS Overview documentation which shows the contrast between QMake and CMake. Please
check out "Qt Project Files" and "CopperSpice Project Using CMake".

https://www.copperspice.com/docs/cs_overview/main-sample-project.html

If you are not familiar with CMake we suggest looking at the build files for our working KitchenSink demo program.
From the following link you will find a link to the source code and the steps to build.

https://www.copperspice.com/docs/cs_overview/demo-ks.html

If you have questions about how to create build files for your project there are several options we can consider.

Barbara
CopperSpice Co-Founder
charlyw64
Posts: 2
Joined: Sat Mar 06 2021 5:54 pm

Re: A few questions regarding porting from Qt to CS

Post by charlyw64 »

Hello Barbara, thanks for your answer.

Unfortunately it seems my first impression was not all that it seemed to be.
barbara wrote: Mon Mar 08 2021 8:46 pm
QString implementation of CS seems to store Utf-8
In CopperSpice we made a major improvement by adding QString8 for UTF-8 and QString16 for UTF-16 support. For clarity the QString class in Qt is not UTF-16 but rather UCS-2.

You are free to create any QString16 you need and then pass this to a 3rdParty library. We strongly suggest you only use QString16 when needed and not all the time. Typically UTF-8 will be much faster since it uses half the memory. If you encounter issues we can go over specific sections and see if any improvements are necessary.
I can already say that any port of our software will require that QString is stored as a 16 bit entity with as few as possible (best none) 8-bit conversions happening. So if your QString16 relies on QString8 to for example facilitate comparisons or interfacing with GUI elements then we can forget about even attempting to port to CS, the resulting memory fragmentation (we already struggle in Qt because having a constant conversion of objects where a large object is replaced by a slightly smaller one results in a lot of allocated memory blocks of large size that are divided into several chunks to hold the smaller object and then no longer can accommodate the larger object, as even when freed the memory allocation routines in C++ will not join up adjacent free blocks) along with the time penalties incurred by such conversions would ruin any chances of keeping our software as performant as it is today.

Variable length character encodings have major drawbacks - yes, they require less storage but the computing time for simple things like string insertions or regexp matching are horrendous. So if I want to do things like QString8 ("abcdÄÖÜßabc").mid (8) you have quite complicated code which results in an order of a magnitude worse code than a 16 bit encoded unicode string. Now imagine that I have operations in our software where I want to get .mid (1000000, 16) (quite common for us). With Utf8 encoded text you can only iterate from the beginning of the text and must perform comparisons if you got a 2 or 4 byte encoded character and skip that character as a whole so instead of a direct address calculation you have a complicated iteration to perform to give me my result...

I will investigate the QString16 implementation in CS but along with the issue of Utf-8 encoded sources - which will result in incorrect code being generated by all compilers we tested - simply try to compile an UTF-8 encoded source containing an attempt to execute ::OutputDebugStringW (tr ("abcdÄÖÜßabcd").toUtf16 ()); and look at the code generated - depending on how wrong the code is, you'll either run into dropped characters at the end or buffer overruns. The chances of us being able to switch to CS are already remote because of these issues.

regards
barbara
Posts: 447
Joined: Sat Apr 04 2015 2:32 am
Contact:

Re: A few questions regarding porting from Qt to CS

Post by barbara »

We did our very best to reply to each and every comment from your original message. I understand you are very worried about what direction to take your product. The CS team is ready to work with you however your follow up comments felt like an attack and we want to try this again from the beginning.

There are several things in your comments that are misleading and we would like to clear them up. I encourage you read this reply so we can help you understand more about strings.

The idea of an encoding for your source code has nothing to do with the encoding of your strings. Neither CS nor Qt has any control over C++ compilers and how they parse keywords, identifiers, and the structure of your code. If you have a compiler error or the output is wrong let's look at it. Yes, this is a wonderful offer we are making and it is sincere.

The discussion about how strings are encoded is completely separate and unrelated to whatever compiler encoding is in use. CopperSpice currently supports two string encodings but you can add other encodings to CS with very little effort. Maybe this is what your code base needs. If you want to provide a good use case that shows where we need to improve CS we are all ears.

I can already say that any port of our software will require that QString is stored as a 16 bit entity with as few as possible (best none) 8-bit conversions happening. So if your QString16 relies on QString8 to for example facilitate comparisons or interfacing with GUI elements . . .
1) QString8 and QString16 classes in CopperSpice are separate classes and do not depend on each other.

2) Both QString8 and QString16 are variable length since UTF-8 and UTF-16 are variable length encodings.

3) Remember, QString in Qt is not UTF-16 but rather UCS-2 and does not handle surrogate pairs. This may be the reason why you are having issues compiling with Qt for your code base. There is no way to know without more information.

4) What we have seen in almost every singe use case is the string issue a developer is having "looks" like X but can easily be solved by changing Y. Rather than be angry with CS we suggested you put together a simple and small sample. We have a decent team with lots of experience in this area.

What if we have a terrific way to solve your issues? Would it not be worth it to see if CS can work for you?
with the issue of Utf-8 encoded sources - which will result in incorrect code being generated by all compilers we tested
When I hear that every compiler is wrong I wonder what problem a developer is really trying to solve. Since you said you are on Windows I will guess you are using MSVC. It sounds likely you converted your source code to UTF-8 but may have forgotten to pass the command line option to set the source encoding or vice-versa. Maybe this is not a Qt problem at all but they are simply unable to help you diagnose the issue. We have some really good MSVC users who would gladly bend over backwards to help you, if you want assistance.

Thank you and everyone else who is reading this post. It takes time to sort out the issue and then think about possible solutions. It is our goal to work with developers and all we ask is that you keep an open mind.

Barbara
janwilmans
Posts: 17
Joined: Wed Oct 16 2019 9:33 pm
Contact:

Re: A few questions regarding porting from Qt to CS

Post by janwilmans »

Excuse me for intruding on the conversation; just curious, what kind of application runs on windows, relies heavily with text-transformation performance and is also CPU resource limited? I would like to understand how encoding strings becomes a bottleneck....
janwilmans
Posts: 17
Joined: Wed Oct 16 2019 9:33 pm
Contact:

Re: A few questions regarding porting from Qt to CS

Post by janwilmans »

Also, quoted from google:
"UCS-2 is obsolete and replaced by UTF-16, which is more powerful, and more efficient (potentially fewer bytes for same number of characters). UCS-2 is fixed width, UTF-16 is variable width with a minimum of two bytes and a maximum of four bytes. UCS-2 and UTF-16 have identical code points for most characters."
barbara
Posts: 447
Joined: Sat Apr 04 2015 2:32 am
Contact:

Re: A few questions regarding porting from Qt to CS

Post by barbara »

As stated in my prior post, we are willing to test and work this out with you. In order to proceed and figure out the problem here is what we will need. If this is an issue in your code we will let you know. If this is an issue in CS, we will let you know what issue we are addressing.

1 A small example of C++ source that compiles

2 A build file using QMake or CMake

3 The result you are getting

4 The result you expected

Barbara
barbara
Posts: 447
Joined: Sat Apr 04 2015 2:32 am
Contact:

Re: A few questions regarding porting from Qt to CS

Post by barbara »

We have some new information about this thread which may be of value to other CopperSpice users.

# Issue One

If you are using MSVC and your locale is not set to UTF-8 you will need to add /utf-8 to your CXX Flags. This is required to set the compiler run time encoding to UTF-8.

Most C++ compilers already assume string literals are in UTF-8 by default. Setting the configuration flag will avoid encoding and translation issues.

# Issue Two

The other issue was related to the encoding of the the users source code files. They were saving their files in LATIN-1 and when running QCoreApplication::translate() translations were corrupted. Please ensure your files are saved with UTF-8 encoding.

Once the CS user made both of these changes translations started to work perfectly.

Barbara
Post Reply