qdrawhelper.cpp bug - casting away const

Discuss anything related to product development
Post Reply
seasoned_geek
Posts: 253
Joined: Thu Jun 11 2020 12:18 pm

qdrawhelper.cpp bug - casting away const

Post by seasoned_geek »

All,

I've been tracking down a dump happening when using QPrintPreviewDialog with CsScintilla and color backgrounds. Really been honking me off because it is quasi-random. I trimmed the stuff pasted below.

Code: Select all

(gdb) bt full
#0  malloc_consolidate (av=av@entry=0x7fffef4c6b80 <main_arena>) at malloc.c:4475
        idx = <optimized out>
        fb = 0x7fffef4c6b90 <main_arena+16>
        maxfb = 0x7fffef4c6bd8 <main_arena+88>
        p = 0x2080900090801f4
        nextp = <optimized out>
        unsorted_bin = 0x7fffef4c6be0 <main_arena+96>
        first_unsorted = <optimized out>
        nextchunk = <optimized out>
        size = <optimized out>
        nextsize = <optimized out>
        prevsize = <optimized out>
        nextinuse = <optimized out>
#1  0x00007fffef371c83 in _int_malloc (av=av@entry=0x7fffef4c6b80 <main_arena>, bytes=bytes@entry=14016) at malloc.c:3699
        nb = <optimized out>
        idx = 113
        bin = <optimized out>
        victim = <optimized out>
        size = <optimized out>
        victim_index = <optimized out>
        remainder = <optimized out>
        remainder_size = <optimized out>
        block = <optimized out>
        bit = <optimized out>
        map = <optimized out>
        fwd = <optimized out>
        bck = <optimized out>
        tcache_unsorted_count = <optimized out>
        tcache_nb = <optimized out>
        tc_idx = <optimized out>
        return_cached = <optimized out>
        __PRETTY_FUNCTION__ = "_int_malloc"
#2  0x00007fffef375b95 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:3428
        av = <optimized out>
        oldtop = 0x5555565fb4d0
        p = <optimized out>
        sz = 14016
        csz = <optimized out>
        oldtopsize = 43824
        mem = <optimized out>
        clearsize = <optimized out>
        nclears = <optimized out>
        d = <optimized out>
        bytes = 14016
        hook = <optimized out>
        __PRETTY_FUNCTION__ = "__libc_calloc"
#3  0x00007ffff4c60837 in QClipData::initialize (this=0x5555565dccd0) at /home/roland/Projects/copperspice/src/gui/painting/qpaintengine_raster.cpp:3691
No locals.
#4  0x00007ffff4b8203a in qt_alphamapblit_uint32 (rasterBuffer=0x5555560acf50, x=45, y=758, color=4289309097, map=0x555556154c80 "", mapWidth=8, mapHeight=9, mapStride=8, clip=0x5555565dccd0) at /home/roland/Projects/copperspice/src/gui/painting/qdrawhelper.cpp:6302
        bottom = 767
        top = 758
        c = 4289309097
        destStride = 770
#5  0x00007ffff4b82251 in qt_alphamapblit_argb32 (rasterBuffer=0x5555560acf50, x=45, y=758, color=..., map=0x555556154c80 "", mapWidth=8, mapHeight=9, mapStride=8, clip=0x5555565dccd0) at /home/roland/Projects/copperspice/src/gui/painting/qdrawhelper.cpp:6351
No locals.
#6  0x00007ffff4c5bd75 in QRasterPaintEngine::alphaPenBlt (this=0x555556159430, src=0x555556154c80, bpl=8, depth=8, rx=45, ry=758, w=8, h=9) at /home/roland/Projects/copperspice/src/gui/painting/qpaintengine_raster.cpp:2555
        d = 0x5555560b4250
        s = 0x5555565dd820
        rb = 0x5555560acf50
        rect = {m_x1 = 45, m_y1 = 758, m_x2 = 52, m_y2 = 766}
        clip = 0x5555565dccd0
        unclipped = false
        blend = 0x7ffff4c61b37 <qt_span_fill_clipRect(int, QSpan const*, void*)>
        scanline = 0x555556154c80 ""
        x0 = -46576
        y0 = 32767
        NSPANS = -46512
        spans = {{x = 0, len = 0, y = 0, coverage = 0 '\000'}, {x = -2720, len = 22108, y = 21845, coverage = 0 '\000'}, {x = 19120, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -31251, len = 62546, y = 32767, coverage = 0 '\000'}, {x = 19136, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -2720, len = 22108, y = 21845, coverage = 0 '\000'}, {x = 19152, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 25088, len = 11745, y = 14696, coverage = 233 '\351'}, {x = 19184, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 19264, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 19184, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 13982, len = 62651, y = 32767, coverage = 0 '\000'}, {x = 4, len = 0, y = 0, coverage = 0 '\000'}, {x = 19264, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 19232, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 21191, len = 62548, y = 32767, coverage = 0 '\000'}, {x = 19232, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 19264, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 19264, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 3, len = 0, y = 0, coverage = 0 '\000'}, {x = 19312, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -5784, len = 62716, y = 32767, coverage = 0 '\000'}, {x = 14096, len = 22111, y = 19968, coverage = 0 '\000'}, {x = -29792, len = 22072, y = 21845, coverage = 0 '\000'}, {x = -10608, len = 63264, y = 32767, coverage = 0 '\000'}, {x = 0, len = 62662, y = 32767, coverage = 0 '\000'}, {x = 0, len = 0, y = 0, coverage = 0 '\000'}, {x = 6613, len = 62650, y = 32767, coverage = 0 '\000'}, {x = 19344, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 255, len = 0, y = 14, coverage = 0 '\000'}, {x = 29056, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -5752, len = 59391, y = 32767, coverage = 0 '\000'}, {x = 19776, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -29792, len = 22072, y = 21845, coverage = 0 '\000'}, {x = 19776, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -14317, len = 62661, y = 32767, coverage = 0 '\000'}, {x = 19488, len = 65535, y = 633, coverage = 0 '\000'}, {x = -29792, len = 22072, y = 21845, coverage = 0 '\000'}, {x = 21280, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 20240, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 10584, len = 22074, y = 64, coverage = 0 '\000'}, {x = 25888, len = 22107, y = 21845, coverage = 0 '\000'}, {x = -3449, len = 51738, y = 2, coverage = 0 '\000'}, {x = 64, len = 0, y = 1600, coverage = 122 'z'}, {x = 19480, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 19460, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 19504, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 25655, len = 62717, y = 32767, coverage = 0 '\000'}, {x = 256, len = 0, y = 64, coverage = 0 '\000'}, {x = -10208, len = 22109, y = 21845, coverage = 0 '\000'}, {x = -16464, len = 22109, y = 21845, coverage = 0 '\000'}, {x = 64, len = 0, y = 64, coverage = 0 '\000'}, {x = 20240, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 25088, len = 11745, y = 14696, coverage = 233 '\351'}, {x = 19536, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -29536, len = 22072, y = 21845, coverage = 0 '\000'}, {x = 19680, len = 65535, y = 64, coverage = 0 '\000'}, {x = 20224, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 19776, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -15089, len = 62716, y = 32767, coverage = 0 '\000'}, {x = 49, len = 0, y = 21845, coverage = 0 '\000'}, {x = 0, len = 0, y = 0, coverage = 0 '\000'}, {x = 21264, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 20224, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 20048, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 19888, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 23976, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -29432, len = 22045, y = 21845, coverage = 0 '\000'}, {x = -29428, len = 22045, y = 21845, coverage = 0 '\000'}, {x = 20356, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 19664, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -17607, len = 62688, y = 32767, coverage = 0 '\000'}, {x = -29432, len = 22045, y = 21845, coverage = 0 '\000'}, {x = 20352, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 19712, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -30803, len = 62689, y = 32767, coverage = 0 '\000'}, {x = 19712, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -29432, len = 22045, y = 21845, coverage = 0 '\000'}, {x = 20352, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -20424, len = 22110, y = 21845, coverage = 0 '\000'}, {x = 19776, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 31577, len = 62689, y = 32767, coverage = 0 '\000'}, {x = 19776, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -29440, len = 22045, y = 21845, coverage = 0 '\000'}, {x = 3526, len = 31802, y = 0, coverage = 0 '\000'}, {x = 20352, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -20424, len = 22110, y = 21845, coverage = 0 '\000'}, {x = -20424, len = 22110, y = 21845, coverage = 0 '\000'}, {x = 19840, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 25564, len = 62689, y = 32767, coverage = 0 '\000'}, {x = -29440, len = 22045, y = 21845, coverage = 0 '\000'}, {x = 3526, len = 31802, y = 0, coverage = 0 '\000'}, {x = 20352, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -20424, len = 22110, y = 21845, coverage = 0 '\000'}, {x = -29432, len = 22045, y = 21845, coverage = 0 '\000'}, {x = -20424, len = 22110, y = 21845, coverage = 0 '\000'}, {x = 19904, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 14887, len = 62689, y = 32767, coverage = 0 '\000'}, {x = 3526, len = 31802, y = 0, coverage = 0 '\000'}, {x = 20352, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 119, len = 0, y = 0, coverage = 0 '\000'}, {x = -20424, len = 22110, y = 21845, coverage = 0 '\000'}, {x = 20880, len = 22045, y = 21845, coverage = 0 '\000'}, {x = -29440, len = 22045, y = 21845, coverage = 0 '\000'}, {x = 19968, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 6640, len = 62689, y = 32767, coverage = 0 '\000'}, {x = -29440, len = 22045, y = 21845, coverage = 0 '\000'}, {x = 20008, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 19968, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -29432, len = 22045, y = 21845, coverage = 0 '\000'}, {x = 19984, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 6738, len = 62689, y = 32767, coverage = 0 '\000'}, {x = 20000, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -29432, len = 22045, y = 21845, coverage = 0 '\000'}, {x = 20016, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -3038, len = 62688, y = 32767, coverage = 0 '\000'}, {x = 20048, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 8, len = 0, y = 0, coverage = 0 '\000'}, {x = 20032, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 1248, len = 62486, y = 32767, coverage = 0 '\000'}, {x = 20096, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 1322, len = 62486, y = 32767, coverage = 0 '\000'}, {x = 0, len = 0, y = 0, coverage = 0 '\000'}, {x = 20144, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 0, len = 0, y = 0, coverage = 0 '\000'}, {x = 20160, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 20144, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 20224, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 20192, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -13993, len = 62485, y = 32767, coverage = 0 '\000'}, {x = 20352, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 0, len = 0, y = 0, coverage = 0 '\000'}, {x = 25200, len = 63298, y = 32767, coverage = 0 '\000'}, {x = 20224, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 23600, len = 21963, y = 21845, coverage = 0 '\000'}, {x = 23600, len = 21963, y = 21845, coverage = 0 '\000'}, {x = 24112, len = 21963, y = 21845, coverage = 0 '\000'}, {x = 23544, len = 21963, y = 21845, coverage = 0 '\000'}, {x = 20208, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 20224, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 20272, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -32530, len = 62485, y = 32767, coverage = 0 '\000'}, {x = 0, len = 0, y = 0, coverage = 0 '\000'}, {x = 25200, len = 63298, y = 32767, coverage = 0 '\000'}, {x = 23600, len = 21963, y = 21845, coverage = 0 '\000'}, {x = 23600, len = 21963, y = 21845, coverage = 0 '\000'}, {x = 24112, len = 21963, y = 21845, coverage = 0 '\000'}, {x = 23544, len = 21963, y = 21845, coverage = 0 '\000'}, {x = -29432, len = 22045, y = 21845, coverage = 0 '\000'}, {x = 25088, len = 11745, y = 14696, coverage = 233 '\351'}, {x = 20304, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -5475, len = 62484, y = 32767, coverage = 0 '\000'}, {x = 0, len = 0, y = 0, coverage = 0 '\000'}, {x = 25184, len = 63298, y = 32767, coverage = 0 '\000'}, {x = 20560, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 12337, len = 62484, y = 32767, coverage = 0 '\000'}, {x = 0, len = 0, y = 0, coverage = 0 '\000'}, {x = 25184, len = 63298, y = 32767, coverage = 0 '\000'}, {x = 20384, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -29440, len = 22045, y = 21845, coverage = 0 '\000'}, {x = 3526, len = 31802, y = 0, coverage = 0 '\000'}, {x = 20784, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -20424, len = 22110, y = 21845, coverage = 0 '\000'}, {x = -20424, len = 22110, y = 21845, coverage = 0 '\000'}, {x = 20448, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 25564, len = 62689, y = 32767, coverage = 0 '\000'}, {x = -29440, len = 22045, y = 21845, coverage = 0 '\000'}, {x = 3526, len = 31802, y = 0, coverage = 0 '\000'}, {x = 20784, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -20424, len = 22110, y = 21845, coverage = 0 '\000'}, {x = 32144, len = 22045, y = 21845, coverage = 0 '\000'}, {x = -2616, len = 22108, y = 21845, coverage = 0 '\000'}, {x = 20480, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 1148, len = 21878, y = 21845, coverage = 0 '\000'}, {x = 20496, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -2616, len = 22108, y = 21845, coverage = 0 '\000'}, {x = 20512, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 16914, len = 21878, y = 21845, coverage = 0 '\000'}, {x = 20528, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -2624, len = 22108, y = 21845, coverage = 0 '\000'}, {x = 20544, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 4464, len = 21878, y = 21845, coverage = 0 '\000'}, {x = 20576, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -2624, len = 22108, y = 21845, coverage = 0 '\000'}, {x = 20576, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -17930, len = 63403, y = 32767, coverage = 0 '\000'}, {x = 20576, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -2624, len = 22108, y = 21845, coverage = 0 '\000'}, {x = 20608, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -17866, len = 63403, y = 32767, coverage = 0 '\000'}, {x = 0, len = 0, y = 0, coverage = 0 '\000'}, {x = -2752, len = 22108, y = 21845, coverage = 0 '\000'}, {x = 20656, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -23704, len = 21882, y = 21845, coverage = 0 '\000'}, {x = 184, len = 0, y = 0, coverage = 0 '\000'}, {x = -2752, len = 22108, y = 21845, coverage = 0 '\000'}, {x = 22064, len = 65535, y = 32767, coverage = 0 '\000'}, {x = -2752, len = 22108, y = 1, coverage = 0 '\000'}, {x = 20752, len = 65535, y = 32767, coverage = 0 '\000'}, {x = 19656, len = 62548, y = 32767, coverage = 0 '\000'}...}
        current = -195918149
        x1 = 32767
        y1 = 1448932704
Looking into #4 above is where I was completely gobsmacked.

Code: Select all

static void qt_alphamapblit_uint32(QRasterBuffer *rasterBuffer,
   int x, int y, quint32 color,
   const uchar *map,
   int mapWidth, int mapHeight, int mapStride,
   const QClipData *clip)
{
   const quint32 c = color;
clip is const

then at line 6302

Code: Select all

const_cast<QClipData *>(clip)->initialize();
That gets us to #3 from above.

Code: Select all

void QClipData::initialize()
{
    if (m_spans) {
      return;
   }

   if (!m_clipLines) {
      m_clipLines = (ClipLine *)calloc(sizeof(ClipLine), clipSpanHeight);
   }

   Q_CHECK_PTR(m_clipLines);

It's difficult to track this issue down because there is no documentation for QClipData. This must be code that was carried over from Qt and has yet to be touched, correct? We declare a pointer const then cast it away so we can allocate RAM?

From what I can parcel out, if sz is the element size: 14016 and oldtopsize is the count: 43824 that's 614237184 bytes.

According to stdint.h

Code: Select all

/* Limit of `size_t' type.  */
# if __WORDSIZE == 64
#  define SIZE_MAX      (18446744073709551615UL)
# else
#  if __WORDSIZE32_SIZE_ULONG
#   define SIZE_MAX     (4294967295UL)
#  else
#   define SIZE_MAX     (4294967295U)
#  endif
# endif

So, even the compiler somehow thought I was 32-bit, I should be able to get that many bytes without

Code: Select all

Segmentation fault (core dumped)
My gut tells me casting away the const caused this problem. 586MEG isn't that much on a system with 24GB of RAM and the code didn't hit the trap for a calloc failure that would have returned a null pointer.

Other than, possibly the only interesting discussion to ever be posted on Stack Overflow,
https://stackoverflow.com/questions/3100193/segfaults-in-malloc-and-malloc-consolidate

I cannot find a prime suspect that does not point a finger back to casting away const.
Post Reply