• 0

C++ buffer to large


Question

Okay i have a buffer

Char *buffer = new char[1024];

 

 

I need to resize this buffer from 1024, i would like to do this WITHOUT copying to another buffer.

Setting the buffer to the value needed is not an option.

 

Any ideas?

Link to comment
Share on other sites

11 answers to this question

Recommended Posts

  • 0

Okay i have a buffer

Char *buffer = new char[1024];

 

I need to resize this buffer from 1024, i would like to do this WITHOUT copying to another buffer.

Setting the buffer to the value needed is not an option.

 

Any ideas?

#include <stdlib.h>
void
main ( void ) {

    char *mybuf = malloc ( 1024 );
    
    /* do some work with mybuf */
    
    /* expand buffer size */
    mybuf = realloc ( mybuf, 2048 );
}
A real world example where I use the same technique for an expandable line buffer (non-reentrant function):

PRIVATE char *
fs_fgetln ( FILE *stream )
{
    static char     *buf = NULL;
    static size_t   size = FGET_SIZE;
    size_t          tmp;
    char            ch, *c;

    if ( !buf )
        buf = malloc ( FGET_SIZE );
    
    c   = buf;
    ch  = fgetc ( stream );
    
    while ( '\n' != ch && EOF != ch ) {
                
        if ( size < ( c + 1 ) - buf ) { 
            
            tmp     = c - buf;
            size    += FGET_SIZE;  
            buf     = realloc ( buf, size );
            c       = buf + tmp;
        }

        *c++ = ch;               
        ch = fgetc ( stream );  
    } 
        
    *c = '\0';
    
    return '\n' == ch ? buf : NULL;
}    
For documentation:

$ man malloc
  • Like 1
Link to comment
Share on other sites

  • 0

It's worth noting that you can't "resize" an array in any language, unless you over-allocate it and extend into the over-allocated space.

 

Sounds like you need an std::vector, or an std::deque.

 

If you need to use an array, you have two options:

 

  • Create a new array, copy the contents, and delete the old array.
  • Over-specify the original array, create an unsigned integer that stores the size, and grow the array as you need to.

Of all the above options though, I'd just use a vector. You get all the performance of an array, with the ability to easily resize the underlying array as required.

Link to comment
Share on other sites

  • 0

Note that there's no guarantee realloc won't copy your buffer internally and return a pointer to a new memory location. Also,

buf = realloc(buf, newsize);
causes a memory leak if realloc fails because now "buf" is null and you lost your reference to your original buffer. A safer way would be:

char* reallocatedBuffer = realloc(originalBuffer, newsize);
if (reallocatedBuffer)
{
    originalBuffer = reallocatedBuffer;
}
else
{
    free(originalBuffer);
    originalBuffer = nullptr;
}
  • Like 1
Link to comment
Share on other sites

  • 0

Note that there's no guarantee realloc won't copy your buffer internally and return a pointer to a new memory location.

As it states in the documentation. However, an architecture specific standard library will be better optimised for such operations than ordinary code. Furthermore, in many cases, the operation can be reduced down to a single cpu instruction. SSE2 auto vectorisation for example means any copying is done in highly optimised chunks with a minimal number of instruction calls.

 

Also,

buf = realloc(buf, newsize);
causes a memory leak if realloc fails because now "buf" is null and you lost your reference to your original buffer. A safer way would be:

char* reallocatedBuffer = realloc(originalBuffer, newsize);
if (reallocatedBuffer)
{
    originalBuffer = reallocatedBuffer;
}
else
{
    free(originalBuffer);
    originalBuffer = nullptr;
}
If realloc fails, it means the system is out of memory. There's little a programmer can do about that. I'd never write that kind of pessimistic code unless I'm allocating huge amounts of memory.
Link to comment
Share on other sites

  • 0

As it states in the documentation. However, an architecture specific standard library will be better optimised for such operations than ordinary code. Furthermore, in many cases, the operation can be reduced down to a single cpu instruction. SSE2 auto vectorisation for example means any copying is done in highly optimised chunks with a minimal number of instruction calls.

Sure, but so is memcpy. My point is that the OP asked for a way to resize an array without copying, and while realloc may avoid it in many cases it offers no guarantee of that.

 

If realloc fails, it means the system is out of memory. There's little a programmer can do about that. I'd never write that kind of pessimistic code unless I'm allocating huge amounts of memory.

Why not just create a little wrapper function with that recovery code like safe_realloc and use that instead? It's not any more difficult to use. If as you say there's nothing you can do then the "recovery" could consist in crashing the application immediately with some specific return code. At least you know where things went wrong instead of letting the program continue with a null pointer and a leak that might not cause any immediate problem but cause more weird issues down the line that will be hard to track. Even with an OOM the program may continue running in an undefined state for a long time, and the program may crash with a seemingly unrelated issue in some remote part of the code. Perhaps the OOM is due to the wrong size being passed to realloc etc. The earlier you react to the issue the easier it is to understand what went wrong.
Link to comment
Share on other sites

  • 0

Sure, but so is memcpy. My point is that the OP asked for a way to resize an array without copying, and while realloc may avoid it in many cases it offers no guarantee of that.

Perhaps I misinterpreted his question. I thought he meant without having to manually copy from one buffer to another. You're right in the fact that realloc may perform a copy behind the scenes. I don't think there's a way to entirely avoid that if one wants to maintain a contiguous block of memory such as an array.

 

Why not just create a little wrapper function with that recovery code like safe_realloc and use that instead? It's not any more difficult to use. If as you say there's nothing you can do then the "recovery" could consist in crashing the application immediately with some specific return code. At least you know where things went wrong instead of letting the program continue with a null pointer and a leak that might not cause any immediate problem but cause more weird issues down the line that will be hard to track.

I'm not wholly against such an idea if someone wants to do that. However, personally, I don't see much of a difference either way because the program will eventually crash when it tries to access the invalid memory. The system will become unstable generally and anything the programmer does to recover will likely require more memory allocations.

 

My one exception to that is when I'm doing large allocations. That's clearly a situation where OOM is expected. Apart from that I take an optimistic approach.

Perhaps the OOM is due to the wrong size being passed to realloc etc.

That's a programmer error and should be corrected before release.
Link to comment
Share on other sites

  • 0

I'm not wholly against such an idea if someone wants to do that. However, personally, I don't see much of a difference either way because the program will eventually crash when it tries to access the invalid memory. The system will become unstable generally and anything the programmer does to recover will likely require more memory allocations.

That's a programmer error and should be corrected before release.

The difference is that if the problem is indeed attributable to programmer error, then doing something rather than nothing on allocation failures allows you to track the problem quickly. Otherwise the problem may manifest itself in ways that don't obviously point to an OOM having happened at that particular point in the program. You'll waste time tracking what the real problem is before even beginning to wonder what could be done to address it. Perhaps the answer is indeed "nothing", but at least you know what's going on. It's an insignificant price to pay in development time for a potentially large reward in debugging time. A tradeoff I would always take.
Link to comment
Share on other sites

  • 0

Perhaps the answer is indeed "nothing", but at least you know what's going on. It's an insignificant price to pay in development time for a potentially large reward in debugging time. A tradeoff I would always take.

If it helps someone debug their code I'm not adverse to the idea. Perhaps a bit of preprocessor magic would do the trick:

 

#ifdef DEBUG
  #define safe_realloc debug_realloc
#else
  #define safe_realloc realloc
#endif 
Or something like that. Valgrind would be my preference though :)
Link to comment
Share on other sites

  • 0

There is a way to avoid potential copies when dealing with an array of large data types / structures / objects, and that's by using an array of pointers. The only contiguous block of memory that's needed is for the pointer array itself. It complicates management somewhat, but it could make resizing more efficient in certain cases.

For example:

#include <stdlib.h>

typedef struct {

	int 	a;
	double 	b;
	char 	c[128];

} ComplexType;

void
main ( void ) {
	
	size_t i;
	
	/* initialise pointer array */
	size_t count = 20;
	ComplexType **array = malloc ( sizeof ( ComplexType* ) * count );

	for ( i = 0; i < count; i++ )
		array [i] = malloc ( sizeof ( ComplexType ) );

	/* expand pointer array */
	count += 40;
	array = realloc ( array, sizeof ( ComplexType* ) * count );

	for ( ; i < count; i++ )
		array [i] = malloc ( sizeof ( ComplexType ) );

	/* free pointer array */
	for ( i = 0; i < count; i++ )
		free ( array [i] );

	free ( array );
}
Link to comment
Share on other sites

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.