c++ need help understanding..

[Richard Grant]

Okay so i am reading over this procedural code at: (for a multi server)

http://www.cplusplus.com/forum/unices/116977/

 

The code is some what similar to what i have come up with except they are using threads to connect to multiple clients.

 

From the code:

pthread_t threadA[3];

....

int noThread = 0; 
while (noThread < 3) // <- here
    {
        cout << "Listening" << endl;

        //this is where client connects. svr will hang in this mode until client conn
        connFd = accept(listenFd, (struct sockaddr *)&clntAdd, &len);

        if (connFd < 0)
        {
            cerr << "Cannot accept connection" << endl;
            return 0;
        }
        else
        {
            cout << "Connection successful" << endl;
        }
        
        pthread_create(&threadA[noThread], NULL, task1, NULL); // <- here
        
        noThread++;
    }
    
    for(int i = 0; i < 3; i++)// <- here
    {
        pthread_join(threadA[i], NULL); // <- here
    }

it looks as if they are only setting up 3 threads to only allow 3 different clients? Is that so?

 

if so what would you do to make it keep accepting code?

(pseudo code is fine I just need an explanation)

[simplezz]

There were a few things in the code that had no documention so its not very clear what is happening

 

Here is a quick list:

What is F_RESULT, NET_EVENT, NET_EVENT_READ, NET_EVENT_WRIT, EUTIL_SET_ERROR, E_SYS_ERROR?

 

Some of the types a little obvious but what are the actual values? Documention links would also be AMAZING!

Those are just conveniences for handling things like return values, testing a specific type of socket event, and general error handling / propagation.

typedef enum { F_SUCCESS, F_FAILURE } F_RESULT;
typedef enum { NET_EVENT_READ, NET_EVENT_WRITE } NET_EVENT;

I wouldn't concern yourself with those. It's the FD_SET, select, and FD_ISSET that are important.

All my code is written against the Linux kernel. So if you want documentation for system functions, look here. Windows functions may differ, but there's generally an equivalent.

 

The whole point of that function I posted was to test that an event was available so other code can call the requisite socket functions without blocking.

F_RESULT
network_read ( t_network *net, t_network_buffer *rb, int timeout )
{
    int          result;
    t_boolean    has_event;

    if ( rb->max_len == rb->used ) {
        EUTIL_SET_ERROR ( NETWORK_E_BUF_FULL );
        return F_FAILURE;
    }

    if ( F_FAILURE == network_has_event ( net->socket, NET_EVENT_READ, timeout, &has_event ) )
        return F_FAILURE;

    if ( !has_event )
        return F_SUCCESS;

    result = recv ( net->socket, rb->buffer + rb->used, rb->max_len - rb->used, 0 );
    
    if ( -1 == result && EINTR != errno ) {    

        EUTIL_SET_ERROR ( E_SYS_ERROR );
        return F_FAILURE;

    } else if ( 0 == result ) {

        EUTIL_SET_ERROR ( NETWORK_E_REMOTE_SHUTDOWN );
        return F_FAILURE;
    }

    rb->used += result;
    
    return F_SUCCESS;
}

[Richard Grant]

What i am thinking is make a seperate thread for excepting connections and store those connections in a list/array (pref. a list since i wont know how many people will be connecting) the data would look like this in the list..

Structure connection{
sockaddr_in client_socket
char* buffer
Boolean connected
}

And i would make a thread like...

[THREAD]
{//Add connections to list
...
}

[THREAD]
{//Do stuff for each (client) connection in list
...
}

[simplezz]

It's using threads to read data from the socket of each connection, the actual connection is established in the main thread.

If you want to remove the limitation of how many connections it will accept, you need to make the fixed sized array threadA expandable or into some kind of other data structure like a list where you can just push/add a thread whenever a new connection comes in. Then when you join the threads, you need to iterate that entire array/list:

 

    for ( int i = 0; i < thread_count; i++ )
        pthread_join ( threadA[i], NULL );

The main loop condition would then turn into:

while ( 1 ) {
...
}

You might want to use a signal flag to terminate it instead. So when your app receives a SIG_TERM or the like, you can do cleanup and gracefully exit.

[Richard Grant]

It's using threads to read data from the socket of each connection, the actual connection is established in the main thread.

If you want to remove the limitation of how many connections it will accept, you need to make the fixed sized array threadA expandable or into some kind of other data structure like a list where you can just push/add a thread whenever a new connection comes in. Then when you join the threads, you need to iterate that entire array/list:

 

    for ( int i = 0; i < thread_count; i++ )
        pthread_join ( threadA[i], NULL );

The main loop condition would then turn into:

while ( 1 ) {
...
}

You might want to use a signal flag to terminate it instead. So when your app receives a SIG_TERM or the like, you can do cleanup and gracefully exit.

 

 

And i would make it scan for connections on a seperate thread right? (not the main thread)

[simplezz]

What i am thinking is make a seperate thread for excepting connections and store those connections in a list/array (pref. a list since i wont know how many people will be connecting)

It depends what operations you want it to do. If it's only adding to the end of it, then a simple resizeable array would be very fast. Either a custom one or if you're going the C++ route, something like a vector. On the other hand, if you plan on doing deletions from anywhere other than the beginning or end, then a list would be more appropriate.

[simplezz]

And i would make it scan for connections on a seperate thread right? (not the main thread)

That depends on what your main thread is doing in the interim. If its only job is to accept connections and delegate their maintenance to sub threads, then you can probably leave it as it is.

One suggestion I would make though, providing you want your app to be responsive (and possibly doing other things as well as accepting connections), is to consider employing select as a means to determine if a listening socket has a connection waiting to be accepted. That allows your main loop to do other things (such as checking for a terminate flag) rather than just blocking forever waiting for a request.

If you do decide to put the listener in its own thread, then select would allow it to terminate in a controlled manner.

[simplezz]

Another thing to consider is that a thread per socket isn't the most efficient use of resources. Again, select can be run on an array of fds, then if a long task needs to be performed when data arrives, fire off a thread.

[Richard Grant]

That depends on what your main thread is doing in the interim. If its only job is to accept connections and delegate their maintenance to sub threads, then you can probably leave it as it is.

One suggestion I would make though, providing you want your app to be responsive (and possibly doing other things as well as accepting connections), is to consider employing select as a means to determine if a listening socket has a connection waiting to be accepted. That allows your main loop to do other things (such as checking for a terminate flag) rather than just blocking forever waiting for a request.

 

I didn't know Select existed http://msdn.microsoft.com/en-us/library/windows/desktop/ms740141%28v=vs.85%29.aspx

 

 

I think i will put the accept client connection in a seperate thread because the loop that my read and send commands are in freese the loop waiting for inputs.

 

By the way thank you for the select command information!

[Richard Grant]

Wait is select only for winsock?  cant use winsock because it doesnt work on linux so i am using

#include <sys/socket.h>

Did some researching sys/socket includes select

[Lant]

If this is just a Windows specific application you might want to look at IO Completion Ports to implement the client communication processing in an asynchronous fashion instead of having multiple threads per client. Sadly this is one area where the Linux/POSIX API isn't is feature rich as the Windows API and to get a similar level of functionality you will need to roll your own.

(Although if there is now a decent kernel API similar to this I would love to hear about it).

 

There is also Boost.ASIO for a cross platform way to implement async IO.

[simplezz]

Wait is select only for winsock?  cant use winsock because it doesnt work on linux so i am using

#include <sys/socket.h>

Did some researching sys/socket includes select

Unfortunately as Andre recently informed me, C++11 doesn't support POSIX sockets. On linux, you need to include <sys/select.h>. Alternatively, you could use something like boost which works on most platforms and has a consistent sockets interface.

Another option is conditional compilation for the various OS-specific function calls and includes.

[Richard Grant]

Unfortunately as Andre recently informed me, C++11 doesn't support POSIX sockets. On linux, you need to include <sys/select.h>. Alternatively, you could use something like boost which works on most platforms and has a consistent sockets interface.

Another option is conditional compilation for the various OS-specific function calls and includes.

 

Unfortunately as Andre recently informed me, C++11 doesn't support POSIX sockets. On linux, you need to include <sys/select.h>. Alternatively, you could use something like boost which works on most platforms and has a consistent sockets interface.

Another option is conditional compilation for the various OS-specific function calls and includes.

 

I am fine with the OS specific fuction calls and includes

If this is just a Windows specific application you might want to look at IO Completion Ports to implement the client communication processing in an asynchronous fashion instead of having multiple threads per client. Sadly this is one area where the Linux/POSIX API isn't is feature rich as the Windows API and to get a similar level of functionality you will need to roll your own.

(Although if there is now a decent kernel API similar to this I would love to hear about it).

 

There is also Boost.ASIO for a cross platform way to implement async IO.

3rd party libraries are out of the question

[simplezz]

sadly this is one area where the Linux/POSIX API isn't is feature rich as the Windows API and to get a similar level of functionality you will need to roll your own.

Pretty sure POSIX AIO (Asynchronous IO) does the same thing as Microsoft's IO completion ports.

http://man7.org/linux/man-pages/man7/aio.7.html

Personally, I prefer select or epoll though. And that's what the major server software use too.

[Richard Grant]

Extra info:

Pretty sure POSIX AIO (Asynchronous IO) does the same thing as Microsoft's IO completion ports.
http://man7.org/linux/man-pages/man7/aio.7.html

Personally, I prefer select or epoll though. And that's what the major server software use too.

 
yeah i personally like the select, im trying to understand what parameters need to be passed.. Its a bit confusing. Would love to have some light shed on it. :) maybe some code? ...lets start with:

 
http://stackoverflow.com/questions/13086085/c-select-only-checks-last-client-socket

while (1) {
    FD_ZERO(&readfds); //I assume this means to set readfds memory as zero
    FD_SET(sockfd, &readfds); // not sure what this does
    highsock = sockfd;
    for (list<int>::iterator it = CliSocks.begin(); it != CliSocks.end(); it++) {
        FD_SET(*it, &readfds); //not to sure what this does
        highsock = *it > highsock ? *it : highsock; //set highsock to the highest value of it and highsock
    }
/**
highsock is set to the socket to prevent modification
highsock is added to 1 to represent the next available socket (not to sure if +1 is needed
readfds is modified in select so we have to reset the value in the loop
the null parameters i have no clue.
**/
    int sockcount = select(highsock + 1, &readfds, NULL, NULL, NULL);

    ...
}

I can not understand how to use this function, i would like some light to be shed on what is happening here and what variables are doing what.

[simplezz]

yeah i personally like the select, im trying to understand what parameters need to be passed.. Its a bit confusing. Would love to have some light shed on it. :) maybe some code? ...lets start with:

I can't speak for the Windows implementation, but on Linux, the crux of it is, you're trying to determine if there's a read or write event available on a particular socket(s). To achieve that end, you fill in an fd_set structure with the sockets in question and pass them as either a read or write parameter. That's what FD_* class of functions/macros are for.

A simple example:

PRIVATE F_RESULT 
network_has_event ( int socket, NET_EVENT src, int timeout, t_boolean *result ) 
{
	fd_set 		 fds;
	struct timeval	 tv;
	int 		 state;
	
	*result = false;
	
	FD_ZERO ( &fds );
	FD_SET ( socket, &fds );
	tv.tv_sec   = ( SECOND <= timeout ) ? timeout / SECOND : 0;
	tv.tv_usec  = ( timeout % SECOND ) * SECOND;
	
	state = select ( socket + 1, ( NET_EVENT_READ == src ) ? &fds : NULL, ( NET_EVENT_WRITE == src ) ? &fds : NULL, NULL, &tv );
	
	if ( -1 == state && EINPROGRESS != errno && EINTR != errno ) {

        	EUTIL_SET_ERROR(E_SYS_ERROR);
		return F_FAILURE;

    	} else if ( 0 < state && FD_ISSET ( socket, &fds ) ) {
        
        	*result = true;
	}
    
	return F_SUCCESS;
}

This only checks a single socket for a read or write event. But it can be scaled to include a set of fds. Once you know that an event is waiting, you can proceed to call the relevant socket function, such as accept, read, write, etc. All this can be done in a non-blocking manner.

[Nick H. Supervisor]

<Threads merged>

While separate threads for separate discussions are fine (and encouraged,) having both the socket select question and this thread seems unnecessary. I suggest both conversations can continue in this one thread. ;) (Y)

[Richard Grant]

More Information:

socket() - If no error occurs, socket returns a descriptor referencing the new socket. Otherwise, a value of INVALID_SOCKET is returned, and a specific error code can be retrieved by calling.

FD_*() - Four macros are provided to manipulate the sets. FD_ZERO() clears a set. FD_SET() and FD_CLR() respectively add and remove a given file descriptor from a set. FD_ISSET() tests to see if a file descriptor is part of the set; this is useful after select() returns.

[Richard Grant]

I can't speak for the Windows implementation, but on Linux, the crux of it is, you're trying to determine if there's a read or write event available on a particular socket(s). To achieve that end, you fill in an fd_set structure with the sockets in question and pass them as either a read or write parameter. That's what FD_* class of functions/macros are for.

A simple example:

PRIVATE F_RESULT 
network_has_event ( int socket, NET_EVENT src, int timeout, t_boolean *result ) 
{
	fd_set 		 fds;
	struct timeval	 tv;
	int 		 state;
	
	*result = false;
	
	FD_ZERO ( &fds );
	FD_SET ( socket, &fds );
	tv.tv_sec   = ( SECOND <= timeout ) ? timeout / SECOND : 0;
	tv.tv_usec  = ( timeout % SECOND ) * SECOND;
	
	state = select ( socket + 1, ( NET_EVENT_READ == src ) ? &fds : NULL, ( NET_EVENT_WRITE == src ) ? &fds : NULL, NULL, &tv );
	
	if ( -1 == state && EINPROGRESS != errno && EINTR != errno ) {

        	EUTIL_SET_ERROR(E_SYS_ERROR);
		return F_FAILURE;

    	} else if ( 0 < state && FD_ISSET ( socket, &fds ) ) {
        
        	*result = true;
	}
    
	return F_SUCCESS;
}

This only checks a single socket for a read or write event. But it can be scaled to include a set of fds. Once you know that an event is waiting, you can proceed to call the relevant socket function, such as accept, read, write, etc. All this can be done in a non-blocking manner.

 

Allow me an hour or so to research everything you just wrote. Thats all just a little over my head :) i will be right back!

[simplezz]

http://stackoverflow.com/questions/13086085/c-select-only-checks-last-client-socket

while (1) {
    FD_ZERO(&readfds); //I assume this means to set readfds memory as zero
    FD_SET(sockfd, &readfds); // not sure what this does
    highsock = sockfd;
    for (list<int>::iterator it = CliSocks.begin(); it != CliSocks.end(); it++) {
        FD_SET(*it, &readfds); //not to sure what this does
        highsock = *it > highsock ? *it : highsock; //set highsock to the highest value of it and highsock
    }

It would probably help if you understood what the fd_set structure is:

 

$ cat /usr/include/sys/select.h | grep "typedef struct" -A 11

/* fd_set for select and pselect.  */
typedef struct
  {
    /* XPG4.2 requires this member name.  Otherwise avoid the name
       from the global namespace.  */
#ifdef __USE_XOPEN
    __fd_mask fds_bits[__FD_SETSIZE / __NFDBITS];
# define __FDS_BITS(set) ((set)->fds_bits)
#else
    __fd_mask __fds_bits[__FD_SETSIZE / __NFDBITS];
# define __FDS_BITS(set) ((set)->__fds_bits)
#endif
  } fd_set;

It's basically a bit array where each fds that is set (FD_SET) indicates the socket that should be tested to see if an event is available for it.

The first parameter to select needs to be the highest socket of the set + 1 in order to check all of them.

[Richard Grant]

It would probably help if you understood what the fd_set structure is:

 

$ cat /usr/include/sys/select.h | grep "typedef struct" -A 11

/* fd_set for select and pselect.  */
typedef struct
  {
    /* XPG4.2 requires this member name.  Otherwise avoid the name
       from the global namespace.  */
#ifdef __USE_XOPEN
    __fd_mask fds_bits[__FD_SETSIZE / __NFDBITS];
# define __FDS_BITS(set) ((set)->fds_bits)
#else
    __fd_mask __fds_bits[__FD_SETSIZE / __NFDBITS];
# define __FDS_BITS(set) ((set)->__fds_bits)
#endif
  } fd_set;

It's basically a bit array where each fds that is set (FD_SET) indicates the socket that should be tested to see if an event is available for it.

The first parameter to select needs to be the highest socket of the set + 1 in order to check all of them.

 

 

This is what i researched:

//Written by Simplezz
//Noted by Richard_Grant
PRIVATE F_RESULT //Ask Simplezz about this
network_has_event ( int socket,
	NET_EVENT src, //Not sure what this is
	int timeout,
	t_boolean *result // Return that a socket is ready?
){
/*
NET_EVENT, NET_EVENT_READ, NET_EVENT_WRIT, EUTIL_SET_ERROR, E_SYS_ERROR  - no documentation
timeval structure info- http://msdn.microsoft.com/en-us/library/windows/desktop/ms740560%28v=vs.85%29.aspx

{
	FD_ZERO- clears a set 
			http://linux.die.net/man/3/fd_zero
}

{
	fd_set- http://msdn.microsoft.com/en-us/library/windows/desktop/ms737873%28v=vs.85%29.aspx
}

tv_sec - Time interval, in seconds.

{
	tv_usec- Time interval, in microseconds. This value is used in combination 
			with the tv_sec member to represent time interval values that are 
			not a multiple of seconds.
} 

{
	select- The select function returns the total number of socket handles that 
			are ready and contained in the fd_set structures, zero if the time 
			limit expired, or SOCKET_ERROR if an error occurred. If the return 
			value is SOCKET_ERROR, WSAGetLastError can be used to retrieve a 
			specific error code.
			http://msdn.microsoft.com/en-us/library/windows/desktop/ms740141%28v=vs.85%29.aspx
}

{
	EINPROGRESS- Operation in progress (POSIX.1)
	http://man7.org/linux/man-pages/man3/errno.3.html
}

{
	 EINTR- Interrupted function call (POSIX.1); see signal(7).	
	 http://man7.org/linux/man-pages/man3/errno.3.html
	 {
		signal(7)- http://man7.org/linux/man-pages/man7/signal.7.html
	 }
}

{
	FD_ISSET()- http://www-01.ibm.com/support/knowledgecenter/SSB23S_1.1.0.10/com.ibm.ztpf-ztpfdf.doc_put.10/gtpc2/cpp_fd_isset.html?cp=SSB23S_1.1.0.10%2F0-3-8-1-0-3-5
}

*/


	fd_set 		 fds; //init. type for testing a given socket for readability
	struct timeval	 tv; //init. type for The maximum time for select to wait
	int 		 state; /**init. return of select (the total number of socket 
							handles that are ready and contained in the fd_set 
							structures)**/
	
	*result = false; //Set pointer value to false (modify external value probably to return that select has returned)
	
	FD_ZERO ( &fds ); //Clear &fds
	FD_SET ( socket, &fds ); // Add socket to fds (which is an array of sockets)
	tv.tv_sec   = ( SECOND <= timeout ) ? timeout / SECOND : 0; /**Genius way to say set seconds of timeout if its 
																	greater or equal to a second**/
	tv.tv_usec  = ( timeout % SECOND ) * SECOND; //Get remainder from timeout and seconds. (milli seconds)


	/*
		Returns number of ready clients/sockets
	*/
	state = select ( socket + 1, //descriptor referencing the next socket that will be available
		( NET_EVENT_READ == src ) ? &fds : NULL, //An optional pointer to a set of sockets to be checked for readability.
		( NET_EVENT_WRITE == src ) ? &fds : NULL, //An optional pointer to a set of sockets to be checked for writability.
		NULL, //An optional pointer to a set of sockets to be checked for errors.
		&tv /**The maximum time for select to wait, provided in the form of a TIMEVAL structure. Set the timeout 
				parameter to null for blocking operations.**/
	);
	
	if ( -1 == state && EINPROGRESS != errno && EINTR != errno ) { //If a problem happened? not to sure about the details
        	EUTIL_SET_ERROR(E_SYS_ERROR); //Not sure what this does
		return F_FAILURE; // Assume this returns 0?

    	} else if ( 0 < state && //No errors with select
    		FD_ISSET ( socket, &fds ) ) //return file descriptor in the file descriptor set
    	{
        
        	*result = true;//Set pointer value to true (modify external value probably to return that select has returned)
	}
    
	return F_SUCCESS; //Assume this returns 1
}

There were a few things in the code that had no documention so its not very clear what is happening

 

Here is a quick list:

What is F_RESULT, NET_EVENT, NET_EVENT_READ, NET_EVENT_WRIT, EUTIL_SET_ERROR, E_SYS_ERROR?

 

Some of the types a little obvious but what are the actual values? Documention links would also be AMAZING!

[Richard Grant]

 

Those are just conveniences for handling things like return values, testing a specific type of socket event, and general error handling / propagation.

typedef enum { F_SUCCESS, F_FAILURE } F_RESULT;
typedef enum { NET_EVENT_READ, NET_EVENT_WRITE } NET_EVENT;

I wouldn't concern yourself with those. It's the FD_SET, select, and FD_ISSET that are important.

All my code is written against the Linux kernel. So if you want documentation for system functions, look here. Windows functions may differ, but there's generally an equivalent.

 

The whole point of that function I posted was to test that an event was available so other code can call the requisite socket functions without blocking.

F_RESULT
network_read ( t_network *net, t_network_buffer *rb, int timeout )
{
    int          result;
    t_boolean    has_event;

    if ( rb->max_len == rb->used ) {
        EUTIL_SET_ERROR ( NETWORK_E_BUF_FULL );
        return F_FAILURE;
    }

    if ( F_FAILURE == network_has_event ( net->socket, NET_EVENT_READ, timeout, &has_event ) )
        return F_FAILURE;

    if ( !has_event )
        return F_SUCCESS;

    result = recv ( net->socket, rb->buffer + rb->used, rb->max_len - rb->used, 0 );
    
    if ( -1 == result && EINTR != errno ) {    

        EUTIL_SET_ERROR ( E_SYS_ERROR );
        return F_FAILURE;

    } else if ( 0 == result ) {

        EUTIL_SET_ERROR ( NETWORK_E_REMOTE_SHUTDOWN );
        return F_FAILURE;
    }

    rb->used += result;
    
    return F_SUCCESS;
}

 

SoLvEd!