Pane_communicator Class Reference

Encapsulates information about distribution of panes on all processes and handles across pane communication for shared nodes, ghost nodes, and ghost cells. More...

#include <Pane_communicator.h>

Collaboration diagram for Pane_communicator:

Classes
struct	Pane_comm_buffers
	Buffers for outgoing and incoming messages of a specific pane to be communicated with another pane (either local or global). More...

Public Types
enum	Buff_type {   RNS, RCS, SHARED_NODE, GNR,   GCR }

Public Member Functions
	Pane_communicator (COM::Window *w, MPI_Comm c=MPI_COMM_WORLD)
	Constructor from a communicator. More...
void	init (void **ptrs, COM_Type type, int ncomp, const int *sizes=NULL, const int *strds=NULL)
	Initialize the communication buffers. More...
void	init (COM::Attribute *att, const COM::Attribute *my_pconn=NULL)
	Initialize the communication buffers. More...
MPI_Comm	mpi_comm () const
	Obtain the MPI communicator for the object. More...
std::vector< COM::Pane * > &	panes ()
	Obtains all the local panes. More...
int	owner_rank (const int pane_id) const
	Obtain the process rank of a given pane. More...
int	total_npanes () const
	Obtain the total number of panes on all processes. More...
int	lpaneid (const int pane_id) const
	For a given pane, obtain an internal pane ID which is unique and contiguous across processes. More...
void	begin_update_shared_nodes (std::vector< std::vector< bool > > *involved=NULL)
	Initiates updating shared nodes by calling MPI_Isend and MPI_Irecv. More...
void	begin_update_ghost_nodes ()
	Initiates updating ghost nodes by calling MPI_Isend and MPI_Irecv. More...
void	begin_update_ghost_cells ()
	Initiates updating ghost nodes by calling MPI_Isend and MPI_Irecv. More...
void	end_update_shared_nodes ()
	Finalizes shared node updating by calling MPI_Waitall on all send requests. More...
void	end_update_ghost_nodes ()
	Finalizes ghost node updating by calling MPI_Waitall on all send requests. More...
void	end_update_ghost_cells ()
	Finalizes ghost cell updating by calling MPI_Waitall on all send requests. More...
void	reduce_on_shared_nodes (MPI_Op)
	Perform a reduction operation using locally cached values of the shared nodes, assuming begin_update_shared_nodes() has been called. More...
void	reduce_maxabs_on_shared_nodes ()
	Reduce to the value with the maximum absolute value using locally cached values of shared nodes. More...
void	reduce_minabs_on_shared_nodes ()
	Reduce to the value with the maximum absolute value using locally cached values of shared nodes. More...
void	reduce_diff_on_shared_nodes ()
	Compute difference of non-zero values of each shared node, assuming there are at most two non-zero values. More...
void	update_ghost_values ()
void	reduce_average_on_shared_nodes ()
	Reduce to the average of values using locally cached values of the shared nodes, assuming begin_update_shared_nodes() has been called. More...

Protected Member Functions
void	init_pane_comm_buffers (std::vector< Pane_comm_buffers > &pcb, const int *ptr, int &index, const int n_items, const int lpid)
	Initialize a Pane_comm_buffers for ghost information. More...
void	end_update ()
	Finalizes updating by calling MPI_Waitall on all send requests. More...
void	begin_update (const Buff_type btype, std::vector< std::vector< bool > > *involved=NULL)
	Initiates updating by calling MPI_Isend and MPI_Irecv. More...

Protected Attributes
int	_my_pconn_id
	The id of the pconn being used. More...
COM::Window *	_appl_window
	Reference to the application window. More...
const MPI_Comm	_comm
	MPI Communicator. More...
std::vector< COM::Pane * >	_panes
	Vector of all local panes. More...
int	_total_npanes
	The total number of panes on all processes. More...
std::map< int, int >	_lpaneid_map
	Mapping from user-defined pane ids to internal IDs, which are unique and contiguous across all processes, which are useful for defining unique tags for MPI messages. More...
int	_type
	The base data type, number of components, and the number of bytes of all components for the data to be communicated. More...
int	_ncomp
int	_ncomp_bytes
std::vector< void * >	_ptrs
	An array of pointers to the data for all local panes. More...
std::vector< int >	_sizes
	The sizes of the arrays for all local panes. More...
std::vector< int >	_strds
	The strides of the arrays for all local panes. More...
std::vector< std::vector < Pane_comm_buffers > >	_shr_buffs
	Shared node pane communication buffers. More...
std::vector< std::vector < Pane_comm_buffers > >	_rns_buffs
	Buffer for real nodes to send. More...
std::vector< std::vector < Pane_comm_buffers > >	_gnr_buffs
	Buffer for ghost nodes to receive. More...
std::vector< std::vector < Pane_comm_buffers > >	_rcs_buffs
	Buffer for real cells to send. More...
std::vector< std::vector < Pane_comm_buffers > >	_gcr_buffs
	Buffer for ghost cells to receive. More...
std::vector< MPI_Request >	_reqs_send
	Arrays of pending nonblocking MPI requests. Same format as _shr_buffs. More...
std::vector< MPI_Request >	_reqs_recv
std::vector< std::pair< int, int > >	_reqs_indices
	The indices in buffs for each pending nonblocking receive request. More...

Private Member Functions
	Pane_communicator (const Pane_communicator &)
Pane_communicator &	operator= (const Pane_communicator &)

Detailed Description

Encapsulates information about distribution of panes on all processes and handles across pane communication for shared nodes, ghost nodes, and ghost cells.

However,there can be no overlap among the 3 types of communication since they share the same set of tags and request buffers.

Definition at line 46 of file Pane_communicator.h.

Member Enumeration Documentation

enum Buff_type

Enumerator
RNS
RCS
SHARED_NODE
GNR
GCR

Definition at line 69 of file Pane_communicator.h.

                 {
    RNS,         // Real nodes to send
    RCS,         // Real cells to send.
    SHARED_NODE, // Shared nodes to send and receive.
    GNR,         // Ghost nodes to receive.
    GCR          // Ghost cells to receive.
  };

Constructor & Destructor Documentation

MAP_BEGIN_NAMESPACE Pane_communicator ( COM::Window *  w, MPI_Comm  c = MPI_COMM_WORLD  )

explicit

Constructor from a communicator.

Also initialize the internal data structures of the communicator, in particular the internal pane IDs.

Definition at line 38 of file Pane_communicator.C.

References _appl_window, _lpaneid_map, _my_pconn_id, _panes, _total_npanes, COM_PCONN, and i.

  : _appl_window( w), _comm(COMMPI_Initialized()?c:MPI_COMM_NULL), 
    _total_npanes(-1) 
{ 
  _my_pconn_id = COM::COM_PCONN;
  _appl_window->panes( _panes);
  const COM::Window::Proc_map &proc_map = _appl_window->proc_map();
  _total_npanes = proc_map.size();

  // Determine a mapping from user-defined pane ids to a set of 
  // internal pane IDs, which are unique and contiguous across all processes
  // to be used for defining unique tags for MPI messages.
  std::map<int,int>::const_iterator it=proc_map.begin();
  for ( int i=0; i<_total_npanes; ++i, ++it) 
    _lpaneid_map[ it->first] = i;
}

Pane_communicator ( const Pane_communicator &  )

private

Member Function Documentation

void begin_update ( const Buff_type  btype, std::vector< std::vector< bool > > *  involved = NULL  )

protected

Initiates updating by calling MPI_Isend and MPI_Irecv.

See begin_update_shared_nodes for description of involved.

Definition at line 205 of file Pane_communicator.C.

References _comm, _gcr_buffs, _gnr_buffs, _my_pconn_id, _ncomp_bytes, _panes, _ptrs, _rcs_buffs, _reqs_indices, _reqs_recv, _reqs_send, _rns_buffs, _shr_buffs, _sizes, _strds, _total_npanes, _type, COM_assertion, COM_assertion_msg, COM_get_sizeof(), COMMPI_Comm_rank(), COMMPI_Initialized(), COMMPI_Irecv(), COMMPI_Isend(), GCR, GNR, i, Pane_communicator::Pane_comm_buffers::inbuf, Pane_communicator::Pane_comm_buffers::index, j, k, MPI_BYTE, MPI_COMM_SELF, n, nj, Pane_communicator::Pane_comm_buffers::outbuf, Pane_communicator::Pane_comm_buffers::rank, rank, RCS, RNS, SHARED_NODE, and Pane_communicator::Pane_comm_buffers::tag.

Referenced by begin_update_ghost_cells(), begin_update_ghost_nodes(), and begin_update_shared_nodes().

                                                                             {
  COM_assertion_msg(_reqs_recv.empty(),
                    "Cannot begin a new update until all prior updates are finished.");
  // Define and initialize variables
  int rank = COMMPI_Initialized() ? COMMPI_Comm_rank( _comm) : 0;

  MPI_Request req;
  _reqs_send.clear(); _reqs_recv.clear(); _reqs_indices.clear();
  int local_npanes = _panes.size();
  int tag_max = _total_npanes*_total_npanes;
  if ( involved) { involved->clear(); involved->resize( local_npanes); }

  // Now copy data to outbuf. First loop through local panes
  for ( int i=0; i<local_npanes; ++i) {
    if ( involved) (*involved)[i].resize( _sizes[i], false);
    
    // Obtain the pane connectivity for the current pane
    const COM::Attribute *pconn = _panes[i]->attribute(_my_pconn_id);
    const int *vs = (const int*)pconn->pointer();

    int strd_bytes = COM_get_sizeof( _type, _strds[i]);
    // Shift the pointer by -1 because node IDs in pconn start from 1
    char *ptr = ((char*)_ptrs[i])-strd_bytes; 

    // Obtain the current buffer
    std::vector< Pane_comm_buffers>   *buffs =NULL;

    if (btype == RNS)                  buffs = &_rns_buffs[i];

    else if (btype == RCS)             buffs = &_rcs_buffs[i]; 

    else if (btype == SHARED_NODE)     buffs = &_shr_buffs[i];

    else if (btype == GNR) {
      buffs = &_gnr_buffs[i];
    }
    else if (btype == GCR) 
    {
      buffs = &_gcr_buffs[i]; 
    }
    else
      COM_assertion_msg(false,"Invalid buffer type");

    // Loop through the communicating panes of the current pane
    for ( int j=0, nj=buffs->size(); j<nj; ++j) {
      Pane_comm_buffers* pcb  = &(*buffs)[j];// The current buffer.
     
      // Fill in outgoing buffers if sending.
      int bufsize=_ncomp_bytes * vs[ pcb->index+1];

      if ( btype != SHARED_NODE || _panes[i]->id() != vs[pcb->index]) { 
        // If not sending shared nodes to itself
        if(btype <= SHARED_NODE){
          pcb->outbuf.resize( bufsize);

          if ( involved) {
            for ( int k=0, from=pcb->index+2,n=vs[pcb->index+1]; 
                  k<n; ++k, ++from)
              (*involved)[i][vs[ from]-1] = true;
          }

          for ( int k=0, from=pcb->index+2,n=vs[pcb->index+1]; 
                k<n; ++k, ++from) {
            std::memcpy( &pcb->outbuf[ _ncomp_bytes*k], 
                         &ptr[ strd_bytes*vs[ from]], _ncomp_bytes);
          }

          // Initiates send operations either locally or remotely
          // if on same communicating process
          if ( rank == pcb->rank) {
            int tag = pcb->tag;

            // If send locally, shift the tag in one of the two directions
            if ( _panes[i]->id() > vs[pcb->index]) tag += tag_max;
            tag = tag%32768;
            // COMMPI uses DUMMY_MPI if MPI not initialized
            int ierr=COMMPI_Isend( &pcb->outbuf[0], pcb->outbuf.size(), 
                                   MPI_BYTE, 0, tag, MPI_COMM_SELF, &req);
            COM_assertion( ierr==0);
            _reqs_send.push_back( req);
          }
          else {
            int ierr=MPI_Isend( &pcb->outbuf[0], pcb->outbuf.size(), MPI_BYTE, 
                                pcb->rank, pcb->tag, _comm, &req);
            COM_assertion( ierr==0);
    
            _reqs_send.push_back( req);
          }
        }

        // Initiates receive operations either locally or remotely
        if(btype >=SHARED_NODE){
          pcb->inbuf.resize( bufsize);

          if ( rank == pcb->rank) {
            int tag = pcb->tag;
            
            // If recv locally, shift the tag in one of the two directions
            // have to have unique tags for send/receive when we are sending
            // between panes on the same process.
            if ( _panes[i]->id() < vs[pcb->index]) tag += tag_max;
            tag = tag%32768;
            int ierr=COMMPI_Irecv( &pcb->inbuf[0], pcb->inbuf.size(), 
                                   MPI_BYTE, 0, tag, MPI_COMM_SELF, &req);
            COM_assertion( ierr==0);
            
            // Push the receive request into _reqs_recv and _reqs_indices
            _reqs_recv.push_back( req); 
            _reqs_indices.push_back( std::make_pair(i,(j<<4)+btype));
          }
          else {
            int ierr=MPI_Irecv( &pcb->inbuf[0], pcb->inbuf.size(), MPI_BYTE, 
                                pcb->rank,pcb->tag, _comm, &req);
            COM_assertion( ierr==0);
            
            // Push the receive request into _reqs_recv and _reqs_indices
            _reqs_recv.push_back( req);
            _reqs_indices.push_back( std::make_pair(i,(j<<4)+btype));
          }
        }
      }
      
      else { // btype == SHARED_NODE &&_panes[i]->id()==vs[pcb->index]
        pcb->inbuf.resize( bufsize);

        // A pane is sending to itself.
        // In a list of nodes which a pane shares with itself, the nodes are
        // listed in pairs.  IE a node list { 1,16,2,17,...} indicates that nodes
        // 1 and 3 are the same and nodes 5 and 13 are the same.
        // So, we copy the data value of a node into the outbuf of its shared node.
        // Then we can transfer the data directly.
        for (int k=0,from=pcb->index+2,n=vs[pcb->index+1]; k<n; k+=2, from+=2){
          std::memcpy( &pcb->inbuf[ _ncomp_bytes*(k+1)], 
                       &ptr[strd_bytes*vs[ from]], _ncomp_bytes);
          std::memcpy( &pcb->inbuf[ _ncomp_bytes*k], 
                       &ptr[strd_bytes*vs[ from+1]], _ncomp_bytes);
        }
      }
    }
  }
  if(COMMPI_Initialized())
    MPI_Barrier(_comm);
}

Here is the call graph for this function:

Here is the caller graph for this function:

void begin_update_ghost_cells ( )

inline

Initiates updating ghost nodes by calling MPI_Isend and MPI_Irecv.

Definition at line 135 of file Pane_communicator.h.

References begin_update(), GCR, and RCS.

                                 {
    begin_update(RCS);
    begin_update(GCR);
  }

Here is the call graph for this function:

void begin_update_ghost_nodes ( )

inline

Initiates updating ghost nodes by calling MPI_Isend and MPI_Irecv.

Definition at line 129 of file Pane_communicator.h.

References begin_update(), GNR, and RNS.

                                 {
    begin_update(RNS);
    begin_update(GNR);
  }

Here is the call graph for this function:

void begin_update_shared_nodes ( std::vector< std::vector< bool > > *  involved = NULL )

inline

Initiates updating shared nodes by calling MPI_Isend and MPI_Irecv.

If involved is not NULL, then at exit it contains a vector of bitmaps, each of which corresponds to a local pane, and the size of each bitmap is the number of real nodes for its corresponding local pane. Upon exit, the bits corresponding to shared nodes are set to true. The information in involved is required to overlap communication and computation during reduction operations.

Definition at line 124 of file Pane_communicator.h.

References begin_update(), and SHARED_NODE.

                                                                            {
    begin_update(SHARED_NODE,involved);
  }

Here is the call graph for this function:

void end_update ( )

protected

Finalizes updating by calling MPI_Waitall on all send requests.

Definition at line 351 of file Pane_communicator.C.

References _comm, _reqs_send, and COM_assertion.

Referenced by end_update_ghost_cells(), end_update_ghost_nodes(), and end_update_shared_nodes().

                                   {
  if ( _comm!=MPI_COMM_NULL) {

    std::vector< MPI_Status> status( _reqs_send.size());
    int ierr=0;
    if (_reqs_send.size())
      ierr=MPI_Waitall( _reqs_send.size(), &_reqs_send[0], &status[0]);
    COM_assertion( ierr==0);
  }
  _reqs_send.resize(0);
}

Here is the caller graph for this function:

void end_update_ghost_cells ( )

inline

Finalizes ghost cell updating by calling MPI_Waitall on all send requests.

Definition at line 152 of file Pane_communicator.h.

References end_update(), and update_ghost_values().

                               {
    update_ghost_values();
    end_update();
  }

Here is the call graph for this function:

void end_update_ghost_nodes ( )

inline

Finalizes ghost node updating by calling MPI_Waitall on all send requests.

Definition at line 146 of file Pane_communicator.h.

References end_update(), and update_ghost_values().

                               {
    update_ghost_values();
    end_update();
  }

Here is the call graph for this function:

void end_update_shared_nodes ( )

inline

Finalizes shared node updating by calling MPI_Waitall on all send requests.

Definition at line 141 of file Pane_communicator.h.

References end_update().

                                 {
    end_update();
  }

Here is the call graph for this function:

void init	(	void **	ptrs,
		COM_Type	type,
		int	ncomp,
		const int *	sizes = NULL,
		const int *	strds = NULL
	)

Initialize the communication buffers.

ptrs is an array of pointers to the data for all local panes type is the base data type (such as MPI_INT) ncomp is the number of component per node sizes is the number of items per pane. If null, then use number of nodes. strides are the strides for each array. If null, then use ncomp for all.

Definition at line 91 of file Pane_communicator.C.

References _gcr_buffs, _gnr_buffs, _my_pconn_id, _ncomp, _ncomp_bytes, _panes, _ptrs, _rcs_buffs, _rns_buffs, _shr_buffs, _sizes, _strds, _type, COM_get_sizeof(), i, init_pane_comm_buffers(), and lpaneid().

Referenced by init(), Rocmap::reduce_average_on_shared_nodes(), Rocmap::reduce_maxabs_on_shared_nodes(), Rocmap::reduce_minabs_on_shared_nodes(), and Rocmap::update_ghosts().

                                                                             {
  
  //=== Initialize local objects from the arguments
  _type = type;
  _ncomp = ncomp;
  _ncomp_bytes = COM_get_sizeof( _type, _ncomp);

  int local_npanes = _panes.size();
  _shr_buffs.resize( local_npanes);
  _rns_buffs.resize( local_npanes);
  _gnr_buffs.resize( local_npanes);
  _rcs_buffs.resize( local_npanes);
  _gcr_buffs.resize( local_npanes);
  _ptrs.clear(); _ptrs.insert( _ptrs.end(), ptrs, ptrs+local_npanes);

  _sizes.clear();
  if ( sizes) 
    _sizes.insert( _sizes.end(), sizes, sizes+local_npanes);
  else {
    // Default values of sizes are the numbers of nodes
    _sizes.resize( local_npanes);
    for ( int i=0; i<local_npanes; ++i)
      _sizes[i] = _panes[i]->size_of_real_nodes();
  }

  _strds.clear(); 
  if ( strds) _strds.insert( _strds.end(), strds, strds+local_npanes);
  else _strds.resize( local_npanes, _ncomp);

  // Allocate buffer space for outbuf and ghosts
  for ( int i=0; i<local_npanes; ++i) {
    int pid = _panes[i]->id(), lpid=lpaneid(pid);

    // Obtain the pane connectivity of the local pane.
    const COM::Attribute *pconn = _panes[i]->attribute(_my_pconn_id);
    const int *vs = (const int*)pconn->pointer();
    int vs_size=pconn->size_of_real_items();

    // Note that the pane connectivity may be inherited from a parent
    // window, and the current window may contain only a subset of the
    // panes of the parent window. Here we should consider only the panes
    // in the current window

    // Initialize buffers for shared nodes.
    int index = 0;
    init_pane_comm_buffers( _shr_buffs[i], vs, index, vs_size, lpid);

    const int vs_gsize=pconn->size_of_items();
    // Initialize the ghost information buffers.
    if(vs_size<vs_gsize)
      init_pane_comm_buffers(_rns_buffs[i], vs, vs_size, vs_gsize, lpid);
    else
      _rns_buffs[i].clear();

    if(vs_size<vs_gsize)
      init_pane_comm_buffers(_gnr_buffs[i], vs, vs_size, vs_gsize, lpid);
    else
      _gnr_buffs[i].clear();

    if(vs_size<vs_gsize)
      init_pane_comm_buffers(_rcs_buffs[i], vs, vs_size, vs_gsize, lpid);
    else
      _rcs_buffs[i].clear();

    if(vs_size<vs_gsize)
      init_pane_comm_buffers(_gcr_buffs[i], vs, vs_size, vs_gsize, lpid);
    else
      _gcr_buffs[i].clear();
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

void init	(	COM::Attribute *	att,
		const COM::Attribute *	my_pconn = NULL
	)

Initialize the communication buffers.

att is a pointer to the attribute my_pconn stores pane-connectivity

Definition at line 56 of file Pane_communicator.C.

References _appl_window, _my_pconn_id, _panes, COM_assertion, COM_assertion_msg, COM_PCONN, i, and init().

                                                           {
  // Note that the attribute must be on the attribute window.
  COM_assertion( att->window() == _appl_window &&
                 ( my_pconn ==NULL || my_pconn->window() == _appl_window));

  if(my_pconn){
    _my_pconn_id = my_pconn->id();
  }
  else{
    _my_pconn_id = COM::COM_PCONN;
    COM_assertion_msg( att && (att->is_nodal() || att->is_elemental()),
                       "Pane_communicator must be initialized with a nodal or elemental attribute");
  }

  int  att_id = att->id();
  int  local_npanes = _panes.size();

  void** pointers = new void*[local_npanes];
  int *sizes = new int[local_npanes];
  int *strides = new int[local_npanes];

  for ( int i=0; i < local_npanes; ++i) {
    COM::Attribute *attribute = _panes[i]->attribute(att_id);
    pointers[i] = attribute->pointer();
    sizes[i] = attribute->size_of_real_items();
    strides[i] = attribute->stride();
  }
  init(pointers, att->data_type(), att->size_of_components(), sizes, strides);
  delete[] pointers;  pointers = NULL;
  delete[] sizes;     sizes=NULL;
  delete[] strides;   strides = NULL;
}

Here is the call graph for this function:

void init_pane_comm_buffers ( std::vector< Pane_comm_buffers > &  pcb, const int *  ptr, int &  index, const int  n_items, const int  lpid  )

protected

Initialize a Pane_comm_buffers for ghost information.

Definition at line 168 of file Pane_communicator.C.

References _total_npanes, COM_assertion_msg, Pane_communicator::Pane_comm_buffers::index, j, lpaneid(), owner_rank(), Pane_communicator::Pane_comm_buffers::rank, and Pane_communicator::Pane_comm_buffers::tag.

Referenced by init().

                                       {
  if ( n_items == 0) return;

  // Determine the number of communicating panes
  int count=0, np= ptr[index];
  int block_size = 1; 

  for (int j=0; j<np; block_size+=ptr[index+block_size+1]+2, ++j) {
    if (owner_rank( ptr[index+block_size]) >=0) ++count;
  }
  COM_assertion_msg(block_size+index <= n_items, "Out of pconn bounds.");
  ++index;

  pcbv.resize( count);

  int qid=0, lqid=0;
  // Loop through communicating panes for shared nodes.
  for (int p=0, j=0; p<np; ++p, index+=ptr[index+1]+2) {
    if (owner_rank( qid=ptr[index]) <0) continue;
 
    // Define a unique tag to be used by MPI_Isend and MPI_Irecv.
    lqid=lpaneid(qid);
    Pane_comm_buffers &pcb = pcbv[j++];
    pcb.rank = owner_rank( qid);
    // Numbers have to be positive, try to avoid conflicting tags with other apps.
    // assuming other apps are low, 100 is a decent guess.
    pcb.tag = 100 + ((lpid>lqid) ? 
                     lpid*_total_npanes+lqid : lqid*_total_npanes+lpid);
    pcb.tag = pcb.tag%32768;
    pcb.index = index;
  }
  COM_assertion_msg( index <= n_items, "Out of bound of pconn");
}

Here is the call graph for this function:

Here is the caller graph for this function:

int lpaneid ( const int  pane_id ) const

inline

For a given pane, obtain an internal pane ID which is unique and contiguous across processes.

Definition at line 112 of file Pane_communicator.h.

References _lpaneid_map, _total_npanes, and COM_assertion.

Referenced by init(), and init_pane_comm_buffers().

                                        {
    COM_assertion( _total_npanes>0); 
    return _lpaneid_map.find(pane_id)->second;
  }

Here is the caller graph for this function:

MPI_Comm mpi_comm ( ) const

inline

Obtain the MPI communicator for the object.

Definition at line 97 of file Pane_communicator.h.

References _comm.

{ return _comm; }

Pane_communicator& operator= ( const Pane_communicator &  )

private

int owner_rank ( const int  pane_id ) const

inline

Obtain the process rank of a given pane.

If the pane does not exist, it will return -1.

Definition at line 104 of file Pane_communicator.h.

References _appl_window.

Referenced by init_pane_comm_buffers(), and reduce_average_on_shared_nodes().

  { return _appl_window->owner_rank( pane_id); }

Here is the caller graph for this function:

std::vector<COM::Pane*>& panes ( )

inline

Obtains all the local panes.

Definition at line 100 of file Pane_communicator.h.

References _panes.

{ return _panes; }

void reduce_average_on_shared_nodes

(

)

Reduce to the average of values using locally cached values of the shared nodes, assuming begin_update_shared_nodes() has been called.

Definition at line 758 of file Pane_communicator.C.

References _my_pconn_id, _ncomp, _panes, _ptrs, _strds, _type, COM_assertion_msg, COM_CHAR, COM_CHARACTER, COM_DOUBLE, COM_DOUBLE_PRECISION, COM_FLOAT, COM_get_sizeof(), COM_INT, COM_INTEGER, COM_REAL, i, j, k, MPI_SUM, owner_rank(), and reduce_on_shared_nodes().

                                                      {  
  // Sum the values from all shared nodes
  reduce_on_shared_nodes(MPI_SUM);

  // A map from shared nodes' local ids to multiplicity
  std::map<int,int> nodes_to_mult;
  std::map<int,int>::iterator nodes_to_mult_pos;
  int num_panes = (int)_panes.size();
  //loop through all local panes, constructing multiplicity of nodes
  //then iterate through nodes, dividing by multiplicity
  for(int i = 0; i < num_panes; ++i){
    const COM::Attribute *pconn = _panes[i]->attribute(_my_pconn_id);
    const int *vs = (const int*)pconn->pointer();
    
    // make sure we only consider shared nodes, not ghost nodes
    int vs_size=pconn->size_of_real_items();
    int strd_bytes = COM_get_sizeof( _type, _strds[i]);
    // Shift the pointer by -1 because node IDs in pconn start from 1
    char *ptr = ((char*)_ptrs[i])-strd_bytes;
    
    // Determine the multiplicity of shared nodes.  Multiplicity is the 
    // number of panes which own the node, so at least 2 for shared nodes.
    nodes_to_mult.clear();
    for(int j=1; j<vs_size; j+= vs[j+1]+2){
      // If this pane hasn't been inherited, skip it
      if (owner_rank( vs[j])>= 0){
        // Loop through nodes, ++mult if already seen.  Else mult=2
        for(int k=0; k< vs[j+1]; ++k){
          nodes_to_mult_pos = nodes_to_mult.find(vs[j+k+2]);
          if (nodes_to_mult_pos != nodes_to_mult.end())
            nodes_to_mult_pos->second++;
          else
            nodes_to_mult.insert(std::make_pair(vs[j+k+2],2));
        }
      }
    }

    // Loop through the list of shared nodes, dividing data by their multiplicity
    // Remember the assumption that stride == ncomp
    int shared_id,mult;
    switch ( _type) {
    case COM_CHAR:
    case COM_CHARACTER:
      for(nodes_to_mult_pos=nodes_to_mult.begin();
          nodes_to_mult_pos!=nodes_to_mult.end();
          ++nodes_to_mult_pos){
        shared_id = nodes_to_mult_pos->first;

        mult = nodes_to_mult_pos->second;
        for(int k = 0; k < _ncomp; ++k){
          ((char*)&ptr[ strd_bytes*shared_id])[k] /= mult;
        }
      }
      break;
    case COM_INT:
    case COM_INTEGER:
      for(nodes_to_mult_pos=nodes_to_mult.begin();
          nodes_to_mult_pos!=nodes_to_mult.end();
          ++nodes_to_mult_pos){
        shared_id = nodes_to_mult_pos->first;

        mult = nodes_to_mult_pos->second;
        for(int k = 0; k < _ncomp; ++k){
          ((int*)&ptr[ strd_bytes*shared_id])[k] /= mult;
        }
      }
      break;
    case COM_FLOAT:
    case COM_REAL:
      for(nodes_to_mult_pos=nodes_to_mult.begin();
          nodes_to_mult_pos!=nodes_to_mult.end();
          ++nodes_to_mult_pos){
        shared_id = nodes_to_mult_pos->first;
        mult = nodes_to_mult_pos->second;
        for(int k = 0; k < _ncomp; ++k){
          ((float*)&ptr[ strd_bytes*shared_id])[k] /= (float)mult;
        }
      }
      break;
    case COM_DOUBLE:
    case COM_DOUBLE_PRECISION:
      for(nodes_to_mult_pos=nodes_to_mult.begin();
          nodes_to_mult_pos!=nodes_to_mult.end();
          ++nodes_to_mult_pos){
        shared_id = nodes_to_mult_pos->first;
        mult = nodes_to_mult_pos->second;
        for(int k = 0; k < _ncomp; ++k){
          ((double*)&ptr[ strd_bytes*shared_id])[k] /= (double)mult;
        }
      }
      break;
    default:
      COM_assertion_msg(false, "Unknown data type"); // Not supported
    }
  }
} 

Here is the call graph for this function:

void reduce_diff_on_shared_nodes

(

)

Compute difference of non-zero values of each shared node, assuming there are at most two non-zero values.

Definition at line 623 of file Pane_communicator.C.

References _comm, _my_pconn_id, _ncomp, _ncomp_bytes, _panes, _ptrs, _reqs_indices, _reqs_recv, _shr_buffs, _strds, _type, COM_assertion, COM_assertion_msg, COM_CHAR, COM_CHARACTER, COM_DOUBLE, COM_DOUBLE_PRECISION, COM_FLOAT, COM_get_sizeof(), COM_INT, COM_INTEGER, COM_REAL, i, Pane_communicator::Pane_comm_buffers::inbuf, Pane_communicator::Pane_comm_buffers::index, j, k, nk, and reduce_diff().

                                                    {
  while ( !_reqs_recv.empty()) {
    int index;

    // Wait for any receive request to finish and then process the request
    if ( _comm!=MPI_COMM_NULL) {
      MPI_Status status;
      int ierr = MPI_Waitany( _reqs_recv.size(), &_reqs_recv[0], 
                              &index, &status);
      COM_assertion( ierr == 0);
    }
    else
      index = _reqs_recv.size()-1;

    // Obtain the indices in _shr_buffs for the receive request
    int i=_reqs_indices[index].first, j=(_reqs_indices[index].second>>4);
    
    int strd_bytes = COM_get_sizeof( _type, _strds[i]);
    // Shift the pointer by -1 because node IDs in pconn start from 1
    char *ptr = ((char*)_ptrs[i])-strd_bytes;

    const COM::Attribute *pconn = _panes[i]->attribute(_my_pconn_id);
    const int *vs = (const int*)pconn->pointer();

    Pane_comm_buffers &pcb = _shr_buffs[i][j];
    COM_assertion( int(pcb.inbuf.size()) == _ncomp_bytes*vs[ pcb.index+1]);

    switch ( _type) {
    case COM_CHAR:
    case COM_CHARACTER:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        reduce_diff( (char*)&pcb.inbuf[ _ncomp_bytes*k],
                     (char*)&ptr[ strd_bytes*vs[ to]], _ncomp);
      break;
    case COM_INT:
    case COM_INTEGER:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        reduce_diff( (int*)&pcb.inbuf[ _ncomp_bytes*k],
                     (int*)&ptr[ strd_bytes*vs[ to]], _ncomp);
      break;
    case COM_FLOAT:
    case COM_REAL:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        reduce_diff( (float*)&pcb.inbuf[ _ncomp_bytes*k], 
                     (float*)&ptr[ strd_bytes*vs[ to]], _ncomp);
      break;
    case COM_DOUBLE:
    case COM_DOUBLE_PRECISION:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        reduce_diff( (double*)&pcb.inbuf[ _ncomp_bytes*k], 
                     (double*)&ptr[ strd_bytes*vs[ to]], _ncomp);
      break;
    default:
      COM_assertion_msg(false, "Unknown data type"); // Not supported
    }

    // Remove the received message from the list
    _reqs_recv.erase(_reqs_recv.begin()+index);
    _reqs_indices.erase(_reqs_indices.begin()+index);
  }
}

Here is the call graph for this function:

void reduce_maxabs_on_shared_nodes

(

)

Reduce to the value with the maximum absolute value using locally cached values of shared nodes.

Definition at line 497 of file Pane_communicator.C.

References _comm, _my_pconn_id, _ncomp, _ncomp_bytes, _panes, _ptrs, _reqs_indices, _reqs_recv, _shr_buffs, _strds, _type, COM_assertion, COM_assertion_msg, COM_CHAR, COM_CHARACTER, COM_DOUBLE, COM_DOUBLE_PRECISION, COM_FLOAT, COM_get_sizeof(), COM_INT, COM_INTEGER, COM_REAL, i, Pane_communicator::Pane_comm_buffers::inbuf, Pane_communicator::Pane_comm_buffers::index, j, k, nk, and reduce_maxabs().

                                                      {
  while ( !_reqs_recv.empty()) {
    int index;

    // Wait for any receive request to finish and then process the request
    if ( _comm!=MPI_COMM_NULL) {
      MPI_Status status;
      int ierr = MPI_Waitany( _reqs_recv.size(), &_reqs_recv[0], 
                              &index, &status);
      COM_assertion( ierr == 0);
    }
    else
      index = _reqs_recv.size()-1;

    // Obtain the indices in _shr_buffs for the receive request
    int i=_reqs_indices[index].first, j=(_reqs_indices[index].second>>4);

    int strd_bytes = COM_get_sizeof( _type, _strds[i]);
    // Shift the pointer by -1 because node IDs in pconn start from 1
    char *ptr = ((char*)_ptrs[i])-strd_bytes;

    const COM::Attribute *pconn = _panes[i]->attribute(_my_pconn_id);
    const int *vs = (const int*)pconn->pointer();

    Pane_comm_buffers &pcb = _shr_buffs[i][j];
    COM_assertion( int(pcb.inbuf.size()) == _ncomp_bytes*vs[ pcb.index+1]);

    switch ( _type) {
    case COM_CHAR:
    case COM_CHARACTER:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        reduce_maxabs( (char*)&pcb.inbuf[ _ncomp_bytes*k],
                       (char*)&ptr[ strd_bytes*vs[ to]], _ncomp);
      break;
    case COM_INT:
    case COM_INTEGER:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        reduce_maxabs( (int*)&pcb.inbuf[ _ncomp_bytes*k],
                       (int*)&ptr[ strd_bytes*vs[ to]], _ncomp);
      break;
    case COM_FLOAT:
    case COM_REAL:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        reduce_maxabs( (float*)&pcb.inbuf[ _ncomp_bytes*k], 
                       (float*)&ptr[ strd_bytes*vs[ to]], _ncomp);
      break;
    case COM_DOUBLE:
    case COM_DOUBLE_PRECISION:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        reduce_maxabs( (double*)&pcb.inbuf[ _ncomp_bytes*k], 
                       (double*)&ptr[ strd_bytes*vs[ to]], _ncomp);
      break;
    default:
      COM_assertion_msg(false, "Unknown data type"); // Not supported
    }

    // Remove the received message from the list
    _reqs_recv.erase(_reqs_recv.begin()+index);
    _reqs_indices.erase(_reqs_indices.begin()+index);
  }
}

Here is the call graph for this function:

void reduce_minabs_on_shared_nodes

(

)

Reduce to the value with the maximum absolute value using locally cached values of shared nodes.

Definition at line 560 of file Pane_communicator.C.

References _comm, _my_pconn_id, _ncomp, _ncomp_bytes, _panes, _ptrs, _reqs_indices, _reqs_recv, _shr_buffs, _strds, _type, COM_assertion, COM_assertion_msg, COM_CHAR, COM_CHARACTER, COM_DOUBLE, COM_DOUBLE_PRECISION, COM_FLOAT, COM_get_sizeof(), COM_INT, COM_INTEGER, COM_REAL, i, Pane_communicator::Pane_comm_buffers::inbuf, Pane_communicator::Pane_comm_buffers::index, j, k, nk, and reduce_minabs().

                                                      {
  while ( !_reqs_recv.empty()) {
    int index;

    // Wait for any receive request to finish and then process the request
    if ( _comm!=MPI_COMM_NULL) {
      MPI_Status status;
      int ierr = MPI_Waitany( _reqs_recv.size(), &_reqs_recv[0], 
                              &index, &status);
      COM_assertion( ierr == 0);
    }
    else
      index = _reqs_recv.size()-1;

    // Obtain the indices in _shr_buffs for the receive request
    int i=_reqs_indices[index].first, j=(_reqs_indices[index].second>>4);

    int strd_bytes = COM_get_sizeof( _type, _strds[i]);
    // Shift the pointer by -1 because node IDs in pconn start from 1
    char *ptr = ((char*)_ptrs[i])-strd_bytes;

    const COM::Attribute *pconn = _panes[i]->attribute(_my_pconn_id);
    const int *vs = (const int*)pconn->pointer();

    Pane_comm_buffers &pcb = _shr_buffs[i][j];
    COM_assertion( int(pcb.inbuf.size()) == _ncomp_bytes*vs[ pcb.index+1]);

    switch ( _type) {
    case COM_CHAR:
    case COM_CHARACTER:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        reduce_minabs( (char*)&pcb.inbuf[ _ncomp_bytes*k],
                       (char*)&ptr[ strd_bytes*vs[ to]], _ncomp);
      break;
    case COM_INT:
    case COM_INTEGER:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        reduce_minabs( (int*)&pcb.inbuf[ _ncomp_bytes*k],
                       (int*)&ptr[ strd_bytes*vs[ to]], _ncomp);
      break;
    case COM_FLOAT:
    case COM_REAL:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        reduce_minabs( (float*)&pcb.inbuf[ _ncomp_bytes*k], 
                       (float*)&ptr[ strd_bytes*vs[ to]], _ncomp);
      break;
    case COM_DOUBLE:
    case COM_DOUBLE_PRECISION:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        reduce_minabs( (double*)&pcb.inbuf[ _ncomp_bytes*k], 
                       (double*)&ptr[ strd_bytes*vs[ to]], _ncomp);
      break;
    default:
      COM_assertion_msg(false, "Unknown data type"); // Not supported
    }

    // Remove the received message from the list
    _reqs_recv.erase(_reqs_recv.begin()+index);
    _reqs_indices.erase(_reqs_indices.begin()+index);
  }
}

Here is the call graph for this function:

void reduce_on_shared_nodes

(

MPI_Op

op

)

Perform a reduction operation using locally cached values of the shared nodes, assuming begin_update_shared_nodes() has been called.

Definition at line 401 of file Pane_communicator.C.

References _comm, _my_pconn_id, _ncomp, _ncomp_bytes, _panes, _ptrs, _reqs_indices, _reqs_recv, _shr_buffs, _strds, _type, COM_assertion, COM_assertion_msg, COM_CHAR, COM_CHARACTER, COM_DOUBLE, COM_DOUBLE_PRECISION, COM_FLOAT, COM_get_sizeof(), COM_INT, COM_INTEGER, COM_REAL, i, Pane_communicator::Pane_comm_buffers::inbuf, Pane_communicator::Pane_comm_buffers::index, j, k, nk, reduce_int(), and reduce_real().

Referenced by reduce_average_on_shared_nodes().

                                                         {
  while ( !_reqs_recv.empty()) {
    int index;

    // Wait for any receive request to finish and then process the request
    if ( _comm!=MPI_COMM_NULL) {
      MPI_Status status;
      int ierr = MPI_Waitany( _reqs_recv.size(), &_reqs_recv[0], 
                              &index, &status);
      COM_assertion( ierr == 0);
    }
    else
      index = _reqs_recv.size()-1;

    // Obtain the indices in _shr_buffs for the receive request
    int i=_reqs_indices[index].first, j=(_reqs_indices[index].second>>4);

    int strd_bytes = COM_get_sizeof( _type, _strds[i]);
    // Shift the pointer by -1 because node IDs in pconn start from 1
    char *ptr = ((char*)_ptrs[i])-strd_bytes;

    const COM::Attribute *pconn = _panes[i]->attribute(_my_pconn_id);
    const int *vs = (const int*)pconn->pointer();

    Pane_comm_buffers &pcb = _shr_buffs[i][j];
    COM_assertion( int(pcb.inbuf.size()) == _ncomp_bytes*vs[ pcb.index+1]);
    
    switch ( _type) {
    case COM_CHAR:
    case COM_CHARACTER:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        reduce_int(op, (char*)&pcb.inbuf[ _ncomp_bytes*k],
                   ((char*)&ptr[strd_bytes*vs[ to]]), _ncomp);
      break;
    case COM_INT:
    case COM_INTEGER:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        reduce_int(op, (int*)&pcb.inbuf[ _ncomp_bytes*k],
                   (int*)&ptr[strd_bytes*vs[ to]], _ncomp);
      break;
    case COM_FLOAT:
    case COM_REAL:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        reduce_real( op, (float*)&pcb.inbuf[ _ncomp_bytes*k],
                     (float*)&ptr[ strd_bytes*vs[ to]], _ncomp);
      break;
    case COM_DOUBLE:
    case COM_DOUBLE_PRECISION:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        reduce_real( op, (double*)&pcb.inbuf[ _ncomp_bytes*k], 
                     (double*)&ptr[strd_bytes*vs[ to]], _ncomp);
      break;
    default:
      COM_assertion_msg(false, "Unknown data type"); // Not supported
    }

    // Remove the received message from the list
    _reqs_recv.erase(_reqs_recv.begin()+index);
    _reqs_indices.erase(_reqs_indices.begin()+index);
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

int total_npanes ( ) const

inline

Obtain the total number of panes on all processes.

Definition at line 108 of file Pane_communicator.h.

References _total_npanes.

{ return _total_npanes; }

void update_ghost_values

(

)

Definition at line 693 of file Pane_communicator.C.

References _comm, _gcr_buffs, _gnr_buffs, _my_pconn_id, _ncomp, _ncomp_bytes, _panes, _ptrs, _reqs_indices, _reqs_recv, _strds, _type, COM_assertion, COM_assertion_msg, COM_CHAR, COM_CHARACTER, COM_DOUBLE, COM_DOUBLE_PRECISION, COM_FLOAT, COM_get_sizeof(), COM_INT, COM_INTEGER, COM_REAL, GNR, i, Pane_communicator::Pane_comm_buffers::inbuf, Pane_communicator::Pane_comm_buffers::index, j, k, nk, and update_value().

Referenced by end_update_ghost_cells(), and end_update_ghost_nodes().

                                            {
  while ( !_reqs_recv.empty()) {
    int index;

    // Wait for any receive request to finish and then process the request
    if ( _comm!=MPI_COMM_NULL) {
      MPI_Status status;
      int ierr = MPI_Waitany( _reqs_recv.size(), &_reqs_recv[0], 
                              &index, &status);
      COM_assertion( ierr == 0);
    }
    else
      index = _reqs_recv.size()-1;

    // Obtain the indices in _shr_buffs for the receive request
    int i=_reqs_indices[index].first, j=(_reqs_indices[index].second>>4);

    int btype = (_reqs_indices[index].second)&15;

    int strd_bytes = COM_get_sizeof( _type, _strds[i]);
    // Shift the pointer by -1 because node IDs in pconn start from 1
    char *ptr = ((char*)_ptrs[i])-strd_bytes;

    const COM::Attribute *pconn = _panes[i]->attribute(_my_pconn_id);
    const int *vs = (const int*)pconn->pointer();

    Pane_comm_buffers &pcb = (btype==GNR) ? _gnr_buffs[i][j] : _gcr_buffs[i][j] ;
    COM_assertion( int(pcb.inbuf.size()) == _ncomp_bytes*vs[ pcb.index+1]);

    switch ( _type) {
    case COM_CHAR:
    case COM_CHARACTER:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        update_value( (char*)&pcb.inbuf[ _ncomp_bytes*k],
                      (char*)&ptr[ strd_bytes*vs[ to]], _ncomp);
      break;
    case COM_INT:
    case COM_INTEGER:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        update_value( (int*)&pcb.inbuf[ _ncomp_bytes*k],
                      (int*)&ptr[ strd_bytes*vs[ to]], _ncomp);
      break;
    case COM_FLOAT:
    case COM_REAL:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        update_value( (float*)&pcb.inbuf[ _ncomp_bytes*k], 
                      (float*)&ptr[ strd_bytes*vs[ to]], _ncomp);
      break;
    case COM_DOUBLE:
    case COM_DOUBLE_PRECISION:
      for ( int k=0, nk=vs[pcb.index+1], to=pcb.index+2; k<nk; ++k, ++to)
        update_value( (double*)&pcb.inbuf[ _ncomp_bytes*k], 
                      (double*)&ptr[ strd_bytes*vs[ to]], _ncomp);
      break;
    default:
      COM_assertion_msg(false, "Unknown data type"); // Not supported
    }

    // Remove the received message from the list
    _reqs_recv.erase(_reqs_recv.begin()+index);
    _reqs_indices.erase(_reqs_indices.begin()+index);
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

Member Data Documentation

COM::Window* _appl_window

protected

Reference to the application window.

Definition at line 196 of file Pane_communicator.h.

Referenced by init(), owner_rank(), and Pane_communicator().

const MPI_Comm _comm

protected

MPI Communicator.

Definition at line 198 of file Pane_communicator.h.

Referenced by begin_update(), end_update(), mpi_comm(), reduce_diff_on_shared_nodes(), reduce_maxabs_on_shared_nodes(), reduce_minabs_on_shared_nodes(), reduce_on_shared_nodes(), and update_ghost_values().

std::vector< std::vector< Pane_comm_buffers> > _gcr_buffs

protected

Buffer for ghost cells to receive.

Definition at line 227 of file Pane_communicator.h.

Referenced by begin_update(), init(), and update_ghost_values().

std::vector< std::vector< Pane_comm_buffers> > _gnr_buffs

protected

Buffer for ghost nodes to receive.

Definition at line 223 of file Pane_communicator.h.

Referenced by begin_update(), init(), and update_ghost_values().

std::map<int,int> _lpaneid_map

protected

Mapping from user-defined pane ids to internal IDs, which are unique and contiguous across all processes, which are useful for defining unique tags for MPI messages.

Definition at line 206 of file Pane_communicator.h.

Referenced by lpaneid(), and Pane_communicator().

int _my_pconn_id

protected

The id of the pconn being used.

Definition at line 193 of file Pane_communicator.h.

Referenced by begin_update(), init(), Pane_communicator(), reduce_average_on_shared_nodes(), reduce_diff_on_shared_nodes(), reduce_maxabs_on_shared_nodes(), reduce_minabs_on_shared_nodes(), reduce_on_shared_nodes(), and update_ghost_values().

int _ncomp

protected

Definition at line 209 of file Pane_communicator.h.

Referenced by init(), reduce_average_on_shared_nodes(), reduce_diff_on_shared_nodes(), reduce_maxabs_on_shared_nodes(), reduce_minabs_on_shared_nodes(), reduce_on_shared_nodes(), and update_ghost_values().

int _ncomp_bytes

protected

Definition at line 209 of file Pane_communicator.h.

Referenced by begin_update(), init(), reduce_diff_on_shared_nodes(), reduce_maxabs_on_shared_nodes(), reduce_minabs_on_shared_nodes(), reduce_on_shared_nodes(), and update_ghost_values().

std::vector<COM::Pane*> _panes

protected

Vector of all local panes.

Definition at line 200 of file Pane_communicator.h.

Referenced by begin_update(), init(), Pane_communicator(), panes(), reduce_average_on_shared_nodes(), reduce_diff_on_shared_nodes(), reduce_maxabs_on_shared_nodes(), reduce_minabs_on_shared_nodes(), reduce_on_shared_nodes(), and update_ghost_values().

std::vector< void*> _ptrs

protected

An array of pointers to the data for all local panes.

Definition at line 211 of file Pane_communicator.h.

Referenced by begin_update(), init(), reduce_average_on_shared_nodes(), reduce_diff_on_shared_nodes(), reduce_maxabs_on_shared_nodes(), reduce_minabs_on_shared_nodes(), reduce_on_shared_nodes(), and update_ghost_values().

std::vector< std::vector< Pane_comm_buffers> > _rcs_buffs

protected

Buffer for real cells to send.

Definition at line 225 of file Pane_communicator.h.

Referenced by begin_update(), and init().

std::vector<std::pair<int,int> > _reqs_indices

protected

The indices in buffs for each pending nonblocking receive request.

Definition at line 231 of file Pane_communicator.h.

Referenced by begin_update(), reduce_diff_on_shared_nodes(), reduce_maxabs_on_shared_nodes(), reduce_minabs_on_shared_nodes(), reduce_on_shared_nodes(), and update_ghost_values().

std::vector<MPI_Request> _reqs_recv

protected

Definition at line 229 of file Pane_communicator.h.

Referenced by begin_update(), reduce_diff_on_shared_nodes(), reduce_maxabs_on_shared_nodes(), reduce_minabs_on_shared_nodes(), reduce_on_shared_nodes(), and update_ghost_values().

std::vector<MPI_Request> _reqs_send

protected

Arrays of pending nonblocking MPI requests. Same format as _shr_buffs.

Definition at line 229 of file Pane_communicator.h.

Referenced by begin_update(), and end_update().

std::vector< std::vector< Pane_comm_buffers> > _rns_buffs

protected

Buffer for real nodes to send.

Definition at line 221 of file Pane_communicator.h.

Referenced by begin_update(), and init().

std::vector< std::vector< Pane_comm_buffers> > _shr_buffs

protected

Shared node pane communication buffers.

The outer vector corresponds to local panes; the inner vectors corresponds to the

Definition at line 219 of file Pane_communicator.h.

Referenced by begin_update(), init(), reduce_diff_on_shared_nodes(), reduce_maxabs_on_shared_nodes(), reduce_minabs_on_shared_nodes(), and reduce_on_shared_nodes().

std::vector< int> _sizes

protected

The sizes of the arrays for all local panes.

Definition at line 213 of file Pane_communicator.h.

Referenced by begin_update(), and init().

std::vector< int> _strds

protected

The strides of the arrays for all local panes.

Definition at line 215 of file Pane_communicator.h.

Referenced by begin_update(), init(), reduce_average_on_shared_nodes(), reduce_diff_on_shared_nodes(), reduce_maxabs_on_shared_nodes(), reduce_minabs_on_shared_nodes(), reduce_on_shared_nodes(), and update_ghost_values().

int _total_npanes

protected

The total number of panes on all processes.

Definition at line 202 of file Pane_communicator.h.

Referenced by begin_update(), init_pane_comm_buffers(), lpaneid(), Pane_communicator(), and total_npanes().

int _type

protected

The base data type, number of components, and the number of bytes of all components for the data to be communicated.

Definition at line 209 of file Pane_communicator.h.

Referenced by begin_update(), init(), reduce_average_on_shared_nodes(), reduce_diff_on_shared_nodes(), reduce_maxabs_on_shared_nodes(), reduce_minabs_on_shared_nodes(), reduce_on_shared_nodes(), and update_ghost_values().

---

The documentation for this class was generated from the following files:

• Pane_communicator.h
• Pane_communicator.C