ug4
pcl::InterfaceCommunicator< TLayout > Class Template Reference

Performs communication between interfaces on different processes. More...

#include <pcl_interface_communicator.h>

Classes

struct  ExtractorInfo
 holds information that will be passed to the extract routines. More...
 

Public Types

typedef Layout::Interface Interface
 
typedef TLayout Layout
 
typedef Layout::Type Type
 

Public Member Functions

bool communicate (int tag=749345)
 sends and receives the collected data. More...
 
bool communicate_and_resume (int tag=749345)
 collects data and communicates it with other processes without waiting for receive More...
 
bool communication_debugging_enabled ()
 returns true if communication debugging is enabled More...
 
void disable_communication_debugging ()
 disables debugging of communication More...
 
void enable_communication_debugging (const ProcessCommunicator &involvedProcs=ProcessCommunicator(PCD_WORLD))
 enables debugging of communication. This has a severe effect on performance! More...
 
template<class TLayoutMap >
void exchange_data (const TLayoutMap &layoutMap, const typename TLayoutMap::Key &keyFrom, const typename TLayoutMap::Key &keyTo, ICommunicationPolicy< TLayout > &commPol)
 internally calls send_data and receive_data with the specified layouts. More...
 
 InterfaceCommunicator ()
 
void receive_data (const Layout &layout, ICommunicationPolicy< TLayout > &commPol)
 registers an communication-policy to receive data on communicate. More...
 
void receive_data (int srcProc, const Interface &interface, ICommunicationPolicy< TLayout > &commPol)
 registers a communication-policy to receive data on communicate. More...
 
void receive_raw (int srcProc, ug::BinaryBuffer &bufOut, int bufSize=-1)
 registers a binary-stream to receive data from a source-proc. More...
 
void receive_raw (int srcProc, void *bufOut, int bufSize)
 registers a buffer to receive data from a source-proc. More...
 
void send_data (const Layout &layout, ICommunicationPolicy< TLayout > &commPol)
 collects data that will be send during communicate. More...
 
void send_data (int targetProc, const Interface &interface, ICommunicationPolicy< TLayout > &commPol)
 collects data that will be send during communicate. More...
 
void send_raw (int targetProc, const void *pBuff, int bufferSize, bool bSizeKnownAtTarget=false)
 sends raw data to a target-proc. More...
 
void wait ()
 waits for the data communicated by communicate_and_resume() and extracts it More...
 

Protected Types

typedef std::map< int, ug::BinaryBufferBufferMap
 
typedef ICommunicationPolicy< LayoutCommPol
 
typedef std::list< ExtractorInfoExtractorInfoList
 A list that holds information about extractors. More...
 

Protected Member Functions

bool collect_layout_buffer_sizes (const TLayout &layout, ICommunicationPolicy< TLayout > &commPol, std::map< int, int > *pMapBuffSizesOut, const layout_tags::multi_level_layout_tag &)
 collects buffer sizes for a given layout and stores them in a map More...
 
bool collect_layout_buffer_sizes (const TLayout &layout, ICommunicationPolicy< TLayout > &commPol, std::map< int, int > *pMapBuffSizesOut, const layout_tags::single_level_layout_tag &)
 collects buffer sizes for a given layout and stores them in a map More...
 
void extract_data (const TLayout &layout, BufferMap &bufMap, CommPol &extractor)
 extract data from stream-pack More...
 
void extract_data (const TLayout &layout, BufferMap &bufMap, CommPol &extractor, const layout_tags::multi_level_layout_tag &)
 
void extract_data (const TLayout &layout, BufferMap &bufMap, CommPol &extractor, const layout_tags::single_level_layout_tag &)
 
void prepare_receiver_buffer_map (BufferMap &bufMap, std::set< int > &curProcs, const TLayout &layout)
 prepare stream-pack-in More...
 
void prepare_receiver_buffer_map (BufferMap &streamPack, std::set< int > &curProcs, const TLayout &layout, const layout_tags::multi_level_layout_tag &)
 specialization of stream-pack preparation for multi-level-layouts More...
 
void prepare_receiver_buffer_map (BufferMap &streamPack, std::set< int > &curProcs, const TLayout &layout, const layout_tags::single_level_layout_tag &)
 specialization of stream-pack preparation for single-level-layouts More...
 
void send_data (const Layout &layout, ICommunicationPolicy< TLayout > &commPol, const layout_tags::multi_level_layout_tag &)
 helper to collect data from multi-level-layouts
More...
 
void send_data (const Layout &layout, ICommunicationPolicy< TLayout > &commPol, const layout_tags::single_level_layout_tag &)
 helper to collect data from single-level-layouts More...
 

Protected Attributes

bool m_bDebugCommunication
 true if the communication shall be debugged. More...
 
bool m_bSendBuffersFixed
 holds info whether all send-buffers are of predetermined fixed size. More...
 
BufferMap m_bufMapIn
 holds the buffers that are used to receive data More...
 
BufferMap m_bufMapOut
 holds the buffers that are used to send data More...
 
int m_curComTag
 This is the tag for the currently performed communication. More...
 
std::set< int > m_curInProcs
 stores in-procs for the next communication step More...
 
std::set< int > m_curOutProcs
 stores out-procs for the next communication step More...
 
ProcessCommunicator m_debugProcComm
 This procComm holds the processes that shall participate during communication-debugging. More...
 
ExtractorInfoList m_extractorInfos
 holds information about the extractors that are awaiting data. More...
 
std::vector< MPI_Request > m_vReceiveRequests
 
std::vector< MPI_Request > m_vSendRequests
 used by communicate, communicate_and_resume and wait, to check whether communication is done. More...
 

Detailed Description

template<class TLayout>
class pcl::InterfaceCommunicator< TLayout >

Performs communication between interfaces on different processes.

Member Typedef Documentation

◆ BufferMap

template<class TLayout >
typedef std::map<int, ug::BinaryBuffer> pcl::InterfaceCommunicator< TLayout >::BufferMap
protected

◆ CommPol

template<class TLayout >
typedef ICommunicationPolicy<Layout> pcl::InterfaceCommunicator< TLayout >::CommPol
protected

◆ ExtractorInfoList

template<class TLayout >
typedef std::list<ExtractorInfo> pcl::InterfaceCommunicator< TLayout >::ExtractorInfoList
protected

A list that holds information about extractors.

◆ Interface

template<class TLayout >
typedef Layout::Interface pcl::InterfaceCommunicator< TLayout >::Interface

◆ Layout

template<class TLayout >
typedef TLayout pcl::InterfaceCommunicator< TLayout >::Layout

◆ Type

template<class TLayout >
typedef Layout::Type pcl::InterfaceCommunicator< TLayout >::Type

Constructor & Destructor Documentation

◆ InterfaceCommunicator()

template<class TLayout >
pcl::InterfaceCommunicator< TLayout >::InterfaceCommunicator

Member Function Documentation

◆ collect_layout_buffer_sizes() [1/2]

template<class TLayout >
bool pcl::InterfaceCommunicator< TLayout >::collect_layout_buffer_sizes ( const TLayout &  layout,
ICommunicationPolicy< TLayout > &  commPol,
std::map< int, int > *  pMapBuffSizesOut,
const layout_tags::multi_level_layout_tag  
)
protected

collects buffer sizes for a given layout and stores them in a map

The given map holds pairs of procID, bufferSize If buffer-sizes can't be determined, false is returned. if pMmapBuffSizesOut == NULL, the method will simply determine whether all buffersizes can be calculated.

◆ collect_layout_buffer_sizes() [2/2]

template<class TLayout >
bool pcl::InterfaceCommunicator< TLayout >::collect_layout_buffer_sizes ( const TLayout &  layout,
ICommunicationPolicy< TLayout > &  commPol,
std::map< int, int > *  pMapBuffSizesOut,
const layout_tags::single_level_layout_tag  
)
protected

collects buffer sizes for a given layout and stores them in a map

The given map holds pairs of procID, bufferSize If buffer-sizes can't be determined, false is returned. if pMmapBuffSizesOut == NULL, the method will simply determine whether all buffersizes can be calculated.

◆ communicate()

template<class TLayout >
bool pcl::InterfaceCommunicator< TLayout >::communicate ( int  tag = 749345)

sends and receives the collected data.

The collected data will be send to the associated processes. The extract routines of the communication-policies which were registered through Communicator::receive_data will be called with the received data. After all received data is processed, the communication-policies are released. Make sure that you will keep your communication-policies in memory until this point.

Note
Calling communicate() is effectively the same as calling communicate_and_resume() directly followed by wait().
Parameters
tagFor internal communications the provided tag is used. The default value is fine in most cases and normally only has to be adjusted if one performs multiple communications at the same time (e.g. with different communicators).

Referenced by ug::AddConnectionsBetweenSlaves(), ug::AddExtraProcessEntriesToSubdomainLayout(), ug::AdditiveToConsistent(), ug::AdditiveToUnique(), ug::SurfaceView::adjust_parallel_surface_states(), ug::ApplySmoothManifoldPosToTopLevelAveragingScheme(), ug::ApplySmoothManifoldPosToTopLevelButterflyScheme(), ug::ApplySmoothManifoldPosToTopLevelLoopScheme(), ug::ApplySmoothVolumePosToTopLevel(), ug::AttachmentAllReduce(), ug::BroadcastVectorFromOne(), ug::BuildDomainDecompositionLayouts(), ug::BuildOneToManyLayout(), ug::CalculateNumElemsVertexAttachmentInTopLevel(), ug::CalculateNumManifoldEdgesVertexAttachmentInParentLevel(), ug::CalculateNumManifoldFacesVertexAttachmentInTopLevel(), ug::ParallelVector< TVector >::check_storage_type(), ug::CheckDistributedObjectConstraintTypes(), ug::ParallelColoring::color(), ug::CommunicateConnections(), ug::CommunicateOnInterfaces(), ug::CopyValues(), ug::CreateAndDistributeGlobalIDs(), ug::CreateSurfaceView(), ug::ComPol_CheckDistributedParentStates< TLayout >::exchange_data(), ug::ComPol_NewConstrainedVerticals< TLayout >::exchange_data(), ug::ExchangeAndAdjustSideErrors(), ug::GatherVectorOnOne(), ug::ParallelDualGraph< TGeomBaseObj, TIndexType, TConnectingObj >::generate_graph(), ug::GenerateGlobalAlgebraIDs(), ug::GenerateGlobalConsecutiveIndices(), ug::GridFunction< TDomain, TAlgebra >::grid_changed_callback(), ug::GridFunction< TDomain, TAlgebra >::grid_distribution_callback(), ug::MarkForAdaption_GradientAverage(), ug::MatAddSlaveRowsToMasterRowOverlap0(), ug::MatMakeConsistentOverlap0(), ug::PartitionMultiGridLevel_ParmetisKway(), ug::ProjectHierarchyToSubdivisionLimit(), ug::CopyAttachmentHandler< TElem, TAttachment >::propagate_to_level(), ug::ReceiveGlobalLayout(), ug::ReceiveMatrix(), ug::ReceiveOnInterfaces(), ug::SendGlobalLayout(), ug::SendMatrix(), ug::SendOnInterfaces(), ug::SideAndElemErrEstData< TDomain >::summarize_err_est_data(), ug::SynchronizeDistInfos(), pcl::TestLayoutIsDoubleEnded(), pcl::TestSizeOfInterfacesInLayoutsMatch(), ug::UniqueToConsistent(), ug::VecBroadcast(), ug::VecCopy(), ug::VecGather(), ug::VecSubtractOneSlaveFromMaster(), and ug::VecSubtractOnLayout().

◆ communicate_and_resume()

template<class TLayout >
bool pcl::InterfaceCommunicator< TLayout >::communicate_and_resume ( int  tag = 749345)

collects data and communicates it with other processes without waiting for receive

Data will be collected from the communication-policies registered in send(...) and communicated to other processes. The method however won't wait until the data has arrived at its target processes and thus can't extract the received data. This has to be done manually by calling wait().

Note
Instead of using communicate_and_resume() and wait() you could simply call communicate. Separating communication and wait however gives you the benefit of being able to continue with other calculations while communication is performed.
A call to communicate_and_resume() has to be followed by a call to wait(). You may not call communicate_and_resume() twice on a single communicator without callin wait() in between.
Parameters
tagFor internal communications the provided tag is used. The default value is fine in most cases and normally only has to be adjusted if one performs multiple communications at the same time (e.g. with different communicators).

References ug::BinaryBuffer::buffer(), pcl::ICommunicationPolicy< TLayout >::get_required_buffer_size(), ug::LIB_PCL, pcl::InterfaceCommunicator< TLayout >::ExtractorInfo::m_extractor, pcl::InterfaceCommunicator< TLayout >::ExtractorInfo::m_interface, pcl::InterfaceCommunicator< TLayout >::ExtractorInfo::m_layout, pcl::InterfaceCommunicator< TLayout >::ExtractorInfo::m_rawSize, pcl::InterfaceCommunicator< TLayout >::ExtractorInfo::m_srcProc, PCL_COMM_WORLD, PCL_PROFILE, ug::BinaryBuffer::reserve(), pcl::SendRecvBuffersMatch(), pcl::SendRecvListsMatch(), ug::BinaryBuffer::set_write_pos(), UG_DLOG, UG_LOG, UG_THROW, pcl::Waitall(), and ug::BinaryBuffer::write_pos().

◆ communication_debugging_enabled()

template<class TLayout >
bool pcl::InterfaceCommunicator< TLayout >::communication_debugging_enabled

returns true if communication debugging is enabled

◆ disable_communication_debugging()

template<class TLayout >
void pcl::InterfaceCommunicator< TLayout >::disable_communication_debugging

disables debugging of communication

◆ enable_communication_debugging()

template<class TLayout >
void pcl::InterfaceCommunicator< TLayout >::enable_communication_debugging ( const ProcessCommunicator involvedProcs = ProcessCommunicator(PCD_WORLD))

enables debugging of communication. This has a severe effect on performance!

communication debugging will execute some code during communicate(), which checks whether matching sends and receives have been scheduled with matching buffer sizes.

If not all processes participate during communication, you have to specify a process-communicator (involvedProcs), which includes all and only the procs, which will call communicate later on. If no process communicator is specified, then PCD_WORLD is used.

Note that communication debugging introduces additional communication which considerable slows down performance. You should only use it temporarily, if you encounter problems during communication.

Don't forget to call disable_communication_debugging(), when you're done with debugging.

References UG_LOG.

◆ exchange_data()

template<class TLayout >
template<class TLayoutMap >
void pcl::InterfaceCommunicator< TLayout >::exchange_data ( const TLayoutMap &  layoutMap,
const typename TLayoutMap::Key &  keyFrom,
const typename TLayoutMap::Key &  keyTo,
ICommunicationPolicy< TLayout > &  commPol 
)

internally calls send_data and receive_data with the specified layouts.

Note that data is not communicated until communicate has been called.

TLayout has to feature the following typedefs and methods:

// The type of the key with which a layout can be identified.
Key
// returns true, if the layout exists.
template <class TType>
bool has_layout(const TLayoutMap::Key& key);
// returns the layout that is associated with the given key.
template <class TType>
TLayout& get_layout(const TLayoutMap::Key& key);

The methods will only be called with type InterfaceCommunicator::Type.

This method is particularily useful if you categorize layouts on a process. If you separate your layouts into master and slave layouts, you could use this method e.g. to copy data from all master-layouts to all slave-layouts of a type with a single call.

Referenced by ug::SurfaceView::adjust_parallel_surface_states(), ug::ApplySmoothManifoldPosToTopLevelAveragingScheme(), ug::ApplySmoothManifoldPosToTopLevelButterflyScheme(), ug::ApplySmoothManifoldPosToTopLevelLoopScheme(), ug::ApplySmoothVolumePosToTopLevel(), ug::AttachmentAllReduce(), ug::CalculateNumElemsVertexAttachmentInTopLevel(), ug::CalculateNumManifoldEdgesVertexAttachmentInParentLevel(), ug::CalculateNumManifoldFacesVertexAttachmentInTopLevel(), ug::CheckDistributedObjectConstraintTypes(), ug::CreateAndDistributeGlobalIDs(), ug::ComPol_NewConstrainedVerticals< TLayout >::exchange_data(), ug::ExchangeAndAdjustSideErrors(), ug::GridFunction< TDomain, TAlgebra >::grid_changed_callback(), ug::GridFunction< TDomain, TAlgebra >::grid_distribution_callback(), ug::MarkForAdaption_GradientAverage(), ug::ProjectHierarchyToSubdivisionLimit(), ug::CopyAttachmentHandler< TElem, TAttachment >::propagate_to_level(), ug::SideAndElemErrEstData< TDomain >::summarize_err_est_data(), and ug::SynchronizeDistInfos().

◆ extract_data() [1/3]

template<class TLayout >
void pcl::InterfaceCommunicator< TLayout >::extract_data ( const TLayout &  layout,
BufferMap bufMap,
CommPol extractor 
)
protected

extract data from stream-pack

References PCL_PROFILE_FUNC.

◆ extract_data() [2/3]

◆ extract_data() [3/3]

◆ prepare_receiver_buffer_map() [1/3]

template<class TLayout >
void pcl::InterfaceCommunicator< TLayout >::prepare_receiver_buffer_map ( BufferMap bufMap,
std::set< int > &  curProcs,
const TLayout &  layout 
)
protected

prepare stream-pack-in

◆ prepare_receiver_buffer_map() [2/3]

template<class TLayout >
void pcl::InterfaceCommunicator< TLayout >::prepare_receiver_buffer_map ( BufferMap streamPack,
std::set< int > &  curProcs,
const TLayout &  layout,
const layout_tags::multi_level_layout_tag  
)
protected

specialization of stream-pack preparation for multi-level-layouts

◆ prepare_receiver_buffer_map() [3/3]

template<class TLayout >
void pcl::InterfaceCommunicator< TLayout >::prepare_receiver_buffer_map ( BufferMap streamPack,
std::set< int > &  curProcs,
const TLayout &  layout,
const layout_tags::single_level_layout_tag  
)
protected

specialization of stream-pack preparation for single-level-layouts

◆ receive_data() [1/2]

template<class TLayout >
void pcl::InterfaceCommunicator< TLayout >::receive_data ( const Layout layout,
ICommunicationPolicy< TLayout > &  commPol 
)

registers an communication-policy to receive data on communicate.

Receives have to be registered before communicate is executed. make sure that your instance of the communication-policy exists until communicate has benn executed.

◆ receive_data() [2/2]

◆ receive_raw() [1/2]

template<class TLayout >
void pcl::InterfaceCommunicator< TLayout >::receive_raw ( int  srcProc,
ug::BinaryBuffer bufOut,
int  bufSize = -1 
)

registers a binary-stream to receive data from a source-proc.

Receives data that has been sent with send_raw. If send_raw was called with bSizeKnownAtTarget == true, an exact bufferSize has to be specified. If bSizeKnownAtTarget was set to false, bufferSize has to be set to -1.

Make sure that binStreamOut exists until communicate has been executed.

Please note that this method should only be used if custom data should be send in a block with data that is communicated through interfaces, since an additional copy-operation has to be performed. If you're only interested in sending raw data, you should take a look into pcl::ProcessCommunicator::receive.

Referenced by ug::AddConnectionsBetweenSlaves(), ug::ParallelColoring::color(), ug::CommunicateOnInterfaces(), ug::NewLayoutCreator::issue(), ug::ParallelNodes::issue(), ug::RowSendingScheme< matrix_type >::issue_send(), ug::ReceiveGlobalLayout(), ug::ReceiveMatrix(), ug::ReceiveOnInterfaces(), pcl::TestLayoutIsDoubleEnded(), and pcl::TestSizeOfInterfacesInLayoutsMatch().

◆ receive_raw() [2/2]

template<class TLayout >
void pcl::InterfaceCommunicator< TLayout >::receive_raw ( int  srcProc,
void *  bufOut,
int  bufSize 
)

registers a buffer to receive data from a source-proc.

Receives data that has been sent with send_raw. This method may only be used if send_raw was called with bSizeKnownAtTarget == true. Call receive_raw with a binary- stream instead, if buffer-sizes are not known.

Make sure that the buffer points to valid memory until communicate has been executed.

Please note that this method should only be used if custom data should be send in a block with data that is communicated through interfaces, since an additional copy-operation has to be performed. If you're only interested in sending raw data, you should take a look into pcl::ProcessCommunicator::receive.

◆ send_data() [1/4]

template<class TLayout >
void pcl::InterfaceCommunicator< TLayout >::send_data ( const Layout layout,
ICommunicationPolicy< TLayout > &  commPol 
)

collects data that will be send during communicate.

Calls ICommunicationPolicy<TLayout>::collect with the specified layout and the binary stream that is associated with the layouts target processes. Note that data will not be send until communicate has been called.

See also
receive_data, exchange_data

References PCL_PROFILE.

◆ send_data() [2/4]

template<class TLayout >
void pcl::InterfaceCommunicator< TLayout >::send_data ( const Layout layout,
ICommunicationPolicy< TLayout > &  commPol,
const layout_tags::multi_level_layout_tag  
)
protected

helper to collect data from multi-level-layouts

◆ send_data() [3/4]

template<class TLayout >
void pcl::InterfaceCommunicator< TLayout >::send_data ( const Layout layout,
ICommunicationPolicy< TLayout > &  commPol,
const layout_tags::single_level_layout_tag  
)
protected

helper to collect data from single-level-layouts

◆ send_data() [4/4]

◆ send_raw()

template<class TLayout >
void pcl::InterfaceCommunicator< TLayout >::send_raw ( int  targetProc,
const void *  pBuff,
int  bufferSize,
bool  bSizeKnownAtTarget = false 
)

sends raw data to a target-proc.

Shedules the data in pBuff to be sent to the target-proc pBuff can be reused or cleared directly after the call returns.

Data sent with this method can be received using receive_raw.

Please note that this method should only be used if custom data should be send in a block with data that is communicated through interfaces, since an additional copy-operation at the target process has to be performed. If you're only interested in sending raw data, you should take a look into pcl::ProcessCommunicator::send.

References pcl::NumProcs(), and ug::BinaryBuffer::write().

Referenced by ug::AddConnectionsBetweenSlaves(), ug::ParallelColoring::color(), ug::CommunicateOnInterfaces(), ug::NewLayoutCreator::issue(), ug::ParallelNodes::issue(), ug::RowSendingScheme< matrix_type >::issue_send(), ug::SendGlobalLayout(), ug::SendMatrix(), ug::SendOnInterfaces(), pcl::TestLayoutIsDoubleEnded(), and pcl::TestSizeOfInterfacesInLayoutsMatch().

◆ wait()

Member Data Documentation

◆ m_bDebugCommunication

template<class TLayout >
bool pcl::InterfaceCommunicator< TLayout >::m_bDebugCommunication
protected

true if the communication shall be debugged.

◆ m_bSendBuffersFixed

template<class TLayout >
bool pcl::InterfaceCommunicator< TLayout >::m_bSendBuffersFixed
protected

holds info whether all send-buffers are of predetermined fixed size.

reset to true after each communication-step.

◆ m_bufMapIn

template<class TLayout >
BufferMap pcl::InterfaceCommunicator< TLayout >::m_bufMapIn
protected

holds the buffers that are used to receive data

◆ m_bufMapOut

template<class TLayout >
BufferMap pcl::InterfaceCommunicator< TLayout >::m_bufMapOut
protected

holds the buffers that are used to send data

◆ m_curComTag

template<class TLayout >
int pcl::InterfaceCommunicator< TLayout >::m_curComTag
protected

This is the tag for the currently performed communication.

Set to -1 if no communication is currently performed.

◆ m_curInProcs

template<class TLayout >
std::set<int> pcl::InterfaceCommunicator< TLayout >::m_curInProcs
protected

stores in-procs for the next communication step

◆ m_curOutProcs

template<class TLayout >
std::set<int> pcl::InterfaceCommunicator< TLayout >::m_curOutProcs
protected

stores out-procs for the next communication step

◆ m_debugProcComm

template<class TLayout >
ProcessCommunicator pcl::InterfaceCommunicator< TLayout >::m_debugProcComm
protected

This procComm holds the processes that shall participate during communication-debugging.

◆ m_extractorInfos

template<class TLayout >
ExtractorInfoList pcl::InterfaceCommunicator< TLayout >::m_extractorInfos
protected

holds information about the extractors that are awaiting data.

◆ m_vReceiveRequests

template<class TLayout >
std::vector<MPI_Request> pcl::InterfaceCommunicator< TLayout >::m_vReceiveRequests
protected

◆ m_vSendRequests

template<class TLayout >
std::vector<MPI_Request> pcl::InterfaceCommunicator< TLayout >::m_vSendRequests
protected

used by communicate, communicate_and_resume and wait, to check whether communication is done.


The documentation for this class was generated from the following files: