Single-threaded implementation of RMA for distributed memory

------------------------------------------------------------------------

Base Assumptions

* All of the local windows are located in process local (not shared or
remotely accessible) memory.

* Only basic datatypes are supported for the target.

* Only active (fence) synchronization is supported.

* The application is single threaded.

* The MPI runtime system is single threaded.

------------------------------------------------------------------------

General Notes

* "Lessons Learned from Implmenenting BSP" by J. Hill and
  D.B. Skillicorn suggests that we should not be performing RMA
  operations as they are requested, but rather queue the entire set of
  operations and perform the operations at the next synchronization
  operation.

------------------------------------------------------------------------

Data Structures

* MPID_Win

  * struct MPIR_Win

  * handles - an array of local window handles (one per process)

    Q: Do we really need local window IDs?  We need to be able to map
    remote handler calls back to a particular window, but we might be
    able to do this using an attribute on a communicator.  Would an
    attribute lookup be too slow?

------------------------------------------------------------------------

MPID_Win_fence

* Since remote handler calls might be sent on another socket or
  processed in another thread, no natural synchronization occurs
  between RHCs and the collective operations.  Therefore, we need to
  know how many RHCs we should expect so that we don't prematurely
  return from the fence.  Likewise, we need to tell the other
  processes how many RHCs we have made.

* We need to block until such time that all incoming RHCs have been
  handled and all local requests and flags have completed.
  
  Q: What is the right interface for this blocking operation?  The
  operation should block, but it needs to guarantee that forward
  progress is being made on both the incoming RHCs and locally posted
  operations.  

  NOTE: We either need to pass dwin to a function or declare/cast the
  counters used in the while statement as volatile, otherwise the
  compiler may not generate instructions to reload the counter values
  before each iteration of the while loop.

  Q: It would be useful if the MPID layer could increment a counter
  (or call a non-blocking function) when the asynchronous request or
  RHC completed.  This seems like a much more ideal interface than
  requests and flags, at least for RMA.  Might something of this
  nature be possible without putting undo burden on the device or
  significantly complicating the ADI?

* Wait for all other processes in the window to complete

  Q: Should we perform a barrier here?  If we eliminate the barrier,
  then all processes still waiting for operations to complete will
  have to enqueue incoming requests from the next epoch until the
  operations from the currrent epoch are complete.  Not performing the
  barrier complicates the RMA operations, but the performance benefit
  may be significant for some cases.  (What are they?  How common are
  they?)

------------------------------------------------------------------------

MPID_Get

* If the target and origin ranks are the same, then copy the data from
  the target buffer to the origin buffer.

* Otherwise, we are attempting to get data from a remote node

  * Post an asynchronous receive for the data

    NOTE: the tag must be unique for this epoch so as to ensure that
    the soon-to-be incoming message is matched with this receive.

    NOTE: the request needs be allocated from the window's active
    requests object so that it can tracked.

  * Issue a remote handler call requesting the data from the remote
    process

    NOTE: the local completion flag needs be allocated from the
    window's active flags object so that it can tracked.

------------------------------------------------------------------------

MPIDI_Win_get_hdlr

* Post an asynchronous send of the requested data to origin process

* Increment the "number of RHCs processed" counter

------------------------------------------------------------------------

MPID_Put

* If the target and origin ranks are the same, then copy the data from
  the origin buffer to the target buffer.

* Otherwise, if the source and target buffers are contiguous and data
  conversion is not required

  NOTE: What I would like to do here is use MPID_Put_contig, but that
  would require that I communicate with the remote process in order to
  agree on a flag.  It would be much better if the target completion
  flag were a counter so that the counter could be prearranged and
  used for all Put operations.

* Otherwise, if the data is sufficiently small

  * If the data is not contiguous, then pack the data into a temporary
    buffer.

    NOTE: This assumes that MPID_Pack() does not add a header to the
    packed data.

  * Issue a remote handler call (MPIDI_Win_put_eager_hdlr) requesting
    the the data be written to the target's local window

* Otherwise, the data is large enough to send in a separate message(s)

  * Issue a remote handler call (MPIDI_Win_put_hdlr) letting the
    target now that data is being sent that needs to be written into
    the target's local window

  * Post an asynchronous send of the origin buffer

  Q: Instead of using MPI_Isend(), should we instead use segments and
  multiple RHCs to send the data?  Would doing so imply that the RMA
  subsystem now needs to do flow control?

Q: Should we have yet another case, where a rendezvous occurs,
guaranteeing that the target is able to post a receive before the send
is issued?  This would allow us to use MPID_Irsend(), potentially
eliminating an extra message.  Rather than having another case, should
we use this technique anytime the data is larger than the eager
message threshold?

------------------------------------------------------------------------

MPIDI_Win_put_eager_hdlr

* Unpack the data into local window buffer, performing data conversion
  if necessary

  Q: How are the header and data obtained?  Depending on the interface
  and the datatype, we should be able to read the header directly into
  the window buffer.

* Increment the "number of RHCs processed" counter

------------------------------------------------------------------------

MPIDI_Win_put_hdlr

  * Post an asynchronous receive of data into the window buffer
    defined in the RHC header

    Q: What should we do if a communication failure occurs?  Is the
    origin somehow notified of the failure?

* Increment the "number of RHCs processed" counter

------------------------------------------------------------------------

MPI_Accumulate

* If the target and origin ranks are the same, then copy the data from
  the target memory region to the origin memory region.

  NOTE: For now, we are assuming the application and the message agent
  are single-threaded so we do not need to hold a mutex before
  performing the operation.

* Otherwise, if the data is sufficiently small:

  * If the data is not contiguous, then pack the data into a temporary
    buffer.

    NOTE: This assumes that MPID_Pack() does not add a header to the
    packed data.

  * Issue a remote handler call (MPIDI_Win_acc_eager_hdlr) requesting
    the the enclosed data be accumulated into target buffer using the
    specified operation.

* Otherwise, the data is large enough that it needs to be segmented.

  Q: On the target side, we don't want to have to unpack the segment
  into a temporary buffer first.  We would like to do the data
  conversion and accumulation directly from the segment that will be
  received.  Does this make it impossible to use the MPIR_Segment API?

------------------------------------------------------------------------

MPIDI_Win_acc_eager_hdlr

* Perform the requested operation, converting the data on the fly if
  necessary

* Increment the "number of RHCs processed" counter

------------------------------------------------------------------------
