Changes between Version 33 and Version 34 of rpc_implementation


Ignore:
Timestamp:
Jan 17, 2019, 3:45:09 PM (5 years ago)
Author:
alain
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • rpc_implementation

    v33 v34  
    1515 * There is one kernel instance in each cluster containing at least one core, one local interrupt controler, and one physical memory bank.
    1616
    17 == 2) Server Core Selection ==
    18 
    19 Any client thread T running in any cluster K can send an PRC request to any cluster K'. In order to share the working load associated with RPC handling, each core in server cluster K'
    20 has a private RPC requests queue, where the client thread must register its RPC request. In principle, the client thread T running on the client core [i] select the waiting queue [i] in server cluster K'. If it is not possible (when the number of cores in cluster K' is smaller than the number of cores in client cluster), ALMOS-MKH selects core [0] in server cluster.
     17== 2) Inter-cluster communication ==
    2118
    2219ALMOS-MKH replicates the KDATA segment (containing the kernel global variables) in all clusters, and uses the same LPA (Local Physical Address) for the KDATA base in all clusters.
    2320Therefore, in two different clusters, a given global variable, identified by its LPA can have different values. This feature is used by by ALMOS-MKH to allow a client thread in cluster K to access a global variable in a server cluster K', building a physical address by concatenation of the LPA with the CXY cluster identifier for the server cluster K'.
     21
     22Any client thread T running in any cluster K can send an RPC request to any cluster K'. Each core in server cluster K' has a private RPC requests queue, where the client thread must register its RPC request. In order to share the working load associated with RPC handling, the client thread T running on the client core [i] select the waiting queue of core [i] in server cluster K'. If it is not possible (when the number of cores in cluster K' is smaller than the number of cores in client cluster), ALMOS-MKH selects core [0] in server cluster.
    2423
    2524For each core [i] in a cluster K, ALMOS-MKH implement the RPC requests queue as a software RPC_FIFO[i,k], implemented as a global variable in the KDATA segment. More precisely, each RPC_FIFO[i] has the type ''remote_fifo_t'', and is a member of the "cluster_manager" structure of cluster [k].
    … …  
    2928 * M is the number of server threads in charge of handling RPC requests stored in a given RPC_FIFO[i,k]. M is bounded by the CONFIG_RPC_THREAD_MAX parameter. For each PRC_FIFO[i,k], it can exist several server threads, because we must avoid the ''head-of-line blocking'' phenomenon, as explained below in section 4. To synchronize these multiple  server threads, the RPC FIFO implements a ''light lock'', that is a non blocking lock : only one RPC thread at a given time can take the lock and become the FIFO owner. Another RPC thread T' failing to take the lock simply returns to IDLE state.
    3029 
    31 == 3) Client / Server Synchronization ==
     30== 3) RPC descriptor format ==
    3231
    33 Simple RPC requests are blocking for the client thread: the client thread register its request in the target RPC_FIFO, blocks on the THREAD_BLOCKED_RPC condition, and deschedules. The client thread is unblocked by the RPC server thread, when the RPC is completed.
     32ALMOS-MKH supports two modes for an RPC request :
     33 * '''simple RPC''' : the client thread send the RPC request to one single server, and is expecting one single response.
     34 * '''parallel RPC''' : the client thread send in parallel several RPC requests to several servers, and is expecting several responses.
     35
     36Both RPC types use the same RPC descriptor format. One entry in the RPC_FIFO (located on the server side) contains a remote pointer (xptr_t) on the RPC descriptor (''rpc_desc_t''), that is stored on the client side. This RPC descriptor contains the following informations:
     37 * The '''index''' field defines the required service type (ALMOS-MKH defines about 30 service types).
     38 * The '''blocked''' field defines the RPC mode : true for a simple RPC, false for a parallel RPC.
     39 * The '''args''' field is an array of 10 uint64_t, containing the service arguments (both input & output).
     40 * The '''thread''' field is a local pointer on the client thread (used by the server thread to unblock the client thread).
     41 * The '''lid''' field defines the client core local index (used by the server thread to send the completion IPI).
     42 * The '''rsp''' field is a local pointer on an expected  responses counter on the client side (can be larger than one for a parallel RPC).
     43
     44The semantic of the '''args''' array depends on the service type, as defined by the '''index''' field.
     45
     46This format supports both simple and parallel RPCs: The client thread initializes the '''responses''' counter with the number of expected responses, and each server thread atomically decrement this counter when the RPC request has been satisfied.
     47
     48== 4) Simple RPC scenario ==
     49
     50Simple RPC requests are blocking for the client thread. The client thread must perform the following tasks:
     511. allocate memory in the client cluster for the RPC descriptor (can be in the client stack),
     521. allocate in the client cluster an expected response counter (can be in the client stack),
     531. initialize this RPC descriptor (this includes RPC arguments marshaling), as well as the responses counter (one expected response),
     541. register the extended pointer on the RPC descriptor in the server FIFO,
     551. send an IPI to the selected core in the server cluster,
     561. blocks and deschedule, waiting to be re-activated by the server thread when the server completed the requested service.
     57
     58For each RPC service type XYZ, ALMOS-MKH define a specific ''rpc_xyz_client()'' function that performs the 3 first tasks, and call the generic ''rpc_send()'' function to perform the three last tasks.
     59
     60On the server side, a  kernel RPC thread is activated at the next scheduling point on the selected server core, as soon as the RPC_FIFO is non-empty. This server thread executes the following tasks:
     611. extract relevant informations from RPC descriptor stored in client cluster,
     621. depending on the RPC index, call the specific ''roc_xyz_server()'' function to perform the RPC arguments unmarshmaling,
     631. call the recant kernel function to and execute the requested service,
     641. atomically increment the responses counter in the client cluster,
     651. if this response is the last expected response, unblocks the client thread, and send an IPI to the client core.
    3466
    3567In order to reduce latency, ALMOS-MKH use IPIs (Inter-Processor Interrupts). The client thread select a core in the server cluster, and send an IPI to the selected server core. An IPI forces the target core to make a scheduling. This reduces the RPC latency when no RPC thread is active for the server core, because the RPC threads are kernel threads that have the highest scheduling priority. When no RPC thread is active for this core, the selected core will activate (or create) a new RPC thread and execute it. When an RPC thread is already active, the IPI forces a scheduling point on the target core, but no new RPC thread is activated (or created).
    3668
    37 Similarly, when the RPC server thread completes, it uses a remote access to unblock the client thread, and send and IPI to the client core to force a scheduling on the client core.
     69== 5) Parallel RPC scenario ==
    3870
    39 == 4) Parallel RPC requests handling ==
     71All RPC services defined by ALMOS-MKH can be used in ''simple'' or ''parallel'' mode.
     72Only the behavior of the client has to be modified for a parallel RPC : To send parallel RPC requests to several servers, the client thread doe not block until the last request has been registered in the last server FIFO.
     73Therefore, to request a RPC service XYZ in parallel mode to N servers, the client function does NOT use the ''rpc_xyz_client()'' function, and follows the following the client scenario:
     741. allocate itself an array of RPC descriptors rpc[N] in client cluster (one per target server),
     751. allocate itself a shared responses counter in client cluster (can be in client stack),
     761. initialize itself the N RPC descriptors, as well as the responses counter (N expected responses),
     771. for all servers, register an extended pointer on rpc[i] in the server[i] FIFO,
     781. for all servers, send an IPI to the selected core in the server[i] cluster,
     791. blocks and deschedule, waiting to be re-activated by the last server thread when it completed the requested service.
    4080
    41 As explained above, a private pool of RPC threads is associated to each RPC_FIFO[i,k] (i.e. to each core). These RPC server threads are dynamically created when required.
     81The tasks 4 and 5 can be done, for each target server, by the generic ''pc_send()'' function.
    4282
    43 At any time, only one RPC thread has the FIFO ownership and can consume RPC requests from the FIFO. Nevertheless, ALMOS-MKH supports supports several RPC server threads per RPC_FIFO because a given RPC thread T handling a given request can block, waiting for a shared resource, such as a peripheral. In that case, the blocked RPC thread T releases the FIFO ownership before blocking and descheduling. This RPC thread T will complete the current RPC request when the blocking condition is solved, and the thread T is rescheduled. If the RPC FIFO is not empty, another RPC thread T' will be scheduled to handle the pending RPC requests. If all existing RPC threads are blocked, a new RPC thread is dynamically created.
     83== 6) Pool of RPC servers ==
     84
     85In order to avoid deadlocks, for each core, ALMOS-MKH defines a private pool of RPC threads  associated to one single RPC_FIFO[i,k].
     86If a given RPC thread extracted request[i] from the FIFO, but is blocked, waiting for a shared resource, the next request[i+i] in the FIFO can be extracted and handled by another RPC thread. In that case, the blocked RPC thread T releases the FIFO ownership before blocking and descheduling. This RPC thread T will complete the current RPC request when the blocking condition is solved, and the thread T is rescheduled. If the RPC FIFO is not empty, another RPC thread T' will be scheduled to handle the pending RPC requests. If all existing RPC threads are blocked, a new RPC thread is dynamically created.
     87
     88At any time, only one RPC thread has the FIFO ownership and can consume RPC requests from the FIFO.
    4489
    4590Therefore, it can exist for each RPC_FIFO[i,k] a variable number M of active RPC threads: the running one is the FIFO owner, and the (M-1) others are blocked on a wait condition.
    46 This number M can temporarily exceed the CONFIG_RPC_THREAD_MAX value, but the but the exceeding server threads are destroyed when the temporary overload is solved.
     91This number M can temporarily exceed the CONFIG_RPC_THREAD_MAX value, but the exceeding server threads are destroyed when the temporary overload is solved.
    4792
    48 == 5) RPC request format ==
    49 
    50 ALMOS-MKH implement two types of RPCs :
    51  * '''simple RPC''' : the client thread send the RPC request to one single server waiting one single response.
    52  * '''multicast RPC''' : the client thread send the same RPC request to several servers, expecting several responses.
    53 
    54 Each entry in the RPC_FIFO contains a remote pointer (xptr_t) on a RPC descriptor (''rpc_desc_t''), that is stored on the client side (in the client thread stack). This RPC descriptor has a fixed format, and contains the following informations:
    55  * The '''index''' field defines the required service type.
    56  * The '''args''' field is an array of 10 uint64_t, containing the service arguments (both input & output).
    57  * The '''thread''' field defines the client thread (used by the server thread to unblock the client thread).
    58  * The '''lid''' field defines the client core local index (used by the server thread to send the completion IPI).
    59  * The '''response''' field defines the number of expected responses.
    60 
    61 The semantic of the '''args''' array depends on the RPC '''index''' field.
    62 
    63 This format supports both simple and multicast RPCs: The client thread initializes the '''response''' field with the number of expected responses, and each server thread atomically decrement this counter when the RPC request has been satisfied.
    64 
    65 == 6) How to define a new RPC ==
     93== 7) How to define a new RPC ==
    6694
    6795To introduce a new RPC service ''rpc_new_service'' in ALMOS-MKH, you need to modify the ''rpc.c'' and ''rpc.h'' files: