Index: trunk/kernel/kern/ksocket.h
===================================================================
--- trunk/kernel/kern/ksocket.h	(revision 669)
+++ trunk/kernel/kern/ksocket.h	(revision 683)
@@ -1,4 +1,4 @@
 /*
- * ksocket.h - kernel socket descriptor and API definition.
+ * ksocket.h - kernel socket definition.
  * 
  * Authors  Alain Greiner    (2016,2017,2018,2019,2020)
@@ -40,6 +40,6 @@
  * existing sockets is split in as many subsets as the number of NIC channels, in order
  * to parallelize the transfers. The distribution key defining the channel index
- * is computed from the (remote_addr/remote_port) couple: by the NIC hardware for the
- * RX packets; by the software for the TX packets, using a dedicated NIC driver function.
+ * is computed from the (remote_addr/remote_port) couple (by the NIC hardware for the
+ * RX packets; by the software for the TX packets) using a dedicated NIC driver function.
  * All sockets that have the same key share the same channel, and each socket is
  * therefore linked to two chdevs : NIC_TX[key] & NIC_RX[key].
@@ -52,15 +52,15 @@
  *   to the associated TX server (mainly used to handle the TCP ACKs).
  * - the kernel "crq" buffer allows to store concurrent remote client connect requests
- *   to a local server socket. It is allocated in socket. 
+ *   to a local server socket. 
  *
  * The synchronisation mechanism between the client threads and the server threads
  * is different for the TX and RX directions:
  *
- * 1) TX stream
+ * 1) TX direction (sent packets)
  *
  * - The internal API between the TX client thread and the NIC_TX server thread defines 
  *   four command types, stored in the "tx_cmd" variable of the socket descriptor:
- *   . SOCKET_TX_CONNECT : TCP client request to start the 3 steps connection handshake.
- *   . SOCKET_TX_ACCEPT  : TCP server request to accept one pending connection request. 
+ *   . SOCKET_TX_CONNECT : request to start the connection handshake (TCP client only).
+ *   . SOCKET_TX_ACCEPT  : request to accept one connection request (TCP server only). 
  *   . SOCKET_TX_SEND    : local (UDP/TCP) request to send data to a remote (UDP/TCP).
  *   . SOCKET_TX_CLOSE   : local TCP socket request remote TCP socket to close connection.
@@ -69,11 +69,11 @@
  *   reset the "tx_error" field, and registers itself in the "tx_client" field.
  *   Then, it unblocks the TX server thread from the BLOCKED_CLIENT condition, blocks itself 
- *   on the BLOCKED_IO condition, and deschedules. For a SEND, the "tx_buf" kernel buffer 
- *   is dynamicaly allocated by the client thread, that copies the payload from the user 
- *   buffer to this kernel buffer, that is used as retransmission buffer, when required.
+ *   on the BLOCKED_IO condition, and deschedules. For a SEND, the client thread copies
+ *   the payload contained in the "u_buf" user buffer to the socket "tx_buf" kernel buffer 
+ *   that is used as retransmission buffer, when required.
  * - A command is valid for the TX server when the socket descriptor "tx_valid" is true.
- *   For a SEND command, the "tx_valid" is reset by the NIC_TX server when the last byte has 
- *   been sent, but the TX client thread is unblocked by the NIC_RX server thread only when
- *   the last byte has been acknowledged, or to report an error.
+ *   For a SEND command, the "tx_valid" is reset by the NIC_TX server thread when the last
+ *   byte has been sent, but the TX client thread is unblocked by the NIC_RX server thread
+ *   only when the last byte has been acknowledged, or to report an error.
  *   For the CONNECT, ACCEPT and CLOSE commands, the "tx_valid" is reset by the NIC_TX server
  *   when the first segment of the handshake has been sent, but the TX client thread is 
@@ -88,9 +88,9 @@
  *   When "tx_valid" or "r2t_valid" are true, the TX server thread build and send an UDP
  *   packet or TCP segment. A single SEND command can require a large number of TCP
- *   segments to move a big data buffer. 
+ *   segments to move a big data buffer, before unblocking the client thread. 
  *   This TX server thread blocks and deschedules on the BLOCKED_ISR condition when there 
  *   the NIC_RX queue is full . It is unblocked by the hardware NIC_TX_ISR. 
- * - In order to detect and report error for multiple simultaneous TX accesses to the same
- *   socket, the client thread makes a double check before posting a new TX command :
+ * - As multiple simultaneous TX accesses to the same socket are forbiden, the client
+ *   thread makes a double check before posting a new TX command :
  *   the "tx_valid" field must be false, and the "tx_client" field must be XPTR_NULL.
  *   The "tx_valid" field is reset by the TX server thread, and the "tx_client"
@@ -136,15 +136,16 @@
  * 3) R2T queue
  *
- * To implement the TCP "3 steps handshake" protocol for connection or to send RST,
- * the RX server thread can directly request the associated TX server thread to send 
- * control packets in  the TX stream, using a dedicate R2T (RX to TX) FIFO stored in 
- * the socket descriptor. Each R2T request occupy one byte in this R2T queue.
+ * The RX server thread can directly request the associated TX server thread to send 
+ * control packets in  the TX stream, using a dedicate R2T (RX to TX) queue embedded in 
+ * the socket descriptor, and implemented as a remote_buf_t FIFO.
+ * It is used for TCP acknowledge and for the TCP three-steps handshake.
+ * Each R2T request occupy exactly one single byte defining the TCP flags to be set.
  *
  * 4) CRQ queue
  *
  * The remote CONNECT requests received by a TCP socket (SYN segments) are stored in a
- * dedicated CRQ FIFO stored in the local socket descriptor. These requests are consumed
- * by the local client thread executing an ACCEPT.
- * Each CRQ request occupy sizeof(connect_request_t) bytes in this CRQ queue.
+ * dedicated CRQ queue, and consumed  by the local client thread executing an ACCEPT.
+ * This CRQ queue is embedded in the local socket descriptor,  and implemented as a
+ * remote_buf_t FIFO. Each request occupy sizeof(connect_request_t) bytes in the queue.
  * The connect_request_t structure containing the request arguments is defined below.
  *
@@ -171,6 +172,4 @@
  * This enum defines the set of command status that can be returned by the NIC_RX and
  * NIC_TX server threads to the TX & RX client threads.
- * The success must be signaled by the null value / the various failure cases are
- * signaled by a non-null value.
  ****************************************************************************************/
 typedef enum socket_cmd_sts_e
@@ -217,7 +216,7 @@
 tcp_socket_state_t;
 
-/***************************************************************************************** 
+/**************************************************************************************** 
  * This structure defines one connection request, registered in the CRQ queue.
- ****************************************************************************************/
+ ***************************************************************************************/
 typedef struct connect_request_s
 {
@@ -229,7 +228,7 @@
 connect_request_t;
 
-/***************************************************************************************** 
+/**************************************************************************************** 
  * This structure defines the socket descriptor.
- ****************************************************************************************/
+ ***************************************************************************************/
 typedef struct socket_s
 {
@@ -253,6 +252,6 @@
     uint8_t         *  tx_buf;       /*! pointer on TX data buffer in kernel space     */
     uint32_t           tx_len;       /*! number of data bytes for a SEND command       */
-    uint32_t           tx_todo;      /*! number of bytes not yet sent                  */
-    xlist_entry_t      tx_temp;      /*! temporary list of sockets (root in TX chdev)  */
+    uint32_t           tx_todo;      /*! number of bytes not yet sent in tx_buf        */
+    uint32_t           tx_ack;       /*! number of bytes acknowledged in tx_buf        */
 
     xlist_entry_t      rx_list;      /*! all sockets attached to same NIC_RX channel   */
@@ -271,4 +270,5 @@
     uint32_t           tx_wnd;       /*! number of acceptable bytes in TX_data stream  */
     uint32_t           tx_una;       /*! first unack byte in TX_data stream            */
+
     uint32_t           rx_nxt;       /*! next expected byte in RX_data stream          */
     uint32_t           rx_wnd;       /*! number of acceptable bytes in RX_data stream  */
@@ -319,9 +319,12 @@
 
 /****************************************************************************************
- * This function is called by the dev_nic_rx_handle_tcp() function, executed by the
- * NIC_RX[channel] server thread, to register a R2T request defined by the <flags>
+ * This blocking function is called by the dev_nic_rx_handle_tcp() function, executed by
+ * the NIC_RX[channel] server thread, to register a R2T request defined by the <flags>
  * argument in the socket R2T queue, specified by the <queue_xp> argument.
  * This function unblocks the NIC_TX[channel] server thread, identified by the <channel>
  * argumentfrom the THREAD_BLOCKED_CLIENT condition.
+ *
+ * WARNING : It contains a waiting loop and return only when an empty slot has been 
+ * found in the R2T queue.
  ****************************************************************************************
  * @ queue_xp   : [in] extended pointer on the R2T qeue descriptor.
@@ -330,6 +333,18 @@
  ***************************************************************************************/
 void socket_put_r2t_request( xptr_t    queue_xp,
-                             uint32_t  flags,
+                             uint8_t   flags,
                              uint32_t  channel );
+
+/****************************************************************************************
+ * This function is called by the nic_tx_server thread to extract an R2T request
+ * (one byte) from a R2T queue, specified by the <queue_xp> argument, to the buffer
+ * defined by the <flags> argument.
+ *****************************************************************************************
+ * @ queue_xp      : [in]  extended pointer on the CRQ queue descriptor.
+ * @ flags         : [out] buffer for TCP flags to be set.
+ * @ return 0 if success / return -1 if queue empty.
+ ***************************************************************************************/
+error_t socket_get_r2t_request (xptr_t    queue_xp,
+                                uint8_t * flags );
   
 /****************************************************************************************
@@ -339,5 +354,5 @@
  * by the <queue_xp> argument.
  ****************************************************************************************
- * @ queue_xp      : [in] extended pointer on the CRQ qeue descriptor.
+ * @ queue_xp      : [in] extended pointer on the CRQ queue descriptor.
  * @ remote_addr   : [in] remote socket IP address.
  * @ remote_port   : [in] remote socket port.
@@ -374,8 +389,10 @@
  ****************************************************************************************
  * @ socket_xp     : [in] extended pointer on socket descriptor.
- $ @ string        : [in] name of calling function.
+ * @ func_str      : [in] name of calling function.
+ * @ string        : [in] string defining the calling context (can be NULL)
  ***************************************************************************************/
 void socket_display( xptr_t         socket_xp,
-                     const char   * func_str );
+                     const char   * func_str,
+                     const char   * string );
 
 
@@ -464,5 +481,5 @@
  * This blocking function contains two blocking conditions because it requests services
  * to both the NIC_RX server thread, and he NIC_TX server thread.
- * It can be split in five steps:
+ * It is structured in five steps:
  * 1) It makes several checkings on the listening socket domain, type, and state.
  * 2) If the socket CRQ queue is empty, the function makes an SOCKET_RX_ACCEPT command
@@ -529,4 +546,56 @@
  * arguments, to a connected (TCP or UDP) socket, identified by the <fdid> argument.
  * The work is actually done by the NIC_TX server thread, and the synchronisation
+ * between the client and the server threads uses the "tx_valid" set/reset flip-flop:
+ * The client thread registers itself in the socket descriptor, registers in the queue 
+ * rooted in the NIC_TX[index] chdev, set "tx_valid", unblocks the server thread, and 
+ * finally blocks on THREAD_BLOCKED_IO, and deschedules. 
+ * When the TX server thread completes the command (all data has been sent for an UDP
+ * socket, or acknowledged for a TCP socket), the server thread reset "rx_valid" and
+ * unblocks the client thread.
+ * This function can be called by a thread running in any cluster.
+ * WARNING : This implementation does not support several concurent SEND commands
+ * on the same socket, as only one TX thread can register in a given socket.
+ ****************************************************************************************
+ * @ fdid      : [in] file descriptor index identifying the socket.
+ * @ u_buf     : [in] pointer on buffer containing packet in user space.
+ * @ length    : [in] packet size in bytes.
+ * @ return number of sent bytes if success / return -1 if failure.
+ ***************************************************************************************/
+int socket_send( uint32_t    fdid,
+                 uint8_t   * u_buf,
+                 uint32_t    length );
+
+/****************************************************************************************
+ * This blocking function implements the recv() syscall.
+ * It is used to receive data that has been stored by the NIC_RX server thread in the
+ * rx_buf of a connected socket, identified by the <fdid> argument.  
+ * The synchronisation between the client and the server threads uses the "rx_valid"
+ * set/reset flip-flop: If "rx_valid" is set, the client simply moves the available
+ * data from the "rx_buf" to the user buffer identified by the <u_buf> and <length>
+ * arguments, and reset the "rx_valid" flip_flop. If "rx_valid" is not set, the client
+ * thread register itself in the socket descriptor, registers in the clients queue rooted
+ * in the NIC_RX[index] chdev, and finally blocks on THREAD_BLOCKED_IO, and deschedules. 
+ * The client thread is re-activated by the RX server, that set the "rx_valid" flip-flop 
+ * as soon as data is available in the "rx_buf". The number of bytes actually transfered
+ * can be less than the user buffer size.
+ * This  function can be called by a thread running in any cluster.
+ * WARNING : This implementation does not support several concurent RECV 
+ * commands on the same socket, as only one RX thread can register in a given socket.
+ ****************************************************************************************
+ * @ fdid        : [in] file descriptor index identifying the local socket.
+ * @ u_buf       : [in] pointer on buffer in user space.
+ * @ length      : [in] buffer size in bytes.
+ * @ return number of received bytes if success / return -1 if failure.
+ ***************************************************************************************/
+int socket_recv( uint32_t    fdid,
+                 uint8_t   * u_buf,
+                 uint32_t    length );
+
+/****************************************************************************************
+ * This blocking function implements the sendto() syscall.
+ * It is used to send data stored in the user buffer, identified the <u_buf> and <length>
+ * to a remote process identified by the <remote_ip> and <remote_port> arguments,
+ * through a local, unconnected (UDP) socket, identified by the <fdid> argument.
+ * The work is actually done by the NIC_TX server thread, and the synchronisation
  * between the client and the server threads uses the "rx_valid" set/reset flip-flop:
  * The client thread registers itself in the socket descriptor, registers in the queue 
@@ -539,18 +608,24 @@
  * WARNING : This implementation does not support several concurent SEND/SENDTO commands
  * on the same socket, as only one TX thread can register in a given socket.
- ****************************************************************************************
- * @ fdid      : [in] file descriptor index identifying the socket.
- * @ u_buf     : [in] pointer on buffer containing packet in user space.
- * @ length    : [in] packet size in bytes.
+ * TODO : this function is not implemented yet.
+ ****************************************************************************************
+ * @ fdid        : [in] file descriptor index identifying the local socket.
+ * @ u_buf       : [in] pointer on buffer containing packet in user space.
+ * @ length      : [in] packet size in bytes.
+ * @ remote_ip   : [in] remote socket IP address.
+ * @ remote_port : [in] remote socket port address.
  * @ return number of sent bytes if success / return -1 if failure.
  ***************************************************************************************/
-int socket_send( uint32_t    fdid,
-                 uint8_t   * u_buf,
-                 uint32_t    length );
-
-/****************************************************************************************
- * This blocking function implements the recv() syscall.
+int socket_sendto( uint32_t   fdid,
+                   uint8_t  * u_buf,
+                   uint32_t   length,
+                   uint32_t   remote_ip,
+                   uint16_t   remote_port );
+
+/****************************************************************************************
+ * This blocking function implements the recvfrom() syscall.
  * It is used to receive data that has been stored by the NIC_RX server thread in the
- * rx_buf of a connected (TCP or UDP) socket, identified by the <fdid> argument.
+ * rx_buf of a non connected socket, identified by the <fdid> argument, from a
+ * remote process identified by the <remote_ip> and <remote_port> arguments.
  * The synchronisation between the client and the server threads uses the "rx_valid"
  * set/reset flip-flop: If "rx_valid" is set, the client simply moves the available
@@ -565,13 +640,18 @@
  * WARNING : This implementation does not support several concurent RECV/RECVFROM 
  * commands on the same socket, as only one RX thread can register in a given socket.
- ****************************************************************************************
- * @ fdid      : [in] file descriptor index identifying the socket.
- * @ u_buf     : [in] pointer on buffer in user space.
- * @ length    : [in] buffer size in bytes.
+ * TODO : this function is not implemented yet.
+ ****************************************************************************************
+ * @ fdid        : [in] file descriptor index identifying the local socket.
+ * @ u_buf       : [in] pointer on buffer in user space.
+ * @ length      : [in] buffer size in bytes.
+ * @ remote_ip   : [in] remote socket IP address.
+ * @ remote_port : [in] remote socket port address.
  * @ return number of received bytes if success / return -1 if failure.
  ***************************************************************************************/
-int socket_recv( uint32_t    fdid,
-                 uint8_t   * u_buf,
-                 uint32_t    length );
+int socket_recvfrom( uint32_t    fdid,
+                     uint8_t   * u_buf,
+                     uint32_t    length,
+                     uint32_t    remote_ip,
+                     uint16_t    remote_port );
 
 /****************************************************************************************
