| 100 | | The KCODE vseg contains the kernel code defined in the ''kernel.elf'' file. Almos-mkh creates one KCODE vseg in each cluster K, to avoid contention. It is a ''private'' vseg, that is accessed only by the threads running in cluster K. It can be an user thread executing a syscall, or it can be a specialized kernel thread (such as an IDLE thread, a DEV thread, or a RPC thread). In each cluster K, the KCODE vseg is registered in the VMM(0,K) associated to the kernel ''process_zero'', that contains all kernel threads, and in the VMM(P,K) of each user process P that has at least one thread running in cluster K. This vseg uses only big pages, that are mapped by the kernel_init function (no on demand paging for this vseg). |
| | 102 | A '''KCODE''' vseg contains the kernel code defined in the ''kernel.elf'' file. To avoid contention and improve locality, almos-mkh replicates this code in all clusters. This code has already been copied in all clusters by the bootloader. In each cluster K, and for all process P in cluster K (including the kernel process_zero), almos-mkh registers the KCODE vseg in all VSL(P,K), and map it to the local copy in all the GPT(P,K). This vseg uses only big pages, and there is no on-demand paging for this type of vseg. With this local mapping all access to the virtual instruction addresses will be simply translated by the MMU to the local physical address. |
| | 103 | |
| | 104 | '''WARNING''' : there is only one vseg defined in the ''kernel.elf'' file, but there is as many KCODE vsegs as the number of clusters. All these vsegs have the same virtual base address and the same size. but the physical adresses (defined in the GPTs) depend on the cluster, because we want to access the local copy. |
| | 105 | This is not a problem because A KCODE vseg is a ''private'' vseg, that is accessed only by local threads. |
| 104 | | This '''public''' vseg contains the global data, statically allocated at compilation time. This vseg is also replicated in all clusters. The values initially contained in these KDATA vsegs are identical, as they are defined in the ''kernel.elf'' file. But they are not read-only, and can evolve differently in different clusters. As the KDATA vsegs are replicated in all clusters, most accesses to a KDATA segment are expected to be done by local threads. These local accesses can use the normal pointers in virtual kernel space. |
| 105 | | |
| 106 | | But there is only one vseg defined in the ''kernel.elf'' file, and there is as many KDATA segments as the number of clusters. Even if most accesses are local, a thread running in cluster K must be able to access a global variable stored in another cluster X, or to send a request to another kernel instance in cluster X, or to scan a globally distributed structure, such as the DQDT or the VFS. To support this cooperation between kernel instances, almos-mkh defines the ''remote_load( cxy , ptr )'' and ''remote_store( cxy , ptr ) functions, where ''ptr'' is a normal pointer in kernel virtual space on a variable stored in the KDATA vseg, and ''cxy'' is the remote cluster identifier. Notice that a given global variable is now identified by and extended pointer ''XPTR( cry , ptr )''. With these remote access primitives, any kernel instance in cluster K can access any global variable in any cluster. |
| | 109 | A '''KDATA''' vseg contains the kernel global data, statically allocated at compilation time, and defined in the ''kernel.elf'' file. To avoid contention and improve locality, almos-mkh replicates the KDATA vseg in all clusters. The corresponding data have already been copied in all clusters. As a physical copy of the KDATA vseg is available in any cluster K, almos-mkh can register this vseg in all VSL(P,K), and map it to this local copy in all GPT(P,K). With this local mapping we expect that most accesses to any KDATA segment will be done by a local thread. |
| | 110 | |
| | 111 | |
| | 112 | '''WARNING''' : there is only one vseg defined in the ''kernel.elf'' file, and there is as many KDATA vsegs as the number of clusters. All these vsegs have the same virtual base address and the same size, but the physical addresses (defined in the GPTs), depends on the cluster, because we generally want to access the local copy. |
| | 113 | This is a problem, because there is two big differences between the KCODE and the KDATA vsegs : |
| | 114 | 1. The values contained in the N KDATA vsegs are initially identical, as they are all defined by the same ''kernel.elf'' file. But they are not read-only, and can evolve differently in different clusters. |
| | 115 | 1. The N KDATA vsegs are ''public'', and define an addressable storage space N times larger than one single KDATA vseg. Even if most accesses are local, a thread running in cluster K must be able to access a global variable stored in another cluster X, or to send a request to another kernel instance in cluster X, or to scan a globally distributed structure, such as the DQDT or the VFS. |
| | 116 | To support this inter-cluster kernel-to-kernel communication, almos-mkh defines the ''remote_load( cxy , ptr )'' and ''remote_store( cxy , ptr ) functions, where ''ptr'' is a normal pointer (in kernel virtual space) on a variable stored in the KDATA vseg, and ''cxy'' is the remote cluster identifier. Notice that a given global variable is now identified by and extended pointer ''XPTR( cry , ptr )''. With these remote access primitives, any kernel instance in cluster K can access any global variable in any cluster. Notice that local accesses can use the normal pointers in virtual kernel space, as the virtual adresses will be simply translated by the MMU to the local physical address. |
| | 117 | |
| | 118 | In other words, almost-mkh clearly distinguish the ''local accesses'', that can use standard pointers, from the ''remote access'' that '''must''' extended pointers. |
| | 119 | This can be seen as a bad constraint, but it can also help to imp to improve the locality, and to identify (and remove) the contention. |
| | 120 | |
| | 121 | The remote_access primitives API is defined in the [https://www-soc.lip6.fr/trac/almos-mkh/browser/trunk/hal/generic/hal_remote.h almos_mkh/hal/generic/hal_remote.h] file. |
| 110 | | Beside the statically allocated global variables, a large number of kernel structures, such as the user ''process'' descriptors, the ''thread'' descriptors, the ''vseg'' descriptors, the ''file'' descriptors, etc. are dynamically descriptors |
| | 125 | The '''KHEAP''' vsegs are used to store dynamically allocated of kernel structures, such as the user ''process'' descriptors, the ''thread'' descriptors, the ''vseg'' descriptors, the ''file'' descriptors, etc. These structure can be requested by any thread running in any cluster, and are defined as global variables, that can be accessed by any thread. To avoid contention and improve locality, almos-mkh build a ''physically distributed kernel heap''. In each cluster K, almos-mkh can register this HEAP vseg in all VSL(P,K), and map it in all the GPT(P,K). with is local mapping, a kernel structure requested by a thread running in any cluster will always be allocated in the local physical memory. |
| | 126 | |
| | 127 | WARNING : To unify the access to remote data (i.e. data stored in a remote cluster), almos-mkh use the same policy for KHEAP and KDATA vsegs: all KHEAP segments have the same virtual base address. The local accesses to the locally allocated kernel structures can use normal pointers that will be translated by the MMU to local physical adresses. The remote access to remotely allocated kernel structures must use the ''remote_load()'' and ''remote_store()'' functions and handle extended pointers. |
