sylware / nyanlinux (public) (License: AFFERO GPLv3) (since 2019-09-09) (hash sha1)
scripts for a lean, from scratch, amd hardware, linux distro
Clone URLs: https://rocketgit.com/user/sylware/nyanlinux ssh://rocketgit@ssh.rocketgit.com/user/sylware/nyanlinux git://git.rocketgit.com/user/sylware/nyanlinux
master
List of commits:
Subject Hash Author Date (UTC)
clear a bit the EDLF64 draft. 4dd34b7fa5ae978dce295cc4a836e367d25f007d Sylvain BERTRAND 2023-07-25 14:06:13
EDLF64 re-entrancy/fix. 404ecd7028b1031ccbb88300c90674618819c5bd Sylvain BERTRAND 2023-07-18 20:30:58
Gfx stack update 8025b4f7dac102ccc5de280df2105e73a31f1910 Sylvain BERTRAND 2023-07-15 15:04:11
EDLF64: clean munmap of the executable itself 1ed67b684b7c154cbb7eea742cb8eef0c7790dbc Sylvain BERTRAND 2023-07-09 17:37:27
last ffpmeg before breaking nyanmp 36cded2ce51a6df8edfcffc71094695fa8ad871c Sylvain BERTRAND 2023-07-09 15:05:32
Gfx stack update b2050ef1941139d255c61c2c82e24e6de364c5b4 Sylvain BERTRAND 2023-07-08 16:58:03
improve EDLF64 draft 48938c705c4f10e898677ec6e321dfad8d892f3f Sylvain BERTRAND 2023-07-07 18:36:15
elf toolbox and less toxic format draft a2ab4235350f04975de513911b7a340e2eb50ef3 Sylvain BERTRAND 2023-07-07 17:26:40
update of some image pagkages 7d8a5e8a5e20ab58e585a5f9d4270054285dba8c Sylvain BERTRAND 2023-07-05 23:14:01
gcc 12.2 won't build c++ threading all the time. d8560808bd62a198bc446153602c658141238c01 Sylvain BERTRAND 2023-07-02 16:56:10
Gfx stack update 50e5991a466541ebb88456918f6cc0bad7e1bfb8 Sylvain BERTRAND 2023-07-01 13:19:49
refreshing various components 22c025e6a7019283775b45742a8095a26d1d9b51 Sylvain BERTRAND 2023-07-01 00:16:51
Gfx stack update d91a56e63cb4e61ce950166ee90977f9cb0f7c57 Sylvain BERTRAND 2023-06-24 17:28:11
Gfx stack update 5053fdee70c5a2656a39391f9ec0924b680cb63f Sylvain BERTRAND 2023-06-17 13:08:44
Gfx stack update a8f4d317da64e4728a07332d7b312b0163a9a117 Sylvain BERTRAND 2023-06-10 13:00:40
Gfx stack update 5a50b752ee1174f40b4ac04b0c4ea40def0f003e Sylvain BERTRAND 2023-06-03 13:07:31
Gfx stack update 8d0226a320e934c81b0ffbca7203ab25cfc801ae Sylvain BERTRAND 2023-05-27 13:19:40
Gfx stack update 1b4924dbae0b406ec5f8e9fdc85d0c9c5899d505 Sylvain BERTRAND 2023-05-20 19:39:21
Gfx stack update 98ed4e3167334c8e605f642d8cadb9249078bf84 Sylvain BERTRAND 2023-05-13 14:54:30
Gfx stack update 9adee44d5eb4278e6beb28ec35cae1d7a61bfa49 Sylvain BERTRAND 2023-05-06 17:58:22
Commit 4dd34b7fa5ae978dce295cc4a836e367d25f007d - clear a bit the EDLF64 draft.
Author: Sylvain BERTRAND
Author date (UTC): 2023-07-25 14:06
Committer name: Sylvain BERTRAND
Committer date (UTC): 2023-07-25 14:06
Parent(s): 404ecd7028b1031ccbb88300c90674618819c5bd
Signer:
Signing key:
Signing status: N
Tree: 6938817c3b1b3a06cd4a46d102fbafa3fdc8f6f7
File Lines added Lines deleted
files/EDLF64.draft 31 42
File files/EDLF64.draft changed (mode: 100644) (index 958379d..a13427f)
1 1 EDLF64 is the JSON/wayland of the executables and dynamic libraries for 64bits platforms with a EDLF64 is the JSON/wayland of the executables and dynamic libraries for 64bits platforms with a
2 thread lock mechanism.
2 reentrancy lock mechanism.
3 3
4 EDLF64=Executable and Dynamic Library Format for 64bits platforms (with a thread lock mechanism).
4 EDLF64=_E_xecutable and _D_ynamic _L_ibrary _F_ormat for _64_bits platforms with a reentrancy lock
5 mechanism.
5 6
6 7 The excrutiating simplicity of the format is intended while doing a good enough job. The excrutiating simplicity of the format is intended while doing a good enough job.
7 8
8 9 To avoid a circular dependency with a high level threading dynamic library, the hardware To avoid a circular dependency with a high level threading dynamic library, the hardware
9 architecture with or without the kernel must provide a pre-inited/ready to use thread lock
10 architecture with or without the kernel must provide a pre-inited/ready to use reentrancy lock
10 11 mechanism. Usually on many cores systems, this is an atomic compare and exchange hardware mechanism. Usually on many cores systems, this is an atomic compare and exchange hardware
11 12 instruction or kernel syscall on memory locations which are already inited with some specific instruction or kernel syscall on memory locations which are already inited with some specific
12 13 content. content.
 
... ... will be only one static instance of loader64 per process inited by the process_e
24 25 The main loader64 instance code is usually dependent only on the hardware architecture and kernel, The main loader64 instance code is usually dependent only on the hardware architecture and kernel,
25 26 to stay independent of all dynamic libraries. to stay independent of all dynamic libraries.
26 27
28 Only one thread can use a loader64 instance at a time. Entry must be guarded by the reentrancy lock
29 mechanism.
30
31 Dynamic libraries should try to presume not they are the only instance in a process: there could be
32 others loaded by other loader64 instances, or the "same" dynamic library but from different files.
33 Dodging related conflicts could be expensive and should be clearly documented in a dynamic library
34 documentation if it is supported or not.
35
27 36 uint64_t loader64_open( /*INPUT*/ void *pathname, /* we presume the pathname is self-sizing */ uint64_t loader64_open( /*INPUT*/ void *pathname, /* we presume the pathname is self-sizing */
28 37 /*OUTPUT*/ uint64_t *handle, void **start); /*OUTPUT*/ uint64_t *handle, void **start);
29 38
30 Must be re-entrant. Usually, the main loader instance of a process which will be used to
31 load a high level threading dynamic library does use the same thread lock mechanism than the
32 resolve function, that to break the circular dependency.
33
34 39 Return values: Return values:
35 40 0 if ok, and a loader handle and the pointer on the start of the loaded dynamic 0 if ok, and a loader handle and the pointer on the start of the loaded dynamic
36 41 library, namely the first byte of the loaded dynamic library ELDF64 header. library, namely the first byte of the loaded dynamic library ELDF64 header.
37 42 11 (-EAGAIN), if the loader is currently busy. 11 (-EAGAIN), if the loader is currently busy.
38 43 other an error did happen. other an error did happen.
39 44
40 Init functions, if any, should be resolved and called right after open. Usually, their C
41 prototypes do include pathname, see below for a recommended init function prototype.
42
43 45 If pathname targets an already loaded file, the same handle/start will be returned by the If pathname targets an already loaded file, the same handle/start will be returned by the
44 46 loader. On linux, the triplet (dev_major/dev_minor/inode) should defines file system loader. On linux, the triplet (dev_major/dev_minor/inode) should defines file system
45 47 unicity. unicity.
 
... ... uint64_t loader64_open( /*INPUT*/ void *pathname, /* we presume the pathname is
53 55
54 56 uint64_t loader64_close(/*INPUT*/ uint64_t handle); uint64_t loader64_close(/*INPUT*/ uint64_t handle);
55 57
56 Must be re-entrant. Usually, the main loader instance of a process which will be used to
57 load a high level threading dynamic library, does use the same thread lock mechanism than
58 the resolve function, that to break the circular dependency.
59
60 58 Return values: Return values:
61 59 0 if ok, non-zero The handle becomes invalid. 0 if ok, non-zero The handle becomes invalid.
62 60 11 (-EAGAIN), if the loader is currently busy. The handle stays valid. 11 (-EAGAIN), if the loader is currently busy. The handle stays valid.
63 61 other if something wrong did happen while closing the edlf64 file. The handle becomes other if something wrong did happen while closing the edlf64 file. The handle becomes
64 62 invalid. invalid.
65
66 Fini functions, if any, should be called (may be resolved if not provided by init functions)
67 right before the close call.
68
69 Init/fini functions have the responsibility to keep the dynamic library state consistent (for
70 instance using reference counting like a loader instance).
71 Dynamic libraries should try to presume not they are the only instance in a process. The process
72 could have other instances via other loader instances. Avoiding those races could be expensive,
73 should be clearly documented in a dynamic library documentation if it is supported or not.
74 63 ==================================================================================================== ====================================================================================================
75 64 EDLF64 is about loading only one RWX memory segment. EDLF64 is about loading only one RWX memory segment.
76 65 ==================================================================================================== ====================================================================================================
77 66 A EDLF64 file may honor the following environment variable in order to lookup for dynamic A EDLF64 file may honor the following environment variable in order to lookup for dynamic
78 67 libraries. Of course, only on platforms where it is possible. Such incompatible platforms may libraries. Of course, only on platforms where it is possible. Such incompatible platforms may
79 defines their own "EDLF64_LIBRARY_PATH way".
68 defines their own "EDLF64_LIBRARY_PATH way" (could have a conflicting name separator).
80 69
81 70 EDLF64_LIBRARY_PATH environment variable to lookup for EDLF64 dynamic libraries: byte string EDLF64_LIBRARY_PATH environment variable to lookup for EDLF64 dynamic libraries: byte string
82 71 ending with a 0x00 byte. Each path from EDLF64_LIBRARY_PATH is prepended to a dynamic library name. ending with a 0x00 byte. Each path from EDLF64_LIBRARY_PATH is prepended to a dynamic library name.
 
... ... C prototype of resolve function:
124 113 /*INPUT*/ uint64_t *symbol_id, /*INPUT*/ uint64_t *symbol_id,
125 114 /*OUTPUT*/ void **symbol_virtual_address); /*OUTPUT*/ void **symbol_virtual_address);
126 115
127 Must be re-entrant/thread-safe without the usage of a high level threading dynamic library.
116 Only one thread can use the resolve function at a time. Entry must be guarded by the reentrancy
117 lock mechanism.
128 118
129 symbol_id is a 64bits unique id identifying a symbol (similar to kernel syscalls). Like kernel
119 symbol_id is a 64bits unique value identifying a symbol (similar to kernel syscalls). Like kernel
130 120 syscalls, those symbol ids must be _EXTREMELY_ stable in time. syscalls, those symbol ids must be _EXTREMELY_ stable in time.
131 121
132 122 Return values: Return values:
 
... ... Return values:
135 125 11 (-EAGAIN), if the resolve function cannot be run right now. 11 (-EAGAIN), if the resolve function cannot be run right now.
136 126 other the symbol was not found. other the symbol was not found.
137 127 ==================================================================================================== ====================================================================================================
138 Recommended C prototype of a basic init function:
128 In init or fini functions, be very careful about circular dependencies with other components. It
129 is worth on the long run to provide permanent reentrancy detection while aborting using a very loud
130 maneer.
131
132 Init/fini functions have the responsibility to keep the dynamic library state consistent (for
133 instance using reference counting like a loader instance).
134
135 Example C prototype of a basic init function:
139 136 uint64_t init( uint64_t init(
140 137 /*INPUT*/ void *process_info, uint64_t process_info_bytes_n, void *pathname, /*INPUT*/ void *process_info, uint64_t process_info_bytes_n, void *pathname,
141 138 uint64_t (*loader64_open)(void *pathname, uint64_t *handle, void **start), uint64_t (*loader64_open)(void *pathname, uint64_t *handle, void **start),
142 uint64_t (*loader64_close)(uint64_t handle),
143 /*OUTPUT*/ void (**fini)(void));
139 uint64_t (*loader64_close)(uint64_t handle));
144 140
145 141 The process_info here may be a variant from the one provided by the kernel to process_entry. The process_info here may be a variant from the one provided by the kernel to process_entry.
146 142 Pathname is a pointer on the pathname used to load this edlf64 file. If successful, it should return Pathname is a pointer on the pathname used to load this edlf64 file. If successful, it should return
 
... ... around some ABI kludge.
150 146
151 147 Alternative C prototype of a basic init function: Alternative C prototype of a basic init function:
152 148 uint64_t init( uint64_t init(
153 /*INPUT*/ void *process_info, uint64_t process_info_bytes_n, int fd,
149 /*INPUT*/ void *process_info, uint64_t process_info_bytes_n,
150 int distribution_dir_fd, int fd,
154 151 uint64_t (*loader64_open)(void *pathname, uint64_t *handle, void **start), uint64_t (*loader64_open)(void *pathname, uint64_t *handle, void **start),
155 uint64_t (*loader64_close)(uint64_t handle),
156 /*OUTPUT*/ void (**fini)(void));
152 uint64_t (*loader64_close)(uint64_t handle));
157 153
158 154 Same thing than the previous one, but with the process file descriptor used to load/mmap the edlf64 Same thing than the previous one, but with the process file descriptor used to load/mmap the edlf64
159 file instead of the pathname. You could even add the directory file descriptor, and since that
160 would make more than 6 parameters, better use a transient structure to pass the whole data.
155 file instead of the pathname supplemented with the process directory file descriptor of the
156 distribution directory.
161 157
162 Recommended C prototype of basic fini function:
158 Example C prototype of basic fini function:
163 159 void fini(void) void fini(void)
164 ----------------------------------------------------------------------------------------------------
165 Dead-locks/circular dependencies can happen while coarse init-ing/fini-ting inter-dependent dynamic
166 libraries. Usually, it means you need to break them down, or have fine-grained init/fini functions
167 specific to a caller/subsystem, and that, probably in several steps.
168 Or the other way around: fuse them all in one common dynamic library with a big init and use other
169 dynamic libraries as user level stubs resolving in this very common dynamic library. Of course,
170 those user level stubs could contain some code "only for the user" to call.
171 160 ==================================================================================================== ====================================================================================================
172 161 NOTES: NOTES:
173 162
Hints:
Before first commit, do not forget to setup your git environment:
git config --global user.name "your_name_here"
git config --global user.email "your@email_here"

Clone this repository using HTTP(S):
git clone https://rocketgit.com/user/sylware/nyanlinux

Clone this repository using ssh (do not forget to upload a key first):
git clone ssh://rocketgit@ssh.rocketgit.com/user/sylware/nyanlinux

Clone this repository using git:
git clone git://git.rocketgit.com/user/sylware/nyanlinux

You are allowed to anonymously push to this repository.
This means that your pushed commits will automatically be transformed into a merge request:
... clone the repository ...
... make some changes and some commits ...
git push origin main