sylware / nyanlinux (public) (License: AFFERO GPLv3) (since 2019-09-09) (hash sha1)
scripts for a lean, from scratch, amd hardware, linux distro
Clone URLs: https://rocketgit.com/user/sylware/nyanlinux ssh://rocketgit@ssh.rocketgit.com/user/sylware/nyanlinux git://git.rocketgit.com/user/sylware/nyanlinux
master
List of commits:
Subject Hash Author Date (UTC)
EDLF64 re-entrancy/fix. 404ecd7028b1031ccbb88300c90674618819c5bd Sylvain BERTRAND 2023-07-18 20:30:58
Gfx stack update 8025b4f7dac102ccc5de280df2105e73a31f1910 Sylvain BERTRAND 2023-07-15 15:04:11
EDLF64: clean munmap of the executable itself 1ed67b684b7c154cbb7eea742cb8eef0c7790dbc Sylvain BERTRAND 2023-07-09 17:37:27
last ffpmeg before breaking nyanmp 36cded2ce51a6df8edfcffc71094695fa8ad871c Sylvain BERTRAND 2023-07-09 15:05:32
Gfx stack update b2050ef1941139d255c61c2c82e24e6de364c5b4 Sylvain BERTRAND 2023-07-08 16:58:03
improve EDLF64 draft 48938c705c4f10e898677ec6e321dfad8d892f3f Sylvain BERTRAND 2023-07-07 18:36:15
elf toolbox and less toxic format draft a2ab4235350f04975de513911b7a340e2eb50ef3 Sylvain BERTRAND 2023-07-07 17:26:40
update of some image pagkages 7d8a5e8a5e20ab58e585a5f9d4270054285dba8c Sylvain BERTRAND 2023-07-05 23:14:01
gcc 12.2 won't build c++ threading all the time. d8560808bd62a198bc446153602c658141238c01 Sylvain BERTRAND 2023-07-02 16:56:10
Gfx stack update 50e5991a466541ebb88456918f6cc0bad7e1bfb8 Sylvain BERTRAND 2023-07-01 13:19:49
refreshing various components 22c025e6a7019283775b45742a8095a26d1d9b51 Sylvain BERTRAND 2023-07-01 00:16:51
Gfx stack update d91a56e63cb4e61ce950166ee90977f9cb0f7c57 Sylvain BERTRAND 2023-06-24 17:28:11
Gfx stack update 5053fdee70c5a2656a39391f9ec0924b680cb63f Sylvain BERTRAND 2023-06-17 13:08:44
Gfx stack update a8f4d317da64e4728a07332d7b312b0163a9a117 Sylvain BERTRAND 2023-06-10 13:00:40
Gfx stack update 5a50b752ee1174f40b4ac04b0c4ea40def0f003e Sylvain BERTRAND 2023-06-03 13:07:31
Gfx stack update 8d0226a320e934c81b0ffbca7203ab25cfc801ae Sylvain BERTRAND 2023-05-27 13:19:40
Gfx stack update 1b4924dbae0b406ec5f8e9fdc85d0c9c5899d505 Sylvain BERTRAND 2023-05-20 19:39:21
Gfx stack update 98ed4e3167334c8e605f642d8cadb9249078bf84 Sylvain BERTRAND 2023-05-13 14:54:30
Gfx stack update 9adee44d5eb4278e6beb28ec35cae1d7a61bfa49 Sylvain BERTRAND 2023-05-06 17:58:22
Gfx stack update a1e169843a89e65cbeefe0582bc27f5981f2501e Sylvain BERTRAND 2023-04-30 13:34:07
Commit 404ecd7028b1031ccbb88300c90674618819c5bd - EDLF64 re-entrancy/fix.
Author: Sylvain BERTRAND
Author date (UTC): 2023-07-18 20:30
Committer name: Sylvain BERTRAND
Committer date (UTC): 2023-07-18 20:30
Parent(s): 8025b4f7dac102ccc5de280df2105e73a31f1910
Signer:
Signing key:
Signing status: N
Tree: 87bf623f1bbcd4b83666601eb0b11c0015194b69
File Lines added Lines deleted
files/EDLF64.draft 54 19
File files/EDLF64.draft changed (mode: 100644) (index ba5ec5f..958379d)
1 EDLF64 is the JSON/wayland of the executables and dynamic libraries for 64bits platforms.
1 EDLF64 is the JSON/wayland of the executables and dynamic libraries for 64bits platforms with a
2 thread lock mechanism.
2 3
3 EDLF64=Executable and Dynamic Library Format for 64bits platforms.
4 EDLF64=Executable and Dynamic Library Format for 64bits platforms (with a thread lock mechanism).
4 5
5 6 The excrutiating simplicity of the format is intended while doing a good enough job. The excrutiating simplicity of the format is intended while doing a good enough job.
6 7
7 Endianness is the one from the CPU. Offsets 0->edlf_hdr_bytes_n-1 are obviously invalid.
8 To avoid a circular dependency with a high level threading dynamic library, the hardware
9 architecture with or without the kernel must provide a pre-inited/ready to use thread lock
10 mechanism. Usually on many cores systems, this is an atomic compare and exchange hardware
11 instruction or kernel syscall on memory locations which are already inited with some specific
12 content.
8 13 ==================================================================================================== ====================================================================================================
9 0x00 "EDLF64",0x00[2],version_b,undef_b[7] (version_b will very probably stay 0x00 forever)
14 0x00 "EDLF64",0x00,version_b (version_b will very probably stay 0x00 forever)
10 15 0x08 alignment, power of two (then cannot be 0, namely at least 1), from "EDLF64" 0x08 alignment, power of two (then cannot be 0, namely at least 1), from "EDLF64"
11 16 0x10 mem_bytes_n. 0x10 mem_bytes_n.
12 17 0x18 process_entry_file_offset, 0 if this is a dynamic library (register passing, no stack). 0x18 process_entry_file_offset, 0 if this is a dynamic library (register passing, no stack).
 
... ... Endianness is the one from the CPU. Offsets 0->edlf_hdr_bytes_n-1 are obviously
16 21 ==================================================================================================== ====================================================================================================
17 22 A loader64 instance will keep a registry of what was loaded via reference counting. Usually, there A loader64 instance will keep a registry of what was loaded via reference counting. Usually, there
18 23 will be only one static instance of loader64 per process inited by the process_entry function. will be only one static instance of loader64 per process inited by the process_entry function.
24 The main loader64 instance code is usually dependent only on the hardware architecture and kernel,
25 to stay independent of all dynamic libraries.
19 26
20 27 uint64_t loader64_open( /*INPUT*/ void *pathname, /* we presume the pathname is self-sizing */ uint64_t loader64_open( /*INPUT*/ void *pathname, /* we presume the pathname is self-sizing */
21 28 /*OUTPUT*/ uint64_t *handle, void **start); /*OUTPUT*/ uint64_t *handle, void **start);
22 29
23 Must be reentrant/thread safe.
30 Must be re-entrant. Usually, the main loader instance of a process which will be used to
31 load a high level threading dynamic library does use the same thread lock mechanism than the
32 resolve function, that to break the circular dependency.
24 33
25 Return 0 if ok, non-zero if an error did happen, and a loader handle and the pointer
26 on the start of the loaded dynamic library, namely the first byte of the loaded dynamic
27 library ELDF64 header.
34 Return values:
35 0 if ok, and a loader handle and the pointer on the start of the loaded dynamic
36 library, namely the first byte of the loaded dynamic library ELDF64 header.
37 11 (-EAGAIN), if the loader is currently busy.
38 other an error did happen.
28 39
29 40 Init functions, if any, should be resolved and called right after open. Usually, their C Init functions, if any, should be resolved and called right after open. Usually, their C
30 41 prototypes do include pathname, see below for a recommended init function prototype. prototypes do include pathname, see below for a recommended init function prototype.
 
... ... uint64_t loader64_open( /*INPUT*/ void *pathname, /* we presume the pathname is
42 53
43 54 uint64_t loader64_close(/*INPUT*/ uint64_t handle); uint64_t loader64_close(/*INPUT*/ uint64_t handle);
44 55
45 Must be reentrant/thread safe. Return 0 if ok, non-zero if something wrong did happen while
46 closing the edlf64 file.
56 Must be re-entrant. Usually, the main loader instance of a process which will be used to
57 load a high level threading dynamic library, does use the same thread lock mechanism than
58 the resolve function, that to break the circular dependency.
59
60 Return values:
61 0 if ok, non-zero The handle becomes invalid.
62 11 (-EAGAIN), if the loader is currently busy. The handle stays valid.
63 other if something wrong did happen while closing the edlf64 file. The handle becomes
64 invalid.
47 65
48 66 Fini functions, if any, should be called (may be resolved if not provided by init functions) Fini functions, if any, should be called (may be resolved if not provided by init functions)
49 67 right before the close call. right before the close call.
50 68
51 69 Init/fini functions have the responsibility to keep the dynamic library state consistent (for Init/fini functions have the responsibility to keep the dynamic library state consistent (for
52 70 instance using reference counting like a loader instance). instance using reference counting like a loader instance).
71 Dynamic libraries should try to presume not they are the only instance in a process. The process
72 could have other instances via other loader instances. Avoiding those races could be expensive,
73 should be clearly documented in a dynamic library documentation if it is supported or not.
53 74 ==================================================================================================== ====================================================================================================
54 75 EDLF64 is about loading only one RWX memory segment. EDLF64 is about loading only one RWX memory segment.
55 76 ==================================================================================================== ====================================================================================================
56 A EDLF64 executable may honor the following environment variable in order to lookup for dynamic
57 libraries. Of course, only on platforms where it is possible. Such incompatible platforms should
77 A EDLF64 file may honor the following environment variable in order to lookup for dynamic
78 libraries. Of course, only on platforms where it is possible. Such incompatible platforms may
58 79 defines their own "EDLF64_LIBRARY_PATH way". defines their own "EDLF64_LIBRARY_PATH way".
59 80
60 81 EDLF64_LIBRARY_PATH environment variable to lookup for EDLF64 dynamic libraries: byte string EDLF64_LIBRARY_PATH environment variable to lookup for EDLF64 dynamic libraries: byte string
 
... ... C prototype of resolve function:
103 124 /*INPUT*/ uint64_t *symbol_id, /*INPUT*/ uint64_t *symbol_id,
104 125 /*OUTPUT*/ void **symbol_virtual_address); /*OUTPUT*/ void **symbol_virtual_address);
105 126
127 Must be re-entrant/thread-safe without the usage of a high level threading dynamic library.
128
106 129 symbol_id is a 64bits unique id identifying a symbol (similar to kernel syscalls). Like kernel symbol_id is a 64bits unique id identifying a symbol (similar to kernel syscalls). Like kernel
107 syscalls, those symbol ids must be _EXTREMELY_ stable in time. You could segment the id space with
108 categories (init calls, fini calls, etc, 64bits is huge).
109 Return 0 if ok, with the virtual address of the symbol via the symbol_virtual_address argument.
130 syscalls, those symbol ids must be _EXTREMELY_ stable in time.
131
132 Return values:
133 0 if ok, with the virtual address of the symbol via the symbol_virtual_address argument. No
134 assumption must be made about the location of this virtual address.
135 11 (-EAGAIN), if the resolve function cannot be run right now.
136 other the symbol was not found.
110 137 ==================================================================================================== ====================================================================================================
111 Recommended C prototype of an init function:
138 Recommended C prototype of a basic init function:
112 139 uint64_t init( uint64_t init(
113 140 /*INPUT*/ void *process_info, uint64_t process_info_bytes_n, void *pathname, /*INPUT*/ void *process_info, uint64_t process_info_bytes_n, void *pathname,
114 141 uint64_t (*loader64_open)(void *pathname, uint64_t *handle, void **start), uint64_t (*loader64_open)(void *pathname, uint64_t *handle, void **start),
 
... ... Pathname is a pointer on the pathname used to load this edlf64 file. If successf
121 148 going with more than 6 parameters, just init a transient structure to pass the whole data to work going with more than 6 parameters, just init a transient structure to pass the whole data to work
122 149 around some ABI kludge. around some ABI kludge.
123 150
124 Alternative C prototype of an init function:
151 Alternative C prototype of a basic init function:
125 152 uint64_t init( uint64_t init(
126 153 /*INPUT*/ void *process_info, uint64_t process_info_bytes_n, int fd, /*INPUT*/ void *process_info, uint64_t process_info_bytes_n, int fd,
127 154 uint64_t (*loader64_open)(void *pathname, uint64_t *handle, void **start), uint64_t (*loader64_open)(void *pathname, uint64_t *handle, void **start),
 
... ... Alternative C prototype of an init function:
129 156 /*OUTPUT*/ void (**fini)(void)); /*OUTPUT*/ void (**fini)(void));
130 157
131 158 Same thing than the previous one, but with the process file descriptor used to load/mmap the edlf64 Same thing than the previous one, but with the process file descriptor used to load/mmap the edlf64
132 file instead of the pathname. You could even add the directory file descriptor.
159 file instead of the pathname. You could even add the directory file descriptor, and since that
160 would make more than 6 parameters, better use a transient structure to pass the whole data.
133 161
134 Recommended C prototype of fini function:
162 Recommended C prototype of basic fini function:
135 163 void fini(void) void fini(void)
136 164 ---------------------------------------------------------------------------------------------------- ----------------------------------------------------------------------------------------------------
165 Dead-locks/circular dependencies can happen while coarse init-ing/fini-ting inter-dependent dynamic
166 libraries. Usually, it means you need to break them down, or have fine-grained init/fini functions
167 specific to a caller/subsystem, and that, probably in several steps.
168 Or the other way around: fuse them all in one common dynamic library with a big init and use other
169 dynamic libraries as user level stubs resolving in this very common dynamic library. Of course,
170 those user level stubs could contain some code "only for the user" to call.
171 ====================================================================================================
137 172 NOTES: NOTES:
138 173
139 174 No static and implicit TLS anymore, explicit initialization is required, for instance via the No static and implicit TLS anymore, explicit initialization is required, for instance via the
Hints:
Before first commit, do not forget to setup your git environment:
git config --global user.name "your_name_here"
git config --global user.email "your@email_here"

Clone this repository using HTTP(S):
git clone https://rocketgit.com/user/sylware/nyanlinux

Clone this repository using ssh (do not forget to upload a key first):
git clone ssh://rocketgit@ssh.rocketgit.com/user/sylware/nyanlinux

Clone this repository using git:
git clone git://git.rocketgit.com/user/sylware/nyanlinux

You are allowed to anonymously push to this repository.
This means that your pushed commits will automatically be transformed into a merge request:
... clone the repository ...
... make some changes and some commits ...
git push origin main