catalinux / rocketgit (public) (License: Affero GPLv3) (since 2015-03-19) (hash sha1)
Main RocketGit repo
Clone URLs: https://rocketgit.com/user/catalinux/rocketgit ssh://rocketgit@ssh.rocketgit.com/user/catalinux/rocketgit git://git.rocketgit.com/user/catalinux/rocketgit
v0.0 v0.1 v0.10 v0.11 v0.12 v0.13 v0.14 v0.15 v0.17 v0.18 v0.19 v0.2 v0.20 v0.21 v0.22 v0.23 v0.24 v0.25 v0.27 v0.28 v0.29 v0.31 v0.32 v0.33 v0.34 v0.35 v0.36 v0.38 v0.39 v0.40 v0.41 v0.42 v0.43 v0.44 v0.45 v0.46 v0.47 v0.48 v0.49 v0.50 v0.51 v0.52 v0.53 v0.54 v0.55 v0.56 v0.57 v0.58 v0.60 v0.61 v0.62 v0.63 v0.64 v0.65 v0.66 v0.67 v0.68 v0.69 v0.70 v0.71 v0.72 v0.73 v0.77 v0.78 v0.79 v0.80 group1/branch1 group2/subgroup1/branch2 main master release
List of commits:
Subject Hash Author Date (UTC)
Collation auto-fix; misc 3ee15faf5ab0e653c7a159f0bebfd05bad8db217 Catalin(ux) M. BOIE 2024-11-20 16:12:55
Collation + block Facebook and OpenAI 43caacb3a908f6b7ffed15688b5d91fc3c004728 Catalin(ux) M. BOIE 2024-10-26 08:08:27
Debian changelog 55542035e461b712b7fd80404ef8a4c02956b1e9 Catalin(ux) M. BOIE 2024-06-07 20:42:31
Builder and worker fixes. ab54e53e1e9c202ce0bda4faa2646e6215de913d Catalin(ux) M. BOIE 2024-06-07 19:58:33
Cosmetic 87887c5717140cb4af477e8662858fd5412a3dd3 Catalin(ux) M. BOIE 2024-06-05 06:53:47
cache locking + show gpg signer b37d36501abd958894686c643985f218470ca988 Catalin(ux) M. BOIE 2024-06-05 06:08:08
Cosmetic 1d638cee22854ab2f98ae3a72afd34ea337c101a Catalin(ux) M. BOIE 2024-06-05 06:06:33
Mostly cosmetic 427e7056c6593359b68317c071d175cc38b1705d Catalin(ux) M. BOIE 2023-12-17 12:01:17
readme rendering: everything not makrdown is treated as text d7c1a9e039e3b7945682ee2e13745cd0d4ad446a Catalin(ux) M. BOIE 2023-06-22 03:40:21
Really bumbed version to 0.78 223d494efb0cc012c72fe8898e2800ce15e4eaf7 Catalin(ux) M. BOIE 2023-06-15 20:36:31
Make rgfs more verbose in case of errors 197c21765870fa6b95bb08c8599de9decf3980eb Catalin(ux) M. BOIE 2023-06-15 20:33:18
Cosmetic d48682ffe54991deedd4989a100a0607cafefab8 Catalin(ux) M. BOIE 2023-06-14 17:11:03
Fixed delete account op 4157f88a19d1b24934743d0193755a6dc3f98c13 Catalin(ux) M. BOIE 2023-06-14 17:10:22
Render readme files on the project page a8c09adb1bceed31f8070386158561b06470a98f Catalin(ux) M. BOIE 2023-06-14 17:09:17
Added rg_sid global variable to simplify the function calling 0047908fcd2ed0302a826487ddd15ca2579a9a83 Catalin(ux) M. BOIE 2023-06-14 17:06:39
Removed delta support for rpms because Fedora will remove it 2c26be90f4bc08a06ab03ab016e9ccaa843780e5 Catalin(ux) M. BOIE 2023-05-03 05:04:40
Added first payments option; fixes all around 7102aed9f239d529723799171553e8c4bd018440 Catalin(ux) M. BOIE 2023-05-02 22:36:28
Fixed a nasty typo in the build system; doc update; cosmetic 1934732b297d6476be7954fabdcaf9eb87678f5e Catalin(ux) M. BOIE 2022-12-21 12:51:53
Send client features to the builder dcf648353662409b9e39a422228ddb6a7c43358c Catalin(ux) M. BOIE 2022-12-21 11:10:00
Add support for SHA-256 git repos 8b88927d353c7b588909d0b1220c8922b32129c0 Catalin(ux) M. BOIE 2022-12-21 11:03:19
Commit 3ee15faf5ab0e653c7a159f0bebfd05bad8db217 - Collation auto-fix; misc
Author: Catalin(ux) M. BOIE
Author date (UTC): 2024-11-20 16:12
Committer name: Catalin(ux) M. BOIE
Committer date (UTC): 2024-11-20 16:12
Parent(s): 43caacb3a908f6b7ffed15688b5d91fc3c004728
Signer:
Signing key:
Signing status: N
Tree: ed574674fce97875966530218d081bc528b438d9
File Lines added Lines deleted
History.txt 2 0
TODO 5 0
inc/builder.inc.php 1 1
inc/conn.inc.php 12 11
inc/struct.inc.php 136 13
inc/util.inc.php 6 1
scripts/builder.php 23 10
scripts/worker.php 29 15
tests/common.php 21 9
tests/config.php 1 1
tests/git-commit-sign-gpg.sh 4 0
tests/state.php 4 3
File History.txt changed (mode: 100644) (index 21cf5c7..07d4a22)
38 38 2024-10-25 - The second NVMe crashed and was replaced by Hetzner. Now: 2024-10-25 - The second NVMe crashed and was replaced by Hetzner. Now:
39 39 nvme1n1 S675NF0WC78074 SAMSUNG MZVL2512HCJQ-00B00 GXA7801Q nvme1n1 S675NF0WC78074 SAMSUNG MZVL2512HCJQ-00B00 GXA7801Q
40 40 nvme0n1 S675NF0X355407 SAMSUNG MZVL2512HCJQ-00B00 GXA7801Q nvme0n1 S675NF0X355407 SAMSUNG MZVL2512HCJQ-00B00 GXA7801Q
41 2024-10-26 - Blocked Facebook and OpenAI for violating the ToS.
42
41 43
File TODO changed (mode: 100644) (index e7857b2..58df4c6)
19 19 [ ] Partner with Solo Keys? [ ] Partner with Solo Keys?
20 20 [ ] Enable disabled environments. [ ] Enable disabled environments.
21 21 [ ] We should not disable CentOS7. Only CentOS8! At least on r1 it is disabled! [ ] We should not disable CentOS7. Only CentOS8! At least on r1 it is disabled!
22 [ ] Clean /var/lib/rocketgit/tmp!
22 23 [ ] [ ]
23 24
24 25 == next release == == next release ==
26 [ ] Add some hints on page /op/admin/distros/list.
25 27 [ ] Set fillfactor for table which changes a lot (keys, events, build_jobs etc.) [ ] Set fillfactor for table which changes a lot (keys, events, build_jobs etc.)
26 28 [ ] Add a test with an empty folder. [ ] Add a test with an empty folder.
27 29 [ ] Double message: [ ] Double message:
 
84 86 [ ] [ ]
85 87
86 88 == Next next release == == Next next release ==
89 [ ] Add mlkem768x25519-sha256 (post-quantum) key type.
90 What about '-sk' variant?
91 [ ] Collect SELinux/AppArmor failures and report them.
87 92 [ ] Pass pkg repo info inside the build process to allow building optimized [ ] Pass pkg repo info inside the build process to allow building optimized
88 93 packages for some architectures. packages for some architectures.
89 94 [ ] Table apikeys has no index on 'key'. Should it? [ ] Table apikeys has no index on 'key'. Should it?
File inc/builder.inc.php changed (mode: 100644) (index ae21e2a..f4ded6d)
... ... function rg_builder_nice_status(&$a)
235 235 $a['vm_start_nice'] = 'n/a'; $a['vm_start_nice'] = 'n/a';
236 236
237 237 if (isset($a['build_sh_start']) && ($a['build_sh_start'] > 0)) if (isset($a['build_sh_start']) && ($a['build_sh_start'] > 0))
238 $a['build_sh_start_nice'] = gmdate('Y-m-d H:i', trim($a['build_sh_start']));
238 $a['build_sh_start_nice'] = gmdate('Y-m-d H:i', intval($a['build_sh_start']));
239 239 else else
240 240 $a['build_sh_start_nice'] = 'n/a'; $a['build_sh_start_nice'] = 'n/a';
241 241
File inc/conn.inc.php changed (mode: 100644) (index be89a34..598ae70)
... ... function rg_conn_shutdown($key, $what)
39 39 /* /*
40 40 * Destroys a connection * Destroys a connection
41 41 */ */
42 function rg_conn_destroy($key)
42 function rg_conn_destroy($key, $msg)
43 43 { {
44 44 global $rg_conns; global $rg_conns;
45 45 global $rg_events; global $rg_events;
 
... ... function rg_conn_destroy($key)
47 47 if (!isset($rg_conns[$key])) if (!isset($rg_conns[$key]))
48 48 rg_internal_error('key not defined!'); rg_internal_error('key not defined!');
49 49
50 rg_log($key . ': bytes_recv=' . $rg_conns[$key]['bytes_recv']
50 rg_log($key . ': conn_destroy: ' . $msg);
51
52 rg_log($key . ': conn_destroy: bytes_recv=' . $rg_conns[$key]['bytes_recv']
51 53 . ' bytes_sent=' . $rg_conns[$key]['bytes_sent']); . ' bytes_sent=' . $rg_conns[$key]['bytes_sent']);
52 54
53 55 @socket_close($rg_conns[$key]['socket']); @socket_close($rg_conns[$key]['socket']);
 
... ... function rg_conn_destroy($key)
61 63 unset($rg_events['w'][$key]); unset($rg_events['w'][$key]);
62 64
63 65 if ($rg_conns[$key]['exit_on_close']) { if ($rg_conns[$key]['exit_on_close']) {
64 rg_log($key . ': exit_on_close is set, so exit');
66 rg_log($key . ': conn_destroy: exit_on_close is set, so exit');
65 67 exit(1); exit(1);
66 68 } }
67 69
 
... ... function rg_conn_send($key)
139 141 //rg_log($key . ': SEND: ' . $s['send']); //rg_log($key . ': SEND: ' . $s['send']);
140 142 $r = @socket_send($s['socket'], $s['send'], strlen($s['send']), 0); $r = @socket_send($s['socket'], $s['send'], strlen($s['send']), 0);
141 143 if ($r === FALSE) { if ($r === FALSE) {
142 rg_log($key . ': Cannot send: ' . socket_strerror(socket_last_error()));
144 $m = 'Cannot send: ' . socket_strerror(socket_last_error());
143 145 $s['func_error']($key); $s['func_error']($key);
144 rg_conn_destroy($key);
146 rg_conn_destroy($key, $m);
145 147 return FALSE; return FALSE;
146 148 } }
147 149
 
... ... function rg_conn_recv($key)
192 194
193 195 $r = @socket_recv($s['socket'], $buf, 16 * 4096, 0); $r = @socket_recv($s['socket'], $buf, 16 * 4096, 0);
194 196 if ($r === FALSE) { if ($r === FALSE) {
195 rg_log($key . ': cannot receive: '
196 . socket_strerror(socket_last_error()));
197 $m = 'cannot receive: ' . socket_strerror(socket_last_error());
197 198 $s['func_error']($key); $s['func_error']($key);
198 rg_conn_destroy($key);
199 rg_conn_destroy($key, $m);
199 200 return FALSE; return FALSE;
200 201 } }
201 202 if ($r === 0) { if ($r === 0) {
202 rg_log($key . ': remote closed normally');
203 $m = 'remote closed normally';
203 204 $s['func_close']($key); $s['func_close']($key);
204 rg_conn_destroy($key);
205 rg_conn_destroy($key, $m);
205 206 return FALSE; return FALSE;
206 207 } }
207 208 //rg_log($key . ': RECV: ' . $buf); //rg_log($key . ': RECV: ' . $buf);
 
... ... function rg_conn_wait($timeout)
247 248 //if (!empty($e2)) //if (!empty($e2))
248 249 // rg_log_ml('error events: ' . rg_array2string($e2)); // rg_log_ml('error events: ' . rg_array2string($e2));
249 250 foreach ($e2 as $key => $sock) foreach ($e2 as $key => $sock)
250 rg_conn_destroy($key);
251 rg_conn_destroy($key, 'socket_select error');
251 252 } }
252 253
253 254 /* /*
File inc/struct.inc.php changed (mode: 100644) (index 364f0e0..2dc9c81)
... ... $rg_sql_struct[51]['other'] = array(
831 831 'collate payments.order_id' => 'ALTER TABLE payments ALTER COLUMN order_id SET DATA TYPE TEXT COLLATE "C"', 'collate payments.order_id' => 'ALTER TABLE payments ALTER COLUMN order_id SET DATA TYPE TEXT COLLATE "C"',
832 832 'collate payments_history.pay_id' => 'ALTER TABLE payments_history ALTER COLUMN pay_id SET DATA TYPE CHAR(16) COLLATE "C"', 'collate payments_history.pay_id' => 'ALTER TABLE payments_history ALTER COLUMN pay_id SET DATA TYPE CHAR(16) COLLATE "C"',
833 833 'ff keys table' => 'ALTER TABLE keys SET (FILLFACTOR = 95)', 'ff keys table' => 'ALTER TABLE keys SET (FILLFACTOR = 95)',
834 'ff keys index key_id' => 'ALTER INDEX keys_pkey SET (FILLFACTOR = 95)',
835 'ff keys vacuum' => 'VACUUM FULL keys'
834 'ff keys index key_id' => 'ALTER INDEX keys_pkey SET (FILLFACTOR = 95)'
836 835 ); );
837 836
838 837 // Here 0.80 was released // Here 0.80 was released
 
... ... $rg_sql_schema_ver = count($rg_sql_struct);
846 845
847 846
848 847
848 /*
849 * Returns information about COLLATION
850 */
851 $rg_sql_collation_info = FALSE;
852 function rg_sql_collation_info($db)
853 {
854 global $rg_sql_collation_info;
855
856 $ret = FALSE;
857 $info = array();
858 do {
859 if ($rg_sql_collation_info !== FALSE) {
860 $ret = $rg_sql_collation_info;
861 break;
862 }
863
864 $r = rg_state_get($db, "schema_version");
865 $sql = 'SELECT datcollate AS col, datcollversion AS db_ver, pg_database_collation_actual_version(oid) AS db_actual_ver'
866 . ' FROM pg_database'
867 . ' WHERE datname = current_database()';
868 $res = rg_sql_query($db, $sql);
869 if ($res === FALSE)
870 break;
871
872 $info = rg_sql_fetch_array($res);
873 rg_sql_free_result($res);
874
875 /*
876 $params = array('name' => $info['col']);
877 $sql = 'SELECT collversion AS coll_ver, pg_collation_actual_version(oid) AS coll_actual_ver'
878 . ' FROM pg_collation'
879 . ' WHERE collname = @@name@@';
880 $res = rg_sql_query_params($db, $sql, $params);
881 if ($res === FALSE)
882 break;
883
884 $row = rg_sql_fetch_array($res);
885 rg_sql_free_result($res);
886
887 $info['coll_ver'] = $row['coll_ver'];
888 $info['coll_actual_ver'] = $row['coll_actual_ver'];
889 */
890
891 $rg_sql_collation_info = $info;
892 $ret = $rg_sql_collation_info;
893 } while (0);
894
895 return $ret;
896 }
897
849 898 /* /*
850 899 * Generate structure * Generate structure
851 900 */ */
 
... ... function rg_sql_struct_run($db, $flags, $old_schema_ver)
864 913
865 914 $ret = TRUE; $ret = TRUE;
866 915
867 $name = rg_sql_escape_identifier($db, rg_sql_dbname($db));
868 rg_log('DEBUG: name=' . $name);
869 $sql = 'ALTER DATABASE ' . $name . ' REFRESH COLLATION VERSION';
870 $res = rg_sql_query($db, $sql);
871 if ($res === FALSE) {
872 if (!$ignore_errors)
873 $ret = FALSE;
874 } else {
875 rg_sql_free_result($res);
876 }
877
878 916 for ($i = $old_schema_ver + 1; $i <= $rg_sql_schema_ver; $i++) { for ($i = $old_schema_ver + 1; $i <= $rg_sql_schema_ver; $i++) {
879 917 foreach ($rg_sql_struct[$i] as $type => $sqls) { foreach ($rg_sql_struct[$i] as $type => $sqls) {
880 918 if (count($sqls) == 0) if (count($sqls) == 0)
 
... ... function rg_sql_struct_update_needed($db)
965 1003 return $ret; return $ret;
966 1004 } }
967 1005
1006 function rg_sql_collation_update_needed($db)
1007 {
1008 rg_log_enter('sql_collation_update_needed');
1009 $ret = array('ok' => 0, 'needed' => 0);
1010 do {
1011 $ci = rg_sql_collation_info($db);
1012 if ($ci === FALSE)
1013 break;
1014
1015 $ret['ok'] = 1;
1016
1017 rg_log('DEBUG: collation info [before]: ' . json_encode($ci));
1018
1019 if (strcmp($ci['db_ver'], $ci['db_actual_ver']) != 0) {
1020 rg_log('collation for db mismatch => rebuild objects');
1021 $ret['needed'] = 1;
1022 }
1023
1024 // TODO: not sure if we need to do something about this
1025 //if (strcmp($ci['coll_ver'], $ci['coll_actual_ver'] != 0)) {
1026 // rg_log('collation mismatch => rebuild objects');
1027 // $rebuild = TRUE;
1028 //}
1029 } while (0);
1030
1031 rg_log_exit('sql_collation_update_needed');
1032 return $ret;
1033 }
1034
968 1035 /* /*
969 1036 * Update schema if needed * Update schema if needed
970 1037 * Returns FALSE in case of error. * Returns FALSE in case of error.
 
... ... function rg_sql_struct_parts_update($db)
1191 1258 return $ret; return $ret;
1192 1259 } }
1193 1260
1261 function rg_sql_collation_update($db)
1262 {
1263 global $rg_sql_collation_info;
1264
1265 rg_prof_start('sql_collation_update');
1266 rg_log_enter('sql_collation_update');
1267
1268 $ret = FALSE;
1269 do {
1270 rg_log('Reindexing database...');
1271 $sql = 'REINDEX DATABASE';
1272 $res = rg_sql_query($db, $sql);
1273 if ($res === FALSE)
1274 break;
1275 rg_sql_free_result($res);
1276
1277 $rg_sql_collation_info = FALSE; // invalidate memory cache
1278 $ci = rg_sql_collation_info($db);
1279 if ($ci === FALSE)
1280 break;
1281 rg_log('DEBUG: collation info [after reindex]: ' . json_encode($ci));
1282
1283 $name = rg_sql_escape_identifier($db, rg_sql_dbname($db));
1284 rg_log('DEBUG: name=' . $name);
1285 $sql = 'ALTER DATABASE ' . $name . ' REFRESH COLLATION VERSION';
1286 $res = rg_sql_query($db, $sql);
1287 if ($res === FALSE)
1288 break;
1289 rg_sql_free_result($res);
1290
1291 $rg_sql_collation_info = FALSE; // invalidate memory cache
1292 $ci = rg_sql_collation_info($db);
1293 if ($ci === FALSE)
1294 break;
1295 rg_log('DEBUG: collation info [after]: ' . json_encode($ci));
1296
1297 $ret = TRUE;
1298 } while (0);
1299
1300 rg_log_exit();
1301 rg_prof_end('sql_collation_update');
1302
1303 return $ret;
1304 }
1305
1194 1306 /* /*
1195 1307 * Helper that will setup the structure and the fixes if needed * Helper that will setup the structure and the fixes if needed
1196 1308 */ */
 
... ... function rg_struct_ok($db)
1202 1314 do { do {
1203 1315 $restart_cache = FALSE; $restart_cache = FALSE;
1204 1316
1317 $r = rg_sql_collation_update_needed($db);
1318 if ($r['ok'] != 1)
1319 break;
1320 $uc = $r['needed'];
1321
1205 1322 $ud = rg_sql_struct_update_needed($db); $ud = rg_sql_struct_update_needed($db);
1206 1323 if ($ud === FALSE) if ($ud === FALSE)
1207 1324 break; break;
 
... ... function rg_struct_ok($db)
1218 1335 break; break;
1219 1336 $locked = TRUE; $locked = TRUE;
1220 1337
1338 while ($uc == 1) {
1339 $r = rg_sql_collation_update($db);
1340 if ($r !== TRUE)
1341 break;
1342 }
1343
1221 1344 if ($ud === 1) { if ($ud === 1) {
1222 1345 $r = rg_sql_struct_update($db, 0); $r = rg_sql_struct_update($db, 0);
1223 1346 if ($r !== TRUE) if ($r !== TRUE)
File inc/util.inc.php changed (mode: 100644) (index fc48440..7387879)
... ... function rg_save_plain($file, $a)
2947 2947 { {
2948 2948 $r = @file_put_contents($file . '.tmp', $a); $r = @file_put_contents($file . '.tmp', $a);
2949 2949 if ($r === FALSE) { if ($r === FALSE) {
2950 rg_internal_error('Cannot save file: ' . rg_php_err());
2950 rg_internal_error('Cannot save file [' . $file . '.tmp]: ' . rg_php_err());
2951 2951 return FALSE; return FALSE;
2952 2952 } }
2953 2953
 
... ... function rg_is_abuser(string $ua)
2973 2973 { {
2974 2974 if (stristr($ua, 'facebookexternalhit/')) return TRUE; if (stristr($ua, 'facebookexternalhit/')) return TRUE;
2975 2975 if (stristr($ua, 'gptbot/')) return TRUE; if (stristr($ua, 'gptbot/')) return TRUE;
2976 if (stristr($ua, 'GoogleOther')) return TRUE;
2976 2977
2977 2978 return FALSE; return FALSE;
2978 2979 } }
 
... ... function rg_is_bot($ua)
2985 2986 if (empty($ua)) return FALSE; if (empty($ua)) return FALSE;
2986 2987
2987 2988 if (stristr($ua, 'bot/')) return TRUE; if (stristr($ua, 'bot/')) return TRUE;
2989 if (stristr($ua, 'bot;')) return TRUE;
2990 if (stristr($ua, 'crawler')) return TRUE;
2988 2991
2989 2992 if (strstr($ua, ' Adsbot/')) return TRUE; if (strstr($ua, ' Adsbot/')) return TRUE;
2990 2993 if (strstr($ua, ' AhrefsBot')) return TRUE; if (strstr($ua, ' AhrefsBot')) return TRUE;
 
... ... function rg_is_bot($ua)
3024 3027 if (strstr($ua, 'Googlebot-News')) return TRUE; if (strstr($ua, 'Googlebot-News')) return TRUE;
3025 3028 if (strstr($ua, 'Googlebot-Video/')) return TRUE; if (strstr($ua, 'Googlebot-Video/')) return TRUE;
3026 3029 if (strstr($ua, 'GoScraper')) return TRUE; if (strstr($ua, 'GoScraper')) return TRUE;
3030 if (strstr($ua, 'HappyWing')) return TRUE;
3027 3031 if (strstr($ua, 'ichiro/')) return TRUE; if (strstr($ua, 'ichiro/')) return TRUE;
3028 3032 if (strstr($ua, 'IonCrawl')) return TRUE; if (strstr($ua, 'IonCrawl')) return TRUE;
3029 3033 if (strstr($ua, 'Internet-structure-research-project-bot')) return TRUE; if (strstr($ua, 'Internet-structure-research-project-bot')) return TRUE;
 
... ... function rg_is_bot($ua)
3034 3038 if (strstr($ua, ' MaCoCu;')) return TRUE; if (strstr($ua, ' MaCoCu;')) return TRUE;
3035 3039 if (strstr($ua, ' Mail.RU_Bot')) return TRUE; if (strstr($ua, ' Mail.RU_Bot')) return TRUE;
3036 3040 if (strstr($ua, ' MauiBot ')) return TRUE; if (strstr($ua, ' MauiBot ')) return TRUE;
3041 if (strstr($ua, 'search.marginalia.nu')) return TRUE;
3037 3042 if (strstr($ua, ' MegaIndex.ru/')) return TRUE; if (strstr($ua, ' MegaIndex.ru/')) return TRUE;
3038 3043 if (strstr($ua, 'meta-externalagent/')) return TRUE; if (strstr($ua, 'meta-externalagent/')) return TRUE;
3039 3044 if (strstr($ua, ' MJ12bot')) return TRUE; if (strstr($ua, ' MJ12bot')) return TRUE;
File scripts/builder.php changed (mode: 100644) (index 05761ec..8d5790c)
... ... function xdispatch_one($key, $data)
374 374 rg_log('job[worker_id]=' rg_log('job[worker_id]='
375 375 . (isset($job['worker_id']) ? $job['worker_id'] : '-')); . (isset($job['worker_id']) ? $job['worker_id'] : '-'));
376 376 rg_log('s[worker_id]=' . $s['worker_id']); rg_log('s[worker_id]=' . $s['worker_id']);
377 $err = array('op' => 'abort_job', 'jid' => $jid, 'errstr' => $m);
377 $err = array('op' => 'abort_job', 'id' => $jid, 'errstr' => $m);
378 378 rg_conn_enq($key, json_encode($err) . "\n"); rg_conn_enq($key, json_encode($err) . "\n");
379 379 break; break;
380 380 } }
 
... ... function rg_process_job($db, &$job)
768 768 global $rg_conns; global $rg_conns;
769 769
770 770 $jid = $job['id']; $jid = $job['id'];
771 rg_log_enter('rg_process_job: jid=' . $jid);
772 771
773 772 $ret = TRUE; $ret = TRUE;
774 773 do { do {
774 if (!array_key_exists('last_logged_reason', $job))
775 $job['last_logged_reason'] = '';
776
775 777 // Job is already in progress? // Job is already in progress?
776 778 if ($job['worker_id'] > 0) { if ($job['worker_id'] > 0) {
777 //rg_log($jid . ': job is already assigned to'
778 // . ' worker [' . $job['worker_name'] . ']'
779 // . ' (id ' . $job['worker_id'] . '); skip it');
779 $reason = 'job is already assigned to'
780 . ' worker [' . $job['worker_name'] . ']'
781 . ' (id ' . $job['worker_id'] . '); skip it';
782 if (strcmp($reason, $job['last_logged_reason']) != 0) {
783 rg_log($jid . ': ' . $reason);
784 $job['last_logged_reason'] = $reason;
785 }
780 786 $ret = TRUE; $ret = TRUE;
781 787 break; break;
782 788 } }
783 789
784 790 // Should we delay because of a previous fail? // Should we delay because of a previous fail?
785 791 if (isset($job['next_try']) && ($job['next_try'] > time())) { if (isset($job['next_try']) && ($job['next_try'] > time())) {
786 //rg_log_debug($jid . ': job is suspended till '
787 // . date('Y-m-d H:i:s', $job['next_try'])
788 // . ' because [' . $job['next_try_reason'] . ']');
792 $reason = 'job is suspended till '
793 . date('Y-m-d H:i:s', $job['next_try'])
794 . ' because [' . $job['next_try_reason'] . ']';
795 if (strcmp($reason, $job['last_logged_reason']) != 0) {
796 rg_log($jid . ': ' . $reason);
797 $job['last_logged_reason'] = $reason;
798 }
789 799 $ret = TRUE; $ret = TRUE;
790 800 break; break;
791 801 } }
 
... ... function rg_process_job($db, &$job)
939 949 $ret = TRUE; $ret = TRUE;
940 950 } while (0); } while (0);
941 951
942 rg_log_exit();
943 952 return $ret; return $ret;
944 953 } }
945 954
 
... ... if (!is_dir($rg_state_dir . '/builder')) {
965 974
966 975 $jobs = array(); $jobs = array();
967 976 $state_dir = $rg_state_dir . '/builder'; $state_dir = $rg_state_dir . '/builder';
968 $saved_jobs = rg_load_files($state_dir, 'job-[0-9]*.ser', 'id');
977 $saved_jobs = rg_load_files($state_dir, '^job-[0-9]*.ser$', 'id');
978 if ($saved_jobs === FALSE) {
979 rg_log_debug('Error loading jobs: ' . rg_util_error());
980 $saved_jobs = array();
981 }
969 982 if (!empty($saved_jobs)) if (!empty($saved_jobs))
970 983 rg_log_debug('Jobs loaded from dir: ' . rg_array2string_short($saved_jobs)); rg_log_debug('Jobs loaded from dir: ' . rg_array2string_short($saved_jobs));
971 984
File scripts/worker.php changed (mode: 100644) (index 5abfb69..9bd1fc5)
... ... function load_config_file($file)
105 105 } else if (strcmp($var, 'include') == 0) { } else if (strcmp($var, 'include') == 0) {
106 106 if (strncmp($value, '/', 1) != 0) if (strncmp($value, '/', 1) != 0)
107 107 $value = dirname($file) . '/' . $value; $value = dirname($file) . '/' . $value;
108 load_config_file($value);
108 $g = glob($value);
109 foreach ($g as $f)
110 load_config_file($f);
109 111 } else if (strcmp($var, 'debug') == 0) { } else if (strcmp($var, 'debug') == 0) {
110 112 rg_debug_set(intval($value)); rg_debug_set(intval($value));
111 113 } else { } else {
 
... ... function start_worker($job)
1533 1535 rg_exec('virsh destroy ' . $ename, '', FALSE, FALSE, FALSE); rg_exec('virsh destroy ' . $ename, '', FALSE, FALSE, FALSE);
1534 1536 rg_exec('virsh undefine --nvram ' . $ename, '', FALSE, FALSE, FALSE); rg_exec('virsh undefine --nvram ' . $ename, '', FALSE, FALSE, FALSE);
1535 1537
1536 $r = rg_exec('qemu-img create -o lazy_refcounts=on,cluster_size=256K'
1538 $r = rg_exec('qemu-img create'
1539 . ' -o lazy_refcounts=on,cluster_size=256K,preallocation=off'
1537 1540 . ' -b ' . $master . ' -F qcow2' . ' -b ' . $master . ' -F qcow2'
1538 1541 . ' -f qcow2 ' . $eimg, '', FALSE, FALSE, FALSE); . ' -f qcow2 ' . $eimg, '', FALSE, FALSE, FALSE);
1539 1542 if ($r['ok'] !== 1) { if ($r['ok'] !== 1) {
 
... ... function start_worker($job)
1547 1550 if ($r !== TRUE) if ($r !== TRUE)
1548 1551 break; break;
1549 1552
1550 $r = rg_exec('qemu-img create -f raw ' . $eimg2
1553 $r = rg_exec('qemu-img create -f raw'
1554 . ' -o preallocation=off'
1555 . ' ' . $eimg2
1551 1556 . ' ' . escapeshellarg($job['disk_size_gib'] . 'G'), . ' ' . escapeshellarg($job['disk_size_gib'] . 'G'),
1552 1557 '', FALSE, FALSE, FALSE); '', FALSE, FALSE, FALSE);
1553 1558 if ($r['ok'] !== 1) { if ($r['ok'] !== 1) {
 
... ... function start_worker($job)
1572 1577 $reason2 = $r['errmsg'] . ': ' . $r['stderr']; $reason2 = $r['errmsg'] . ': ' . $r['stderr'];
1573 1578 break; break;
1574 1579 } }
1580 rg_log('mkfs done');
1575 1581
1576 1582 $r = @mkdir($job['main'] . '/root', 0700); $r = @mkdir($job['main'] . '/root', 0700);
1577 1583 if ($r === FALSE) { if ($r === FALSE) {
 
... ... function xhandle_one($key, $data)
2054 2060 $u = @json_decode($data, TRUE); $u = @json_decode($data, TRUE);
2055 2061 if ($u === NULL) { if ($u === NULL) {
2056 2062 rg_log($key . ': JSON: ' . $data); rg_log($key . ': JSON: ' . $data);
2057 rg_log_ml($key . ': cannot decode JSON: ' . json_last_error_msg());
2058 $err = array('errstr' => 'cannot decode json');
2063 $m = 'cannot decode JSON: ' . json_last_error_msg();
2064 $err = array('errstr' => $m);
2059 2065 rg_conn_enq('master', json_encode($err) . "\n"); rg_conn_enq('master', json_encode($err) . "\n");
2060 rg_conn_destroy($key);
2066 rg_conn_destroy($key, $m);
2061 2067 return; return;
2062 2068 } }
2069 //rg_log_debug('JSON: ' . print_r($u, TRUE));
2063 2070
2064 $jid = isset($u['id']) ? $u['id'] : 0;
2071 $jid = array_key_exists('id', $u) ? $u['id'] : 0;
2072 //rg_log_debug('jid=' . $jid);
2065 2073
2066 2074 if (isset($u['op'])) { if (isset($u['op'])) {
2067 2075 $op = $u['op']; $op = $u['op'];
 
... ... function xhandle_one($key, $data)
2093 2101
2094 2102 // From here, we require a jid // From here, we require a jid
2095 2103 if ($jid == 0) { if ($jid == 0) {
2096 $err = array('errstr' => 'job id not specified');
2104 $m = 'job id not specified in JSON [' . $data . ']';
2105 $err = array('errstr' => $m);
2097 2106 rg_conn_enq('master', json_encode($err) . "\n"); rg_conn_enq('master', json_encode($err) . "\n");
2098 rg_conn_destroy($key);
2107 rg_conn_destroy($key, $m);
2099 2108 return; return;
2100 2109 } }
2101 2110
 
... ... function xhandle_one($key, $data)
2136 2145 if (strcmp($op, 'DRE') == 0) { // DRE = done received if (strcmp($op, 'DRE') == 0) { // DRE = done received
2137 2146 if (!isset($jobs[$jid])) { if (!isset($jobs[$jid])) {
2138 2147 $k = array_keys($jobs); $k = array_keys($jobs);
2139 rg_log($key . ': ' . $jid . ': DEBUG:'
2148 rg_log($key . ': ' . $jid . ': '
2140 2149 . ' DRE for a non existing job! Ignore cmd.'); . ' DRE for a non existing job! Ignore cmd.');
2141 rg_log($key . ': ' . $jid . ': DEBUG:'
2150 rg_log($key . ': ' . $jid . ': '
2142 2151 . ' existing jobs: ' . implode(',', $k) . '.'); . ' existing jobs: ' . implode(',', $k) . '.');
2143 2152 // We already received a DRE message // We already received a DRE message
2144 2153 return; return;
 
... ... function xhandle_one($key, $data)
2194 2203 } }
2195 2204 $job['artifacts_state'][$aid]['wait_status'] = 0; $job['artifacts_state'][$aid]['wait_status'] = 0;
2196 2205 } else { } else {
2197 rg_log_ml($key . ': cannot handle op: ' . $u['op'] . ': ' . rg_array2string($u));
2206 rg_log_ml($key . ': cannot handle json: ' . $data);
2198 2207 } }
2199 2208
2200 2209 unset($jobs); unset($jobs);
 
... ... function vm_do_stats(&$job, $cd)
2893 2902
2894 2903 do { do {
2895 2904 if (!isset($features['vm_stats'])) { if (!isset($features['vm_stats'])) {
2896 rg_log_debug('vm_stats feature is not supported by the server');
2905 rg_log_debug('vm_stats feature is not supported by the server (or not received yet)');
2897 2906 rg_log_debug('supported: ' . rg_array2string($features)); rg_log_debug('supported: ' . rg_array2string($features));
2898 2907 break; break;
2899 2908 } }
 
... ... $features = array();
3101 3110 $stats = array('jobs' => 0); $stats = array('jobs' => 0);
3102 3111
3103 3112 rg_log('Loading job-*.ser files...'); rg_log('Loading job-*.ser files...');
3104 $jobs = rg_load_files($conf['state'], 'job-[0-9]*.ser', 'id');
3113 $jobs = rg_load_files($conf['state'], '^job-[0-9]*.ser$', 'id');
3114 if ($jobs === FALSE) {
3115 rg_log_debug('Error loading jobs: ' . rg_util_error());
3116 exit(1);
3117 }
3105 3118 if (!empty($jobs)) { if (!empty($jobs)) {
3106 3119 $k = array_keys($jobs); $k = array_keys($jobs);
3107 3120 rg_log_debug('Jobs loaded from dir: ' . implode(',', $k) . '.'); rg_log_debug('Jobs loaded from dir: ' . implode(',', $k) . '.');
 
... ... while(1) {
3140 3153 rg_log($jid . ': DEBUG: Pid ' . $pid . ' exited' rg_log($jid . ': DEBUG: Pid ' . $pid . ' exited'
3141 3154 . ' with status ' . $status . '! Switch to DONE'); . ' with status ' . $status . '! Switch to DONE');
3142 3155 $jobs[$jid]['error'] = 'helper error'; $jobs[$jid]['error'] = 'helper error';
3143 $jobs[$jid]['error2'] = 'helper error';
3156 $jobs[$jid]['error2'] = 'helper error (exited with code ' . $status . ')';
3144 3157 $jobs[$jid]['state'] = RG_JOB_DONE; $jobs[$jid]['state'] = RG_JOB_DONE;
3145 3158 $jobs[$jid]['dirty'] = 1; $jobs[$jid]['dirty'] = 1;
3146 3159 save_job($job); save_job($job);
 
... ... while(1) {
3211 3224 $stats['jobs']++; $stats['jobs']++;
3212 3225 } else if (!isset($job['helper_start']) } else if (!isset($job['helper_start'])
3213 3226 || ($job['helper_start'] + 300 < time())) { || ($job['helper_start'] + 300 < time())) {
3227 rg_log_debug('vms: ' . print_r($vms));
3214 3228 // TODO: we should check if pid is still valid? // TODO: we should check if pid is still valid?
3215 3229 rg_log($jid . ': Waited 300s without the VM to show up!' rg_log($jid . ': Waited 300s without the VM to show up!'
3216 3230 . ' Set state to INIT.'); . ' Set state to INIT.');
File tests/common.php changed (mode: 100644) (index 60ab999..871b18f)
... ... rg_cache_core_set($db, 'state::http_allow', '9002');
29 29 rg_cache_core_set($db, 'state::https_allow', '9003'); rg_cache_core_set($db, 'state::https_allow', '9003');
30 30 $test_url = rg_base_url($db, '', ''); $test_url = rg_base_url($db, '', '');
31 31
32 // Force debug by setting the settings
33 $r = rg_state_set($db, 'debug', '1');
34 if ($r === FALSE) {
35 rg_internal_error('Cannot set state (debug var)!');
36 exit(1);
37 }
38 rg_debug_set(1);
39
40
41 32 if (isset($rg_no_db) && $rg_no_db) if (isset($rg_no_db) && $rg_no_db)
42 33 return; return;
43 34
44 35
45 36 rg_log("Redo schema..."); rg_log("Redo schema...");
46 37
38 $r = rg_sql_collation_update_needed($db);
39 if ($r['ok'] != 1) {
40 rg_log('collation error: ' . rg_sql_error());
41 exit(1);
42 }
43 if ($r['needed'] == 1) {
44 $r = rg_sql_collation_update($db, 0);
45 if ($r !== TRUE) {
46 rg_log("Cannot update collation (" . rg_sql_error() . ")!");
47 exit(1);
48 }
49 }
50
47 51 $r = rg_sql_struct_update($db, 0); $r = rg_sql_struct_update($db, 0);
48 52 if ($r !== TRUE) { if ($r !== TRUE) {
49 53 rg_log("Cannot create struct (" . rg_sql_error() . ")!"); rg_log("Cannot create struct (" . rg_sql_error() . ")!");
 
... ... if ($r !== TRUE) {
62 66 exit(1); exit(1);
63 67 } }
64 68
69 // Force debug by setting the settings
70 $r = rg_state_set($db, 'debug', '1');
71 if ($r === FALSE) {
72 rg_internal_error('Cannot set state (debug)!');
73 exit(1);
74 }
75 rg_debug_set(1);
76
65 77 rg_log("common.php finished"); rg_log("common.php finished");
66 78 rg_log(''); rg_log('');
67 79
File tests/config.php changed (mode: 100644) (index 226ca11..447eaa5)
2 2
3 3 // Defaults // Defaults
4 4 $rg_base = dirname(__FILE__); $rg_base = dirname(__FILE__);
5 $rg_sql = "dbname=trg connect_timeout=10";
5 $rg_sql = "user=rocketgit dbname=trg connect_timeout=10";
6 6 if (!isset($rg_sql_debug)) if (!isset($rg_sql_debug))
7 7 $rg_sql_debug = 1; $rg_sql_debug = 1;
8 8 $rg_session_time = 3600; $rg_session_time = 3600;
File tests/git-commit-sign-gpg.sh added (mode: 100755) (index 0000000..4ff927e)
1 #!/bin/bash
2
3 exec gpg --passphrase='' --homedir=../git-commit-sign-gpg "${@}"
4
File tests/state.php changed (mode: 100644) (index f29e6c2..c59b67e)
2 2 error_reporting(E_ALL | E_STRICT); error_reporting(E_ALL | E_STRICT);
3 3 ini_set("track_errors", "On"); ini_set("track_errors", "On");
4 4
5 $rg_sql_debug = 1;
6 $rg_cache_debug = TRUE;
7 $test_normal = TRUE;
8
5 9 $INC = dirname(__FILE__) . "/../inc"; $INC = dirname(__FILE__) . "/../inc";
6 10 require_once(dirname(__FILE__) . "/config.php"); require_once(dirname(__FILE__) . "/config.php");
7 11 require_once($INC . "/init.inc.php"); require_once($INC . "/init.inc.php");
 
... ... rg_log_set_file("state.log");
14 18
15 19 require_once("common.php"); require_once("common.php");
16 20
17 $rg_sql_debug = 1;
18 $rg_cache_debug = TRUE;
19
20 21 $k = 'key-' . rg_id(16); $k = 'key-' . rg_id(16);
21 22
22 23 rg_log(''); rg_log('');
Date/time (UTC) Type Misc Labels
2024-11-20 16:22 build fedora-40-x86_64 worker/r1 builder/color=fff worker_elap/58s wait_time/1s date/2024-11-20 time/16:21
Hints:
Before first commit, do not forget to setup your git environment:
git config --global user.name "your_name_here"
git config --global user.email "your@email_here"

Clone this repository using HTTP(S):
git clone https://rocketgit.com/user/catalinux/rocketgit

Clone this repository using ssh (do not forget to upload a key first):
git clone ssh://rocketgit@ssh.rocketgit.com/user/catalinux/rocketgit

Clone this repository using git:
git clone git://git.rocketgit.com/user/catalinux/rocketgit

You are allowed to anonymously push to this repository.
This means that your pushed commits will automatically be transformed into a merge request:
... clone the repository ...
... make some changes and some commits ...
git push origin main