catalinux / rocketgit (public) (License: Affero GPLv3) (since 2015-03-19) (hash sha1)
Main RocketGit repo
Clone URLs: https://rocketgit.com/user/catalinux/rocketgit ssh://rocketgit@ssh.rocketgit.com/user/catalinux/rocketgit git://git.rocketgit.com/user/catalinux/rocketgit
v0.0 v0.1 v0.10 v0.11 v0.12 v0.13 v0.14 v0.15 v0.17 v0.18 v0.19 v0.2 v0.20 v0.21 v0.22 v0.23 v0.24 v0.25 v0.27 v0.28 v0.29 v0.31 v0.32 v0.33 v0.34 v0.35 v0.36 v0.38 v0.39 v0.40 v0.41 v0.42 v0.43 v0.44 v0.45 v0.46 v0.47 v0.48 v0.49 v0.50 v0.51 v0.52 v0.53 v0.54 v0.55 v0.56 v0.57 v0.58 v0.60 v0.61 v0.62 v0.63 v0.64 v0.65 v0.66 v0.67 v0.68 v0.69 v0.70 v0.71 v0.72 v0.73 v0.77 v0.78 group1/branch1 group2/subgroup1/branch2 main master release
List of commits:
Subject Hash Author Date (UTC)
builder: show queue in the web page 6cab7820b0bd2622afb77391a9e9515e0e837eef Catalin(ux) M. BOIE 2020-01-06 12:49:26
totp: Allow also years for 'val' command d666df0cec948f938469b9eb12a1b981cb41cfa1 Catalin(ux) M. BOIE 2020-01-04 14:39:33
worker: change git clone parameters because we could not clone 8eb2727ac17e12e148f39c72e1f4bd1e31319b33 Catalin(ux) M. BOIE 2019-12-19 03:34:26
Added a TODO for worker 492fb4546829f53adacd74c4647c3634f7c37065 Catalin(ux) M. BOIE 2019-12-18 22:23:12
worker improvements 4387b00291b5848aa08c303f9d62a126a6806a35 Catalin(ux) M. BOIE 2019-12-18 22:17:23
.spec: require tidy for building the tests c0deab2ce24388c91d2bc2e96338a260c9657dd6 Catalin(ux) M. BOIE 2019-12-18 22:15:29
nginx configuration update 015c9404229f2ff7fb91cc57ab21289f01878334 Catalin(ux) M. BOIE 2019-12-18 21:55:13
Added Business model to comparison c62d466580aca0ff3371f3ba369959c7ef7631d0 Catalin(ux) M. BOIE 2019-11-21 20:44:16
CSS small correction ded9ef6bce5bebfa815bc7ae5733bc8e0810af3b Catalin(ux) M. BOIE 2019-11-21 18:20:41
Added LDAP/qrencode to features/thanks pages d269d59787f61045f5831c6985899c28da08ebeb Catalin(ux) M. BOIE 2019-11-21 18:20:22
Invalidate slave table cache when updating the db structure, else new slaves will not be created 8e1bf7feeb87818179e7152a356e0f408da706f2 Catalin(ux) M. BOIE 2019-11-19 19:33:15
docker: push also the latest label a72e2e74355d772ae45bb99798ad47082390943c Catalin(ux) M. BOIE 2019-11-19 19:32:11
More statistics changes (load, cpu, mem) e7f38e72405a227788a7011ea17dd57d0d4de67c Catalin(ux) M. BOIE 2019-11-19 19:31:24
Cosmetic 0a0b258e8993cb5035e018b049018ac41b99eb01 Catalin(ux) M. BOIE 2019-11-17 11:09:56
Insist on respecting the privacy b464bcc35476ffa23a83beaf956e8f7b15163d7c Catalin(ux) M. BOIE 2019-11-17 11:03:36
Remove PayPal donation because of legal uncertainly cd938af4c5cdbd81190e535973588dee6b697d34 Catalin(ux) M. BOIE 2019-11-17 11:03:07
nginx sample update 1f9eb2d293e91139f397cc45cb206d8994471134 Catalin(ux) M. BOIE 2019-11-17 09:10:02
Mostly cosmetic 3bf09056ede99df46c1b59802e6cf2c7bebf4f13 Catalin(ux) M. BOIE 2019-11-17 09:09:31
Added statistics for the installation ed743a601cd187431ccd80b57113ba9bb5704035 Catalin(ux) M. BOIE 2019-11-17 09:08:33
ldap: if we have no server, it is useless to continue processing c022b68a942f471e51cb8343bfef1cda390b27b1 Catalin(ux) M. BOIE 2019-11-16 06:30:47
Commit 6cab7820b0bd2622afb77391a9e9515e0e837eef - builder: show queue in the web page
Author: Catalin(ux) M. BOIE
Author date (UTC): 2020-01-06 12:49
Committer name: Catalin(ux) M. BOIE
Committer date (UTC): 2020-01-06 12:49
Parent(s): d666df0cec948f938469b9eb12a1b981cb41cfa1
Signing key:
Tree: f1d9c6ac00ff2be6c20e1c59eea51c61be28e85f
File Lines added Lines deleted
inc/builder.inc.php 121 17
inc/util.inc.php 63 5
inc/workers.inc.php 6 2
root/themes/default/builder/cmds/footer.html 0 0
root/themes/default/builder/cmds/header.html 8 0
root/themes/default/builder/cmds/line.html 6 0
root/themes/default/builder/cmds/nodata.html 1 1
root/themes/default/builder/load_err.html 1 1
root/themes/default/builder/packages.html 6 0
root/themes/default/builder/queue/footer.html 4 0
root/themes/default/builder/queue/header.html 16 0
root/themes/default/builder/queue/line.html 26 0
root/themes/default/builder/queue/nodata.html 1 1
root/themes/default/user/settings/workers/menu.html 2 0
scripts/builder.php 104 74
scripts/builder.sh 3 3
scripts/worker.TODO 23 2
File inc/builder.inc.php changed (mode: 100644) (index 30282fc..94c6c2e)
1 1 <?php <?php
2 2 $INC = isset($INC) ? $INC : dirname(__FILE__); $INC = isset($INC) ? $INC : dirname(__FILE__);
3 require_once($INC . '/workers.inc.php');
3 require_once($INC . '/repo.inc.php');
4 require_once($INC . '/user.inc.php');
4 5
5 6 /* /*
6 7 * Function to load job list * Function to load job list
7 8 */ */
8 function rg_builder_load_jobs($db)
9 function rg_builder_load_jobs($db, $where)
9 10 { {
10 11 rg_log_enter('builder_load_jobs'); rg_log_enter('builder_load_jobs');
11 12
 
... ... function rg_builder_load_jobs($db)
15 16
16 17 while (1) { while (1) {
17 18 $sql = 'SELECT * FROM build_jobs' $sql = 'SELECT * FROM build_jobs'
18 . ' WHERE done = 0'
19 . ' ORDER BY prio DESC';
19 . ' WHERE ' . $where
20 . ' ORDER BY itime DESC'
21 . ' LIMIT 200';
20 22 $res = rg_sql_query($db, $sql); $res = rg_sql_query($db, $sql);
21 23 if ($res === FALSE) if ($res === FALSE)
22 24 break; break;
 
... ... function rg_builder_load_jobs($db)
24 26 $jid = $row['id']; $jid = $row['id'];
25 27
26 28 // TODO: set 'url' when adding the job in queue! // TODO: set 'url' when adding the job in queue!
27 $final = @unserialize($row['request']);
28 if ($final === FALSE) {
29 rg_internal_error('cannot unserialize!');
30 continue;
29 // Why not now?
30
31 $row['request'] = rg_unserialize($row['request']);
32 if ($row['request'] === FALSE) {
33 rg_internal_error('cannot unserialize request!');
34 $row['request'] = array();
35 }
36 $row['status'] = rg_unserialize($row['status']);
37 if ($row['status'] === FALSE) {
38 rg_internal_error('cannot unserialize status!');
39 $row['status'] = array();
31 40 } }
32 $final['id'] = $jid;
33 $final['repo_id'] = $row['repo_id'];
34 $final['itime'] = $row['itime'];
35 $final['prio'] = $row['prio'];
36 41
37 $ret['list'][$jid] = $final;
42 $ret['list'][$jid] = $row;
38 43 } }
39 44 rg_sql_free_result($res); rg_sql_free_result($res);
40 45
 
... ... function rg_builder_add($db, $repo_id, $d)
59 64 'repo_id' => $repo_id, 'repo_id' => $repo_id,
60 65 'prio' => 10, // TODO 'prio' => 10, // TODO
61 66 'itime' => time(), 'itime' => time(),
62 'request' => serialize($d),
67 'request' => rg_serialize($d),
63 68 'done' => 0, 'done' => 0,
64 69 'status' => '' 'status' => ''
65 70 ); );
 
... ... function rg_builder_done($db, $job, $s)
133 138 'head' => $job['head'], 'head' => $job['head'],
134 139 'type' => 'build', 'type' => 'build',
135 140 'misc' => $env, 'misc' => $env,
136 'labels' => serialize($labels),
141 'labels' => rg_serialize($labels),
137 142 'itime' => time() 'itime' => time()
138 143 ); );
139 144 $sql = 'DELETE FROM commit_labels' $sql = 'DELETE FROM commit_labels'
 
... ... function rg_builder_done($db, $job, $s)
160 165 $params = array( $params = array(
161 166 'id' => $job['id'], 'id' => $job['id'],
162 167 'done' => $job['done'], 'done' => $job['done'],
163 'status' => serialize($s)
168 'status' => rg_serialize($s)
164 169 ); );
165 170 $sql = 'UPDATE build_jobs SET done = @@done@@' $sql = 'UPDATE build_jobs SET done = @@done@@'
166 171 . ', status = @@status@@' . ', status = @@status@@'
 
... ... function rg_builder_done($db, $job, $s)
202 207 rg_log_exit(); rg_log_exit();
203 208 } }
204 209
205 ?>
210 /*
211 * Nice status for a job
212 */
213 function rg_builder_nice_status(&$a)
214 {
215 //rg_log_ml('nice_status: a: ' . print_r($a, TRUE));
216 $a['start_nice'] = gmdate('Y-m-d H:i', intval($a['start']));
217 $a['net_ok_nice'] = gmdate('Y-m-d H:i', intval($a['net_ok']));
218 $a['pkgs_ok_nice'] = gmdate('Y-m-d H:i', intval($a['pkgs_ok']));
219 $a['done_nice'] = gmdate('Y-m-d H:i', intval($a['done']));
220
221 foreach ($a['cmds'] as &$i) {
222 $i['start_nice'] = gmdate('Y-m-d H:i', intval($i['start']));
223 $i['done_nice'] = gmdate('Y-m-d H:i', intval($i['done']));
224 $i['elap'] = rg_human_time_interval($i['done'] - $i['start'] + 1);
225
226 $_a = rg_xss_safe($i['log']);
227 $i['HTML:log_nlbr'] = nl2br($_a);
228 }
229 }
230
231 /*
232 * Cosmetic fixes
233 */
234 function rg_builder_cosmetic($db, &$row)
235 {
236 if (isset($row['itime']))
237 $row['itime_nice'] = gmdate('Y-m-d H:i', $row['itime']);
238
239 if (!isset($row['done']))
240 $row['done'] = 0;
241
242 if ($row['done'] > 0)
243 $row['done_nice'] = gmdate('Y-m-d H:i', $row['done']);
244 else
245 $row['done_nice'] = 'n/a';
246
247 $ri = rg_repo_info($db, $row['repo_id'], 0, '');
248 if ($ri['ok'] == 1) {
249 $row['ri'] = $ri;
250
251 $row['user'] = rg_user_nice($db, $row['ri']['uid']);
252 } else {
253 $row['ri'] = array('name' => 'n/a');
254 $row['user'] = 'n/a';
255 }
256 }
257
258 /*
259 * Lists the build jobs
260 */
261 function rg_builder_list_high_level($db, $rg, $op, $paras)
262 {
263 rg_prof_start('builder_list_high_level');
264 rg_log_enter('builder_list_high_level op=' . $op);
265
266 $ret = '';
267
268 $errmsg = array();
269 $rg['HTML:status'] = '';
270
271 if ($rg['login_ui']['is_admin'] == 1)
272 $where = '1 = 1';
273 else
274 $where = 'repo_id IN (SELECT repo_id FROM repos'
275 . ' WHERE uid = ' . $rg['login_ui']['uid'] . ')';
276
277 if (strcmp($op, 'queue') == 0)
278 $where .= ' AND done = 0';
279
280 $r = rg_builder_load_jobs($db, $where);
281 if ($r['ok'] != 1)
282 return rg_template('builder/load_err.html', $rg, TRUE /*xss*/);
283
284 // Preparing the list for template
285 $d = array();
286 foreach ($r['list'] as $_id => &$i) {
287 rg_builder_cosmetic($db, $i);
288 if (empty($i['status'])) {
289 $i['HTML:status_list'] = 'n/a';
290 } else {
291 if (!empty($i['status']['packages'])) {
292 $_x = array();
293 $_x['packages'] = rg_xss_safe($i['status']['packages']);
294 $i['HTML:status_packages'] = rg_template('builder/packages.html', $_x, TRUE/*xss*/);
295 } else {
296 $i['HTML:status_packages'] = '';
297 }
298
299 $s = rg_builder_nice_status($i['status']);
300 $i['HTML:status_list'] = rg_template_table('builder/cmds',
301 $i['status']['cmds'], $rg);
302 }
303 $d[] = $i;
304 }
305
306 rg_log_ml('DEBUG: d: ' . print_r($d, TRUE));
307 return rg_template_table('builder/queue', $d, $rg);
308 }
309
File inc/util.inc.php changed (mode: 100644) (index 361104a..460a99a)
... ... function rg_valid_referer()
2121 2121 } }
2122 2122
2123 2123 /* /*
2124 * Returns the age of an object
2124 * Returns the human formatted time based on a number of seconds
2125 * @diff - time in seconds
2125 2126 */ */
2126 function rg_age($ts)
2127 function rg_human_time_interval($diff)
2127 2128 { {
2128 $diff = time() - $ts;
2129
2130 2129 $ret = ($diff % 60) . 's'; $ret = ($diff % 60) . 's';
2131 2130 $rest = intval($diff / 60); $rest = intval($diff / 60);
2132 2131 if ($rest == 0) if ($rest == 0)
 
... ... function rg_age($ts)
2152 2151 return $ret; return $ret;
2153 2152 } }
2154 2153
2154 /*
2155 * Returns the age of an object
2156 */
2157 function rg_age($ts)
2158 {
2159 $diff = time() - $ts;
2160 return rg_human_time_interval($diff);
2161 }
2162
2155 2163 /* /*
2156 2164 * Returns the path into the temporary area * Returns the path into the temporary area
2157 2165 */ */
 
... ... function rg_visible_string($a)
2412 2420 return $ret; return $ret;
2413 2421 } }
2414 2422
2415 ?>
2423 /*
2424 * Get latest bytes from a file
2425 * Note: bytes not chars
2426 */
2427 function rg_file_get_tail($f, $bytes)
2428 {
2429 $size = @filesize($f);
2430 if ($size === FALSE)
2431 return '';
2432
2433 if ($size <= $bytes)
2434 return @file_get_contents($f);
2435
2436 $r = @file_get_contents($f, FALSE, FALSE, -1, $bytes);
2437 if ($r === FALSE)
2438 return '';
2439
2440 $pos = strpos($r, "\n");
2441 if ($pos === FALSE)
2442 return $r;
2443
2444 return substr($r, $pos + 1);
2445 }
2446
2447 /*
2448 * Unserialize JSON or php 'serialize'
2449 */
2450 function rg_unserialize($s)
2451 {
2452 if (strncmp($s, '{', 1) == 0) {
2453 $r = @json_decode($s, TRUE);
2454 if ($r !== NULL)
2455 return $r;
2456 }
2457
2458 if (strncmp($s, 'a:', 2) == 0) {
2459 $r = @unserialize($s);
2460 if ($r !== FALSE)
2461 return $r;
2462 }
2463
2464 return FALSE;
2465 }
2466
2467 /*
2468 * Serialize data
2469 */
2470 function rg_serialize($a)
2471 {
2472 return @json_encode($a, TRUE);
2473 }
File inc/workers.inc.php changed (mode: 100644) (index 755a9f6..c342192)
1 1 <?php <?php
2 2 $INC = isset($INC) ? $INC : dirname(__FILE__); $INC = isset($INC) ? $INC : dirname(__FILE__);
3 3 require_once($INC . '/events.inc.php'); require_once($INC . '/events.inc.php');
4 require_once($INC . '/builder.inc.php');
4 5
5 6 $rg_worker_error = ''; $rg_worker_error = '';
6 7
 
... ... function rg_worker_high_level($db, &$rg, $paras)
660 661 $ret .= rg_worker_add_high_level($db, $rg, $op, $paras); $ret .= rg_worker_add_high_level($db, $rg, $op, $paras);
661 662 break; break;
662 663
664 case 'queue':
665 case 'queue_all':
666 $ret .= rg_builder_list_high_level($db, $rg, $op, $paras);
667 break;
668
663 669 default: default:
664 670 $ret .= rg_worker_list_high_level($db, $rg, $paras); $ret .= rg_worker_list_high_level($db, $rg, $paras);
665 671 break; break;
 
... ... function rg_worker_high_level($db, &$rg, $paras)
674 680 rg_prof_end('worker_high_level'); rg_prof_end('worker_high_level');
675 681 return $ret; return $ret;
676 682 } }
677
678 ?>
File root/themes/default/builder/cmds/footer.html copied from file root/themes/default/repo/bug/list/footer.html (similarity 100%)
File root/themes/default/builder/cmds/header.html added (mode: 100644) (index 0000000..dd2df1f)
1 <table>
2 <tr>
3 <th>Date added (UTC)</th>
4 <th>Done (UTC)</th>
5 <th>Elapsed time</th>
6 <th>Log</th>
7 </tr>
8
File root/themes/default/builder/cmds/line.html added (mode: 100644) (index 0000000..5ba0273)
1 <tr>
2 <td>@@start_nice@@</td>
3 <td>@@done_nice@@</td>
4 <td>@@elap@@</td>
5 <td><small>@@log_nlbr@@</small></td>
6 </tr>
File root/themes/default/builder/cmds/nodata.html copied from file root/themes/default/invalid_menu.html (similarity 60%) (mode: 100644) (index 269ac4f..d07767a)
1 1 <div class="mess warning"> <div class="mess warning">
2 Invalid menu.
2 No commands present.
3 3 </div> </div>
File root/themes/default/builder/load_err.html copied from file root/themes/default/invalid_menu.html (similarity 59%) (mode: 100644) (index 269ac4f..1dfc6b2)
1 1 <div class="mess warning"> <div class="mess warning">
2 Invalid menu.
2 Cannot load the queue.
3 3 </div> </div>
File root/themes/default/builder/packages.html added (mode: 100644) (index 0000000..f8aa35e)
1 <details>
2 <summary>Extra packages installation log</summary>
3 <small>
4 <pre>@@packages@@</pre>
5 </small>
6 </details>
File root/themes/default/builder/queue/footer.html added (mode: 100644) (index 0000000..e7c4ead)
1 </table>
2
3 <input type="submit" name="button" value="Delete selected jobs" />
4 </form>
File root/themes/default/builder/queue/header.html added (mode: 100644) (index 0000000..ef5cb8a)
1 <form method="post" action="/op/admin/ldap/list" class="form_table">
2 <input type="hidden" name="delete" value="1" />
3 <input type="hidden" name="token" value="@@rg_form_token@@" />
4
5 <table>
6 <tr>
7 <th>Select</th>
8 <th>ID</th>
9 <th>User</th>
10 <th>Repo</th>
11 <th>Priority</th>
12 <th>Date added (UTC)</th>
13 <th>Done (UTC)</th>
14 <th>Request & status</th>
15 </tr>
16
File root/themes/default/builder/queue/line.html added (mode: 100644) (index 0000000..02d1e56)
1 <tr>
2 <td><input type="checkbox" name="delete_list[@@id@@]" /></td>
3 <td>@@id@@</td>
4 <td>@@user@@</td>
5 <td>@@ri::name@@</td>
6 <td>@@prio@@</td>
7 <td>@@itime_nice@@</td>
8 <td>@@done_nice@@</td>
9 <td>
10 env=@@request::env@@
11 <br />
12 url=@@request::url@@
13 <br />
14 head=@@request::head@@
15 <br />
16 hook_id=@@request::hook_id@@
17 <br />
18
19 @@status_packages@@
20
21 <details>
22 <summary>Commands details</summary>
23 @@status_list@@
24 </details>
25 </td>
26 </tr>
File root/themes/default/builder/queue/nodata.html copied from file root/themes/default/admin/plans/list/nodata.html (similarity 57%) (mode: 100644) (index adc3941..abe6ca0)
1 1 <div class="mess warning"> <div class="mess warning">
2 No plans defined yet.
2 No build jobs in queue.
3 3 </div> </div>
File root/themes/default/user/settings/workers/menu.html changed (mode: 100644) (index 9c0f80a..a514f4b)
2 2 <ul> <ul>
3 3 <li@@if(@@menu::worker::list@@ == 1){{ class="selected"}}><a href="@@url@@/list">List</a></li> <li@@if(@@menu::worker::list@@ == 1){{ class="selected"}}><a href="@@url@@/list">List</a></li>
4 4 <li@@if(@@menu::worker::add@@ == 1){{ class="selected"}}><a href="@@url@@/add">Add</a></li> <li@@if(@@menu::worker::add@@ == 1){{ class="selected"}}><a href="@@url@@/add">Add</a></li>
5 <li@@if(@@menu::worker::queue@@ == 1){{ class="selected"}}><a href="@@url@@/queue">Pending jobs</a></li>
6 <li@@if(@@menu::worker::queue_all@@ == 1){{ class="selected"}}><a href="@@url@@/queue_all">All jobs</a></li>
5 7 </ul> </ul>
6 8 </div> </div>
File scripts/builder.php changed (mode: 100644) (index 40a46fc..de4f9bf)
3 3 // It takes care of build jobs. // It takes care of build jobs.
4 4 // It can be run on the same machine as the webserver. // It can be run on the same machine as the webserver.
5 5 error_reporting(E_ALL); error_reporting(E_ALL);
6 ini_set("track_errors", "On");
6 ini_set('track_errors', 'On');
7 7 set_time_limit(0); set_time_limit(0);
8 8
9 9 $_s = microtime(TRUE); $_s = microtime(TRUE);
10 10
11 require_once("/etc/rocketgit/config.php");
12
13 $INC = dirname(__FILE__) . "/../inc";
14 require_once($INC . "/init.inc.php");
15 require_once($INC . "/log.inc.php");
16 require_once($INC . "/sql.inc.php");
17 require_once($INC . "/struct.inc.php");
18 require_once($INC . "/events.inc.php");
19 require_once($INC . "/repo.inc.php");
20 require_once($INC . "/prof.inc.php");
21 require_once($INC . "/mr.inc.php");
22 require_once($INC . "/keys.inc.php");
23 require_once($INC . "/user.inc.php");
24 require_once($INC . "/bug.inc.php");
25 require_once($INC . "/fixes.inc.php");
26 require_once($INC . "/plan.inc.php");
27 require_once($INC . "/admin.inc.php");
28 require_once($INC . "/ver.php");
29 require_once($INC . "/builder.inc.php");
30 require_once($INC . "/conn.inc.php");
31 require_once($INC . "/workers.inc.php");
11 require_once('/etc/rocketgit/config.php');
12
13 $INC = dirname(__FILE__) . '/../inc';
14 require_once($INC . '/init.inc.php');
15 require_once($INC . '/log.inc.php');
16 require_once($INC . '/sql.inc.php');
17 require_once($INC . '/struct.inc.php');
18 require_once($INC . '/events.inc.php');
19 require_once($INC . '/repo.inc.php');
20 require_once($INC . '/prof.inc.php');
21 require_once($INC . '/mr.inc.php');
22 require_once($INC . '/keys.inc.php');
23 require_once($INC . '/user.inc.php');
24 require_once($INC . '/bug.inc.php');
25 require_once($INC . '/fixes.inc.php');
26 require_once($INC . '/plan.inc.php');
27 require_once($INC . '/admin.inc.php');
28 require_once($INC . '/ver.php');
29 require_once($INC . '/builder.inc.php');
30 require_once($INC . '/conn.inc.php');
31 require_once($INC . '/workers.inc.php');
32 32
33 33 if ($rg_builder_port == 0) if ($rg_builder_port == 0)
34 34 exit(0); exit(0);
 
... ... function xdestroy($key)
43 43 } }
44 44
45 45 /* /*
46 * Called when a new client connects
46 * Called when a new worker connects
47 47 */ */
48 48 function xnew($key, $arg) function xnew($key, $arg)
49 49 { {
 
... ... function xnew($key, $arg)
51 51 global $workers; global $workers;
52 52
53 53 $s = &$rg_conns[$key]; $s = &$rg_conns[$key];
54 $s['func_cmd'] = 'xdispatch';
54 $s['func_data'] = 'xdispatch';
55 55 $s['func_destroy'] = 'xdestroy'; $s['func_destroy'] = 'xdestroy';
56 56 $s['db'] = $arg; $s['db'] = $arg;
57 57 $s['auth'] = 0; $s['auth'] = 0;
 
... ... function xnew($key, $arg)
60 60 } }
61 61
62 62 /* /*
63 * Dispatch a command from a worker
63 * Dispatch a command from a worker (one json)
64 64 */ */
65 function xdispatch($key, $line)
65 function xdispatch_one($key, $data)
66 66 { {
67 67 global $rg_conns; global $rg_conns;
68 68 global $jobs; global $jobs;
69 69
70 rg_log('Dispatch[' . $key . ']');
70 rg_log($key . ': dispatching');
71 71
72 72 $s = &$rg_conns[$key]; $s = &$rg_conns[$key];
73 73
74 $cmd = substr($line, 0, 4);
75 $d = trim(substr($line, 4));
76 $x = stripcslashes($d);
77 $u = @unserialize($x);
78 if ($u === FALSE) {
79 rg_conn_enq($key, 'ERR malformed command' . "\n");
74 $u = @json_decode($data, TRUE);
75 if ($u === NULL) {
76 $err = array(
77 'errstr' => 'cannot decode JSON:' . json_last_error_msg()
78 );
79 rg_log_ml('Cannot decode JSON: ' . json_last_error_msg());
80 rg_conn_enq($key, json_encode($err));
80 81 rg_conn_shutdown($key, 2); rg_conn_shutdown($key, 2);
81 82 return; return;
82 83 } }
83 84
84 rg_log_ml('cmd=' . $cmd . ' u: ' . print_r($u, TRUE));
85 rg_log_ml('DEBUG: u: ' . print_r($u, TRUE));
85 86
86 if (strcmp($cmd, 'ANN ') == 0) {
87 if (strcmp($u['op'], 'ANN') == 0) {
87 88 $now = time(); $now = time();
88 89 if (($u['boot_time'] < $now - 30) || ($u['boot_time'] > $now + 30)) { if (($u['boot_time'] < $now - 30) || ($u['boot_time'] > $now + 30)) {
90 $err = array(
91 'errstr' => 'boot_time is too old; time desync or replay attack?'
92 );
89 93 rg_log('boot_time is too old; abort'); rg_log('boot_time is too old; abort');
90 rg_conn_enq($key, 'ERR time not in sync between worker'
91 . ' and server' . "\n");
94 rg_conn_enq($key, json_encode($err) . "\n");
92 95 rg_conn_shutdown($key, 2); rg_conn_shutdown($key, 2);
93 96 return; return;
94 97 } }
 
... ... function xdispatch($key, $line)
100 103 $worker_uid = 0; $worker_uid = 0;
101 104 } else if (!isset($u['user'])) { } else if (!isset($u['user'])) {
102 105 rg_log('user field is not present; abort'); rg_log('user field is not present; abort');
103 rg_conn_enq($key, 'ERR user not defined in conf file' . "\n");
106 $err = array('errstr' => 'user not defined in conf file');
107 rg_conn_enq($key, json_encode($err) . "\n");
104 108 rg_conn_shutdown($key, 2); rg_conn_shutdown($key, 2);
105 109 return; return;
106 110 } else { } else {
107 111 $w_ui = rg_user_info($s['db'], 0, $u['user'], ''); $w_ui = rg_user_info($s['db'], 0, $u['user'], '');
108 112 if ($w_ui['exists'] !== 1) { if ($w_ui['exists'] !== 1) {
109 113 rg_log('invalid user; abort'); rg_log('invalid user; abort');
110 rg_conn_enq($key, 'ERR invalid user' . "\n");
114 $err = array('errstr' => 'invalid user');
115 rg_conn_enq($key, json_encode($err) . "\n");
111 116 rg_conn_shutdown($key, 2); rg_conn_shutdown($key, 2);
112 117 return; return;
113 118 } }
 
... ... function xdispatch($key, $line)
120 125 if ($wi === -1) { if ($wi === -1) {
121 126 rg_log('cannot load worker info: ' rg_log('cannot load worker info: '
122 127 . rg_worker_error() . '; abort'); . rg_worker_error() . '; abort');
123 rg_conn_enq($key, 'ERR internal error' . "\n");
128 $err = array('errstr' => 'internal error');
129 rg_conn_enq($key, json_encode($err) . "\n");
124 130 rg_conn_shutdown($key, 2); rg_conn_shutdown($key, 2);
125 131 return; return;
126 132 } }
127 133 if ($wi === 0) { if ($wi === 0) {
128 134 rg_log('name [' . $u['name'] . '] not found; abort'); rg_log('name [' . $u['name'] . '] not found; abort');
129 $err = 'ERR builder name not found, add it in the web'
130 . ' interface!' . "\n";
131 rg_conn_enq($key, $err);
135 $err = array('errstr' => 'builder name not found, add it'
136 . ' in the web interface');
137 rg_conn_enq($key, json_encode($err) . "\n");
132 138 rg_conn_shutdown($key, 2); rg_conn_shutdown($key, 2);
133 139 return; return;
134 140 } }
 
... ... function xdispatch($key, $line)
137 143 if (strcmp($sign, $u['sign']) != 0) { if (strcmp($sign, $u['sign']) != 0) {
138 144 rg_log('signature is not ok [' . $sign . ']' rg_log('signature is not ok [' . $sign . ']'
139 145 . ' != [' . $u['sign'] . ']'); . ' != [' . $u['sign'] . ']');
140 rg_conn_enq($key, 'ERR wrong signature' . "\n");
146 $err = array('errstr' => 'wrong signature');
147 rg_conn_enq($key, json_encode($err) . "\n");
141 148 rg_conn_shutdown($key, 2); rg_conn_shutdown($key, 2);
142 149 return; return;
143 150 } }
 
... ... function xdispatch($key, $line)
157 164 $a['ip'] = rg_fix_ip($s['ip']); $a['ip'] = rg_fix_ip($s['ip']);
158 165 rg_worker_update($s['db'], $worker_uid, $wi['id'], $a); rg_worker_update($s['db'], $worker_uid, $wi['id'], $a);
159 166
160 rg_log($key . ':Peer [' . $u['name'] . '] announce');
167 rg_log($key . ': peer [' . $u['name'] . '] announce processed.');
161 168 return; return;
162 169 } }
163 170
164 171 if ($s['auth'] != 1) { if ($s['auth'] != 1) {
165 172 rg_log($key . ':Client not authenticated!'); rg_log($key . ':Client not authenticated!');
166 $a = array('error' => 'client not authenticated');
167 $cmd = 'ERR ' . rg_conn_prepare($a) . "\n";
168 rg_conn_enq($key, $cmd);
173 $err = array('errstr' => 'client not authenticated');
174 rg_conn_enq($key, json_encode($err) . "\n");
169 175 return; return;
170 176 } }
171 177
172 if (strcmp($cmd, 'STA ') == 0) {
178 if (strcmp($u['op'], 'STA') == 0) {
173 179 $jid = $u['id']; $jid = $u['id'];
174 180 $jobs[$jid]['worker'] = $key; $jobs[$jid]['worker'] = $key;
175 181 $jobs[$jid]['worker_name'] = $s['ann']['name']; $jobs[$jid]['worker_name'] = $s['ann']['name'];
 
... ... function xdispatch($key, $line)
179 185 return; return;
180 186 } }
181 187
182 if (strcmp($cmd, 'DON ') == 0) {
188 if (strcmp($u['op'], 'DON') == 0) {
183 189 $s['active_jobs'] -= 1; $s['active_jobs'] -= 1;
184 190
185 191 $jid = $u['id']; $jid = $u['id'];
 
... ... function xdispatch($key, $line)
195 201 $r = rg_builder_done($s['db'], $jobs[$jid], $u['status']); $r = rg_builder_done($s['db'], $jobs[$jid], $u['status']);
196 202 if ($r === TRUE) { if ($r === TRUE) {
197 203 unset($jobs[$jid]); unset($jobs[$jid]);
198 // Send DoneREceived - so client will delete the jobs
199 $a = array('id' => $job['id']);
200 $cmd = 'DRE ' . rg_conn_prepare($a) . "\n";
201 rg_conn_enq($key, $cmd);
204 // Send DoneREceived - so client will delete the job
205 $a = array('op' => 'DRE', 'id' => $job['id']);
206 rg_conn_enq($key, json_encode($a) . "\n");
202 207 } }
203 208 return; return;
204 209 } }
205 210
206 rg_log('Unknown command [' . $cmd . ']!');
211 rg_log('Unknown command [' . $u['op'] . ']!');
212 $err = array('errstr' => 'unknown op');
213 rg_conn_enq($key, json_encode($err) . "\n");
214 rg_conn_shutdown($key, 2);
215 }
216
217 /*
218 * Dispatch a command from a worker
219 */
220 function xdispatch($key, $data)
221 {
222 $ret = 0;
223 while (1) {
224 $pos = strpos($data, "\n");
225 if ($pos === FALSE)
226 return $ret;
227
228 $one = substr($data, 0, $pos);
229 xdispatch_one($key, $one);
230 $data = substr($data, $pos + 1);
231 $ret += $pos + 1;
232 }
233
234 return $ret;
207 235 } }
208 236
209 237 function rg_process_job($db, &$job) function rg_process_job($db, &$job)
 
... ... function rg_process_job($db, &$job)
281 309
282 310 // Send only what is really needed // Send only what is really needed
283 311 $job2 = array(); $job2 = array();
312 $job2['op'] = 'BLD';
284 313 $job2['cmds'] = $job['cmds']; $job2['cmds'] = $job['cmds'];
285 314 $job2['packages'] = $job['packages']; $job2['packages'] = $job['packages'];
286 315 $job2['hook_id'] = $job['hook_id']; $job2['hook_id'] = $job['hook_id'];
 
... ... function rg_process_job($db, &$job)
289 318 $job2['env'] = $job['env']; $job2['env'] = $job['env'];
290 319 $job2['id'] = $job['id']; $job2['id'] = $job['id'];
291 320
292 $cmd = 'BLD ' . rg_conn_prepare($job2) . "\n";
293 rg_conn_enq($key, $cmd);
321 rg_conn_enq($key, json_encode($job2) . "\n");
294 322 // TODO: get a confirmation? // TODO: get a confirmation?
295 323 $job['worker'] = $key; $job['worker'] = $key;
296 324 $job['worker_started'] = 0; $job['worker_started'] = 0;
 
... ... function rg_process_job($db, &$job)
311 339 } }
312 340
313 341
314 rg_prof_start("MAIN");
342 rg_prof_start('MAIN');
343
344 rg_log_set_file($rg_log_dir . '/builder.log');
345 rg_log_set_sid('000000'); // to spread the logs
315 346
316 rg_log_set_file($rg_log_dir . "/builder.log");
317 rg_log_set_sid("000000"); // to spread the logs
347 rg_lock_or_exit('builder.lock');
318 348
319 rg_log("Start (ver=$rocketgit_version)...");
349 rg_log('Start (ver=' . $rocketgit_version . ')...');
320 350
321 rg_sql_app("rg-builder");
351 rg_sql_app('rg-builder');
322 352 $db = rg_sql_open($rg_sql); $db = rg_sql_open($rg_sql);
323 353 if ($db === FALSE) { if ($db === FALSE) {
324 rg_internal_error("Cannot connect to database!");
354 rg_internal_error('Cannot connect to database!');
325 355 exit(1); exit(1);
326 356 } }
327 357
 
... ... socket_set_option($socket, SOL_SOCKET, SO_REUSEADDR, 1);
339 369
340 370 $r = @socket_bind($socket, $rg_builder_bind, $rg_builder_port); $r = @socket_bind($socket, $rg_builder_bind, $rg_builder_port);
341 371 if ($r === FALSE) { if ($r === FALSE) {
342 rg_internal_error("Cannot bind socket!");
372 rg_internal_error('Cannot bind socket!');
343 373 exit(1); exit(1);
344 374 } }
345 375
346 376 $r = @socket_listen($socket, 128); $r = @socket_listen($socket, 128);
347 377 if ($r === FALSE) { if ($r === FALSE) {
348 rg_internal_error("Cannot set queue length on socket!");
378 rg_internal_error('Cannot set queue length on socket!');
349 379 exit(1); exit(1);
350 380 } }
351 381
 
... ... do {
368 398 clearstatcache(); clearstatcache();
369 399 $mtime = @filemtime(__FILE__); $mtime = @filemtime(__FILE__);
370 400 if ($mtime != $original_mtime) { if ($mtime != $original_mtime) {
371 rg_log("mtime=$mtime, original_mtime=$original_mtime");
401 rg_log('mtime=' . $mtime . ', original_mtime=' . $original_mtime);
372 402 rg_log('File changed. Exiting...'); rg_log('File changed. Exiting...');
373 403 break; break;
374 404 } }
375 405
376 406 if ($workers > 0) { if ($workers > 0) {
377 $r = rg_builder_load_jobs($db);
407 $r = rg_builder_load_jobs($db, 'done = 0');
378 408 if ($r['ok'] != 1) { if ($r['ok'] != 1) {
379 409 rg_log('Cannot load jobs from database! Sleeping 30s...'); rg_log('Cannot load jobs from database! Sleeping 30s...');
380 410 sleep(30); sleep(30);
381 411 continue; continue;
382 412 } }
383 413
414 $_r = TRUE;
384 415 foreach ($r['list'] as $jid => $job) { foreach ($r['list'] as $jid => $job) {
385 416 if (!isset($jobs[$jid])) { if (!isset($jobs[$jid])) {
386 417 $job['worker'] = ''; $job['worker'] = '';
387 $job['avoid'] = array(); // to avoid workers
418 $job['avoid'] = array(); // to avoid some workers
388 419 $jobs[$jid] = $job; $jobs[$jid] = $job;
389 420 } }
390 421
391 $r = rg_process_job($db, $jobs[$jid]);
392 if ($r === FALSE)
422 $_r = rg_process_job($db, $jobs[$jid]);
423 if ($_r === FALSE)
393 424 break; break;
394 425 } }
395 if ($r === FALSE)
426 if ($_r === FALSE)
396 427 break; break;
397 428 } }
398 429
399 rg_log("Waiting for connections...");
400 430 rg_conn_wait(10); rg_conn_wait(10);
401 431 } while (1); } while (1);
402 432
403 433 @socket_close($socket); @socket_close($socket);
404 434
405 rg_log("Exiting...");
435 rg_log('Exiting...');
406 436
407 rg_prof_end("MAIN");
437 rg_prof_end('MAIN');
408 438 rg_prof_log(); rg_prof_log();
409 439 ?> ?>
File scripts/builder.sh copied from file scripts/events.sh (similarity 65%) (mode: 100755) (index be1ec0b..93fd506)
1 1 #!/bin/bash #!/bin/bash
2 2
3 # This is a wrapper for events.php, to not wait a lot after it exits
3 # This is a wrapper for builder.php, to not wait a lot after it exits
4 4
5 5 . /usr/share/rocketgit/scripts/common.sh . /usr/share/rocketgit/scripts/common.sh
6 6
7 7 check_context check_context
8 8
9 exec 100<>/var/lib/rocketgit/locks/events.sh.lock
9 exec 100<>/var/lib/rocketgit/locks/builder.sh.lock
10 10
11 11 flock --exclusive --nonblock 100 flock --exclusive --nonblock 100
12 12 if [ "${?}" != "0" ]; then if [ "${?}" != "0" ]; then
 
... ... if [ "${?}" != "0" ]; then
14 14 fi fi
15 15
16 16 while [ 1 ]; do while [ 1 ]; do
17 php /usr/share/rocketgit/scripts/events.php
17 php /usr/share/rocketgit/scripts/builder.php
18 18
19 19 # in case of errors, we will wait, to not go into an infinite loop # in case of errors, we will wait, to not go into an infinite loop
20 20 if [ "${?}" != "0" ]; then if [ "${?}" != "0" ]; then
File scripts/worker.TODO changed (mode: 100644) (index ab89528..0d3ded3)
1 [ ] Switch to pools of workers and dynamically attach disks.
1 2 <disk type="block" device="disk"> <disk type="block" device="disk">
2 3 <driver name="qemu" type="raw" cache="none" io="native" iothread="2"/> <driver name="qemu" type="raw" cache="none" io="native" iothread="2"/>
3 4 <source dev="/dev/xxx"/> <source dev="/dev/xxx"/>
 
8 9
9 10 [ ] Refuse jobs if the number of cpus/mem/etc. is too big for all workers. [ ] Refuse jobs if the number of cpus/mem/etc. is too big for all workers.
10 11 [ ] Clean images after build. [ ] Clean images after build.
11 [ ] Add templates to rocketgit git.
12 [ ] Add the templates to rocketgit git.
12 13 [ ] Add a secure channel for comunication between worker and server. [ ] Add a secure channel for comunication between worker and server.
13 [ ]
14 Done. Use it. For example, to detect if the VM started.
15 And, then, for streaming logs. Maybe for abrupt abort?
16 [ ] In case of errors, seems the error is not stored in Settings/Workers/List.
17 Instead, it seems that everything was ok. :(
18 [ ] Log repo/branch/etc. which triggered the webhook.
19 [ ] We need to be able to restart/reload_the_conf at any time.
20 Think about raising the number of machines in the pool.
21 So we need to have the current tasks in a fs queue.
22 [ ] Activate SELinux with a custom profile.
23 [ ] Activate AppArmor with a custom profile.
24 [ ] About resources availability: who is able to set them?
25 The web interface of the worker? I think the worker.
26 [ ] Log what project and what branch triggered the build.
27 [ ] Add notification channels to be able to notify if the build was ok/bad.
28 At least notify the project owner.
29 [ ] Take load in consideration?
30 [ ] Specify a pool size per 'env' section?
31 [ ] Specify a global max pool size?
32 [ ] When strting, load all $conf['state'] . '/job-*.ser' files and send a DON
33 command to the server.
34 [ ]
Hints:
Before first commit, do not forget to setup your git environment:
git config --global user.name "your_name_here"
git config --global user.email "your@email_here"

Clone this repository using HTTP(S):
git clone https://rocketgit.com/user/catalinux/rocketgit

Clone this repository using ssh (do not forget to upload a key first):
git clone ssh://rocketgit@ssh.rocketgit.com/user/catalinux/rocketgit

Clone this repository using git:
git clone git://git.rocketgit.com/user/catalinux/rocketgit

You are allowed to anonymously push to this repository.
This means that your pushed commits will automatically be transformed into a merge request:
... clone the repository ...
... make some changes and some commits ...
git push origin main