RocketGit

catalinux / rocketgit (public) (License: Affero GPLv3) (since 2015-03-19) (hash sha1)

Main RocketGit repo

Clone URLs: https://rocketgit.com/user/catalinux/rocketgit ssh://rocketgit@ssh.rocketgit.com/user/catalinux/rocketgit git://git.rocketgit.com/user/catalinux/rocketgit

List of commits:

Subject	Hash	Author	Date (UTC)
worker improvements	4387b00291b5848aa08c303f9d62a126a6806a35	Catalin(ux) M. BOIE	2019-12-18 22:17:23
.spec: require tidy for building the tests	c0deab2ce24388c91d2bc2e96338a260c9657dd6	Catalin(ux) M. BOIE	2019-12-18 22:15:29
nginx configuration update	015c9404229f2ff7fb91cc57ab21289f01878334	Catalin(ux) M. BOIE	2019-12-18 21:55:13
Added Business model to comparison	c62d466580aca0ff3371f3ba369959c7ef7631d0	Catalin(ux) M. BOIE	2019-11-21 20:44:16
CSS small correction	ded9ef6bce5bebfa815bc7ae5733bc8e0810af3b	Catalin(ux) M. BOIE	2019-11-21 18:20:41
Added LDAP/qrencode to features/thanks pages	d269d59787f61045f5831c6985899c28da08ebeb	Catalin(ux) M. BOIE	2019-11-21 18:20:22
Invalidate slave table cache when updating the db structure, else new slaves will not be created	8e1bf7feeb87818179e7152a356e0f408da706f2	Catalin(ux) M. BOIE	2019-11-19 19:33:15
docker: push also the latest label	a72e2e74355d772ae45bb99798ad47082390943c	Catalin(ux) M. BOIE	2019-11-19 19:32:11
More statistics changes (load, cpu, mem)	e7f38e72405a227788a7011ea17dd57d0d4de67c	Catalin(ux) M. BOIE	2019-11-19 19:31:24
Cosmetic	0a0b258e8993cb5035e018b049018ac41b99eb01	Catalin(ux) M. BOIE	2019-11-17 11:09:56
Insist on respecting the privacy	b464bcc35476ffa23a83beaf956e8f7b15163d7c	Catalin(ux) M. BOIE	2019-11-17 11:03:36
Remove PayPal donation because of legal uncertainly	cd938af4c5cdbd81190e535973588dee6b697d34	Catalin(ux) M. BOIE	2019-11-17 11:03:07
nginx sample update	1f9eb2d293e91139f397cc45cb206d8994471134	Catalin(ux) M. BOIE	2019-11-17 09:10:02
Mostly cosmetic	3bf09056ede99df46c1b59802e6cf2c7bebf4f13	Catalin(ux) M. BOIE	2019-11-17 09:09:31
Added statistics for the installation	ed743a601cd187431ccd80b57113ba9bb5704035	Catalin(ux) M. BOIE	2019-11-17 09:08:33
ldap: if we have no server, it is useless to continue processing	c022b68a942f471e51cb8343bfef1cda390b27b1	Catalin(ux) M. BOIE	2019-11-16 06:30:47
Do not try to access a repo if is deleted	0a6b1bb704fbeaf60ba053ecc0c323a80f0e397c	Catalin(ux) M. BOIE	2019-11-15 19:49:31
Some git tweaks to deal with empty repositories	c42a22e41cd6d29862e13077e60374ceb7b0c946	Catalin(ux) M. BOIE	2019-11-14 03:30:12
Texts updates + cosmetic	70246eecb7b63460d28f20b5e2f3c9c1c50852ae	Catalin(ux) M. BOIE	2019-11-14 03:29:33
Docker updates	aa4cf359af2c7c1e20e4f33c478f2bbcd134f081	Catalin(ux) M. BOIE	2019-11-14 03:24:05

Commit 4387b00291b5848aa08c303f9d62a126a6806a35 - worker improvements
Author: Catalin(ux) M. BOIE
Author date (UTC): 2019-12-18 22:17
Committer name: Catalin(ux) M. BOIE
Committer date (UTC): 2019-12-18 22:17
Parent(s): c0deab2ce24388c91d2bc2e96338a260c9657dd6
Signer:
Signing key:
Signing status: N
Tree: e6ef6d0572c20ed27c42191f791be1601f3b4c38

File	Lines added	Lines deleted
scripts/worker.php	70	51

File scripts/worker.php changed (mode: 100644) (index 046554f..59692d0)
...	...	if (!isset($_SERVER['argv'][2]))
32	32	else	else
33	33	$conf_file = $_SERVER['argv'][2];	$conf_file = $_SERVER['argv'][2];
34	34
35		// TODO: use different files for different workers!
36	35	rg_log_set_file($rg_log_dir . '/worker-' . $name . '.log');	rg_log_set_file($rg_log_dir . '/worker-' . $name . '.log');
37	36	rg_log_set_sid("000000"); // to spread the logs	rg_log_set_sid("000000"); // to spread the logs
38	37

...	...	function start_worker($job)
127	126	global $php_errormsg;	global $php_errormsg;
128	127
129	128	$env = $conf['env'][$job['env']];	$env = $conf['env'][$job['env']];
130		//rg_log_ml('DEBUG: env: ' . print_r($env, TRUE));
	129		rg_log_ml('DEBUG: env: ' . print_r($env, TRUE));
131	130
132	131	$jid = $job['id'];	$jid = $job['id'];
133	132	$emain = escapeshellarg($job['main']);	$emain = escapeshellarg($job['main']);

...	...	function start_worker($job)
137	136	$master = escapeshellarg($env['image']);	$master = escapeshellarg($env['image']);
138	137	$img = escapeshellarg($job['main'] . '/image.qcow2');	$img = escapeshellarg($job['main'] . '/image.qcow2');
139	138	$img2 = escapeshellarg($job['main'] . '/image2.raw');	$img2 = escapeshellarg($job['main'] . '/image2.raw');
	139		// TODO: add bellow configuration to the web form
	140		if (!isset($job['disk_size_gib']))
	141		$job['disk_size_gib'] = '10';
	142		if (!isset($job['mem_mib']))
	143		$job['mem_mib'] = '500';
	144		if (!isset($job['cpus']))
	145		$job['cpus'] = '1';
140	146
141	147	$do_umount = FALSE;	$do_umount = FALSE;
142	148	$err = TRUE;	$err = TRUE;
	149		$reason = ''; $reason2 = '';
143	150	while (1) {	while (1) {
144	151	rg_exec('virsh destroy ' . $ename, '', FALSE, FALSE, FALSE);	rg_exec('virsh destroy ' . $ename, '', FALSE, FALSE, FALSE);
145		rg_exec('virsh undefine ' . $ename, '', FALSE, FALSE, FALSE);
	152		rg_exec('virsh undefine --nvram ' . $ename, '', FALSE, FALSE, FALSE);
146	153
147	154	$r = rg_del_tree($job['main']);	$r = rg_del_tree($job['main']);
148	155	if ($r === FALSE) {	if ($r === FALSE) {

...	...	function start_worker($job)
162	169	. escapeshellarg($job['main']), '', FALSE, FALSE, FALSE);	. escapeshellarg($job['main']), '', FALSE, FALSE, FALSE);
163	170	if ($r['ok'] !== 1) {	if ($r['ok'] !== 1) {
164	171	$reason = 'cannot chown build dir to qemu user: ' . $r['errmsg'];	$reason = 'cannot chown build dir to qemu user: ' . $r['errmsg'];
	172		$reason2 = $r['stderr'];
165	173	break;	break;
166	174	}	}
167	175
168		// Save & confirm the receiving (TODO: fsync)
169	176	// TODO: This will be used to clean up on a restart	// TODO: This will be used to clean up on a restart
170		$r = @file_put_contents($job['main'] . '/job.ser',
	177		$r = @file_put_contents($job['main'] . '/job.ser.tmp',
171	178	serialize($job));	serialize($job));
172	179	if ($r === FALSE) {	if ($r === FALSE) {
173	180	$reason = 'cannot store job';	$reason = 'cannot store job';
174	181	break;	break;
175	182	}	}
	183		$r = @rename($job['main'] . '/job.ser.tmp', $job['main'] . '/job.ser');
	184		if ($r === FALSE) {
	185		$reason = 'cannot rename job file';
	186		break;
	187		}
176	188
177		$r = rg_exec('qemu-img create -b ' . $master
	189		$r = rg_exec('qemu-img create -o lazy_refcounts=on,cluster_size=256K'
	190		. ' -b ' . $master
178	191	. ' -f qcow2 ' . $img, '', FALSE, FALSE, FALSE);	. ' -f qcow2 ' . $img, '', FALSE, FALSE, FALSE);
179	192	if ($r['ok'] !== 1) {	if ($r['ok'] !== 1) {
180	193	$reason = 'cannot create image: ' . $r['errmsg'];	$reason = 'cannot create image: ' . $r['errmsg'];
	194		$reason2 = $r['stderr'];
181	195	break;	break;
182	196	}	}
183	197
184		$r = rg_exec('qemu-img create -f raw ' . $img2 . ' 1G',
	198		// TODO: let the user specify the maximum disk space?
	199		$r = rg_exec('qemu-img create -f raw ' . $img2
	200		. ' ' . escapeshellarg($job['disk_size_gib'] . 'G'),
185	201	'', FALSE, FALSE, FALSE);	'', FALSE, FALSE, FALSE);
186	202	if ($r['ok'] !== 1) {	if ($r['ok'] !== 1) {
187	203	$reason = 'cannot create image2: ' . $r['errmsg'];	$reason = 'cannot create image2: ' . $r['errmsg'];
	204		$reason2 = $r['stderr'];
188	205	break;	break;
189	206	}	}
190	207
191		// Seems that mkfs is not in PATH when it is runned from cron
	208		// Seems that mkfs is not in PATH when we are running from cron.
192	209	$path = getenv('PATH');	$path = getenv('PATH');
193	210	putenv('PATH=' . $path . ':/usr/sbin');	putenv('PATH=' . $path . ':/usr/sbin');
194	211
195	212	$r = rg_exec('mkfs.ext4 -L RG ' . $img2, '', FALSE, FALSE, FALSE);	$r = rg_exec('mkfs.ext4 -L RG ' . $img2, '', FALSE, FALSE, FALSE);
196	213	if ($r['ok'] !== 1) {	if ($r['ok'] !== 1) {
197	214	$reason = 'cannot create fs: ' . $r['errmsg'];	$reason = 'cannot create fs: ' . $r['errmsg'];
	215		$reason2 = $r['stderr'];
198	216	break;	break;
199	217	}	}
200	218

...	...	function start_worker($job)
208	226	'', FALSE, FALSE, FALSE);	'', FALSE, FALSE, FALSE);
209	227	if ($r['ok'] !== 1) {	if ($r['ok'] !== 1) {
210	228	$reason = 'cannot mount fs: ' . $r['errmsg'];	$reason = 'cannot mount fs: ' . $r['errmsg'];
	229		$reason2 = $r['stderr'];
211	230	break;	break;
212	231	}	}
213	232	$do_umount = TRUE;	$do_umount = TRUE;

...	...	function start_worker($job)
222	241	rg_log('DEBUG: env: ' . $_env);	rg_log('DEBUG: env: ' . $_env);
223	242	putenv($_env);	putenv($_env);
224	243	$cmd = ' git clone --depth 1'	$cmd = ' git clone --depth 1'
	244		. ' --branch ' . escapeshellarg($job['head'])
	245		. ' --single-branch --recurse-submodules'
	246		. ' --shallow-submodules'
225	247	. ' ' . escapeshellarg($job['url'])	. ' ' . escapeshellarg($job['url'])
226	248	. ' ' . $emain . '/root/git';	. ' ' . $emain . '/root/git';
227	249	$r = rg_exec($cmd, '', FALSE, FALSE, FALSE);	$r = rg_exec($cmd, '', FALSE, FALSE, FALSE);
228	250	if ($r['ok'] !== 1) {	if ($r['ok'] !== 1) {
229	251	$reason = 'cannot clone: ' . $r['errmsg'];	$reason = 'cannot clone: ' . $r['errmsg'];
	252		$reason2 = $r['stderr'];
230	253	break;	break;
231	254	}	}
232	255

...	...	function start_worker($job)
327	350	$r = rg_exec('umount ' . $emain . '/root', '', FALSE, FALSE, FALSE);	$r = rg_exec('umount ' . $emain . '/root', '', FALSE, FALSE, FALSE);
328	351	if ($r['ok'] !== 1) {	if ($r['ok'] !== 1) {
329	352	$reason = 'cannot umount fs: ' . $r['errmsg'];	$reason = 'cannot umount fs: ' . $r['errmsg'];
	353		$reason2 = $r['stderr'];
330	354	break;	break;
331	355	}	}
332	356	$do_umount = FALSE;	$do_umount = FALSE;

...	...	function start_worker($job)
335	359	$r = rg_exec('chown qemu:qemu ' . $img2, '', FALSE, FALSE, FALSE);	$r = rg_exec('chown qemu:qemu ' . $img2, '', FALSE, FALSE, FALSE);
336	360	if ($r['ok'] !== 1) {	if ($r['ok'] !== 1) {
337	361	$reason = 'cannot chown image to qemu user: ' . $r['errmsg'];	$reason = 'cannot chown image to qemu user: ' . $r['errmsg'];
	362		$reason2 = $r['stderr'];
	363		break;
	364		}
	365
	366		// TODO: store the path to the templates in the configuration file.
	367		$template = @file_get_contents('/var/lib/libvirt/images/rgw/templates/'
	368		. escapeshellarg($env['arch']) . '.xml');
	369		if ($template === FALSE) {
	370		$reason = 'cannot load template';
	371		break;
	372		}
	373		str_replace('@@name@@', $name, $template);
	374		str_replace('@@mem@@', $job['mem_mib'], $template);
	375		str_replace('@@cpus@@', $job['cpus'], $template);
	376		str_replace('@@disk0@@', $img, $template);
	377		str_replace('@@disk1@@', $img2, $template);
	378		// TODO: allow firewall specification
	379		str_replace('@@net0@@', '<interface type=\'network\'><source network=\'default\'/><model type=\'virtio\'/></interface>', $template);
	380		// TODO: take care of XML injection?
	381		str_replace('@@chan1_path@@', $job['main'] . '/x.chan', $template);
	382		$r = @file_put_contents($job['main'] . '/machine.xml');
	383		if ($r === FALSE) {
	384		$reason = 'cannot store template file.';
338	385	break;	break;
339	386	}	}
340	387
341		// . ' --noautoconsole'
342		// . ' --security type=dynamic,relabel=yes'
343		// . ' --filesystem source=' . $emain . '/root')
344		// . ',target=rg,mode=mapped' // passthrough
345		// . ' --security type=static,label=' . $context . ',relabel=yes'
346		$r = rg_exec('virt-install'
347		. ' --name ' . $ename
348		. ' --arch ' . escapeshellarg($env['arch'])
349		. ' --memory 400'
350		. ' --vcpus 1'
351		. ' --graphics none'
352		. ' --network network=default'
353		. ' --security type=dynamic'
354		. ' --os-variant ' . escapeshellarg($env['os-variant'])
355		. ' --import'
356		. ' --disk path=' . $img . ',discard=unmap'
357		. ' --disk path=' . $img2 . ',discard=unmap'
358		. ' --rng /dev/random'
359		. ' --memballoon virtio'
360		. ' --console pty,target_type=virtio'
361		. ' ' . $env['paras'], '', FALSE, FALSE, FALSE);
	388		$r = rg_exec('virsh define ' . escapeshellarg($job['main'] . '/machine.xml'));
362	389	if ($r['ok'] !== 1) {	if ($r['ok'] !== 1) {
363		$reason = 'cannot define and start virtual machine: '
364		. $r['errmsg'];
	390		$reason = 'cannot define machine: ' . $r['errmsg'];
	391		$reason2 = $r['stderr'];
	392		break;
	393		}
	394
	395		$r = rg_exec('virsh start ' . $ename);
	396		if ($r['ok'] !== 1) {
	397		$reason = 'cannot start machine: ' . $r['errmsg'];
	398		$reason2 = $r['stderr'];
365	399	break;	break;
366	400	}	}
367	401

...	...	function start_worker($job)
372	406	rg_exec('umount ' . $emain . '/root', '', FALSE, FALSE, FALSE);	rg_exec('umount ' . $emain . '/root', '', FALSE, FALSE, FALSE);
373	407
374	408	// Seems that any error above must retrigger the build on other worker	// Seems that any error above must retrigger the build on other worker
375		if ($err)
	409		if ($err) {
376	410	@file_put_contents($job['main'] . '/error.log', $reason);	@file_put_contents($job['main'] . '/error.log', $reason);
	411		@file_put_contents($job['main'] . '/error2.log', $reason2);
	412		}
377	413	}	}
378	414
379	415	/*	/*

...	...	if ($r === FALSE) {
550	586	. $conf['port'] . '!');	. $conf['port'] . '!');
551	587	exit(1);	exit(1);
552	588	}	}
	589		rg_log('Connected.');
553	590
554	591	rg_conn_new('master', $socket);	rg_conn_new('master', $socket);
555	592	$rg_conns['master']['exit_on_close'] = 1;	$rg_conns['master']['exit_on_close'] = 1;

...	...	while(1) {
628	665	unset($xjob['main']);	unset($xjob['main']);
629	666	rg_conn_enq('master', 'DON ' . rg_conn_prepare($xjob) . "\n");	rg_conn_enq('master', 'DON ' . rg_conn_prepare($xjob) . "\n");
630	667
631		// TODO: do we destroy the pool in case of crash?
632		$cmd = 'virsh pool-destroy rocketgit-j-' . $jid;
633		$r = rg_exec($cmd, '', FALSE, FALSE, FALSE);
634		if ($r['ok'] != 1) {
635		$job['error'] = 'Could not destroy pool: ' . $r['errmsg'];
636		rg_log('Error: ' . $job['error']);
637		//break; TODO: do we need to do this?!
638		}
639
640		// TODO: do we clean the pool in case of crash?
641		// $cmd = 'virsh pool-undefine rocketgit-j-' . $jid;
642		// $r = rg_exec($cmd, '', FALSE, FALSE, FALSE);
643		// if ($r['ok'] != 1) {
644		// $job['error'] = 'Could not undefine pool: ' . $r['errmsg'];
645		// rg_log('Error: ' . $job['error']);
646		// //break; TODO: do we need to do this?!
647		// }
648
649	668	// // TODO: do we clean the machine in case of crash?	// // TODO: do we clean the machine in case of crash?
650		// $cmd = 'virsh undefine rg-worker-' . escapeshellarg($jid);
	669		// $cmd = 'virsh undefine --nvram rg-worker-' . escapeshellarg($jid);
651	670	// $r = rg_exec($cmd, '', FALSE, FALSE, FALSE);	// $r = rg_exec($cmd, '', FALSE, FALSE, FALSE);
652	671	// if ($r['ok'] != 1) {	// if ($r['ok'] != 1) {
653	672	// $job['error'] = 'Could not undefine machine: ' . $r['errmsg'];	// $job['error'] = 'Could not undefine machine: ' . $r['errmsg'];

Hints:
Before first commit, do not forget to setup your git environment:

git config --global user.name "your_name_here"
git config --global user.email "your@email_here"

Clone this repository using HTTP(S):

git clone https://rocketgit.com/user/catalinux/rocketgit

Clone this repository using ssh (do not forget to upload a key first):

git clone ssh://rocketgit@ssh.rocketgit.com/user/catalinux/rocketgit

Clone this repository using git:

git clone git://git.rocketgit.com/user/catalinux/rocketgit

You are allowed to anonymously push to this repository.
This means that your pushed commits will automatically be transformed into a merge request:

... clone the repository ...
... make some changes and some commits ...
git push origin main