RocketGit

catalinux / rocketgit (public) (License: Affero GPLv3) (since 2015-03-19) (hash sha1)

Main RocketGit repo

Clone URLs: https://rocketgit.com/user/catalinux/rocketgit ssh://rocketgit@ssh.rocketgit.com/user/catalinux/rocketgit git://git.rocketgit.com/user/catalinux/rocketgit

List of commits:

Subject	Hash	Author	Date (UTC)
replaced 'slave' with 'part'	348a2afcaedd77c7513b5e7dd05364f5e9ae7a87	Catalin(ux) M. BOIE	2020-08-20 03:40:43
Make cron tasks more resilient in case the machine is stopped	afbe8607b4608862787bf696f71027beb345701f	Catalin(ux) M. BOIE	2020-08-19 06:08:34
Cosmetic	21956573027d83eb45913b6bf30eeb57e602aea4	Catalin(ux) M. BOIE	2020-08-19 06:07:45
Added time of upload for artifacts	fd33fe0aeef09f66d74f3ec1554312a2fa5e2642	Catalin(ux) M. BOIE	2020-08-19 06:06:54
css fixes for table scroll	77e486f3c9e3c9356985d3c21bc713d8a8d0e223	Catalin(ux) M. BOIE	2020-08-19 06:02:09
Added git repo size	9828b1fe96e2217ead8863f7a7c0ee3599d16633	Catalin(ux) M. BOIE	2020-08-19 06:01:27
Change artifacts path to be able to compute correct sizes	efff16c88415dfca0a4b52981b44620e52778c03	Catalin(ux) M. BOIE	2020-08-19 05:56:17
Some state was not under transaction	fd51b92442fb83fc6941f94b22236f5014f0062c	Catalin(ux) M. BOIE	2020-08-19 03:24:28
Added artifacts size to repos and users tables	2a8469e3416e63e9bb21ba0e412583fc69fea115	Catalin(ux) M. BOIE	2020-08-19 03:23:58
No need to invalidate last part ts because we already check	75cf0ae1721d9473bb8df780cc61f5c6247a7a6c	Catalin(ux) M. BOIE	2020-08-19 03:20:19
Fixed some spaces in profiling	929b95c5d842ffe306031f2ba99d23e435855155	Catalin(ux) M. BOIE	2020-08-18 07:11:30
Added a link to tutorials in the welcome mail	2ef5795e53a26fdc1f9a2adda7bbf7a4f4ccb81f	Catalin(ux) M. BOIE	2020-08-17 07:08:15
Change the mail goal	bffa2a1f9a4b185916b700b3e8455bbf6f7a82bf	Catalin(ux) M. BOIE	2020-08-17 06:59:35
One of the tests was not ran	bb295b59daee2158c1a9c435a267d4d2377d36e9	Catalin(ux) M. BOIE	2020-08-17 06:47:52
Doc improvement	039b77222a8fb6e38be0b9a35329c13f86c890af	Catalin(ux) M. BOIE	2020-08-17 06:47:27
Fixed form generation test	b1ca5d96c15271fbe8a595a22227d15f917779dc	Catalin(ux) M. BOIE	2020-08-17 06:46:56
Do not allow adding bugs if not logged in	32db254689f7419fcea479985bbeff5cc579f7e7	Catalin(ux) M. BOIE	2020-08-17 06:45:46
markdown format improvements	bc1cc389bc851af6d0fb186bcfaf47525e235b6b	Catalin(ux) M. BOIE	2020-08-17 06:45:01
Set the user's last IP also if she never logged in	556ff3e7df07863f977caac1cb25906f8c307583	Catalin(ux) M. BOIE	2020-08-17 06:44:03
Cosmetic	d2a744bd020154afa13961f9d3d3da4c51853ff4	Catalin(ux) M. BOIE	2020-08-17 06:42:23

Commit 348a2afcaedd77c7513b5e7dd05364f5e9ae7a87 - replaced 'slave' with 'part'
Author: Catalin(ux) M. BOIE
Author date (UTC): 2020-08-20 03:40
Committer name: Catalin(ux) M. BOIE
Committer date (UTC): 2020-08-20 03:40
Parent(s): afbe8607b4608862787bf696f71027beb345701f
Signing key:
Tree: ce65162fa668ffbfc2f2c9c28ee969b7971e9718

File	Lines added	Lines deleted
TODO	2	3
inc/struct.inc.php	32	32
scripts/cron.php	1	1
tests/common.php	2	2

File inc/struct.inc.php changed (mode: 100644) (index 6e0db53..891ff5e)
...	...	include_once($INC . "/util.inc.php");
6	6	define("RG_DROP_TABLES", 1);	define("RG_DROP_TABLES", 1);
7	7	define("RG_IGNORE_ERRORS", 1 << 1);	define("RG_IGNORE_ERRORS", 1 << 1);
8	8
9		$rg_sql_struct_slaves = array();
	9		$rg_sql_struct_parts = array();
10	10	$rg_sql_struct = array();	$rg_sql_struct = array();
11	11	$rg_sql_struct[1] = array();	$rg_sql_struct[1] = array();
12	12	$rg_sql_struct[1]['tables'] = array(	$rg_sql_struct[1]['tables'] = array(

...	...	$rg_sql_struct[10]['tables'] = array(
207	207	. ", message TEXT NOT NULL)"	. ", message TEXT NOT NULL)"
208	208	);	);
209	209	$rg_sql_struct[10]['other'] = array();	$rg_sql_struct[10]['other'] = array();
210		$rg_sql_struct_slaves['repo_history'] = array();
	210		$rg_sql_struct_parts['repo_history'] = array();
211	211
212	212	$rg_sql_struct[11] = array();	$rg_sql_struct[11] = array();
213	213	$rg_sql_struct[11]['tables'] = array(	$rg_sql_struct[11]['tables'] = array(

...	...	$rg_sql_struct[45]['other'] = array(
669	669	'worker_stats id' =>	'worker_stats id' =>
670	670	'CREATE INDEX worker_stats_i_worker_id on worker_stats(worker_id)'	'CREATE INDEX worker_stats_i_worker_id on worker_stats(worker_id)'
671	671	);	);
672		$rg_sql_struct_slaves['stats'] = array();
673		$rg_sql_struct_slaves['worker_stats'] = array('extra_index' => array('worker_id'));
674		$rg_sql_struct_slaves['conns'] = array();
	672		$rg_sql_struct_parts['stats'] = array();
	673		$rg_sql_struct_parts['worker_stats'] = array('extra_index' => array('worker_id'));
	674		$rg_sql_struct_parts['conns'] = array();
675	675	// Here, 0.72 was released.	// Here, 0.72 was released.
676	676
677	677	$rg_sql_struct[46] = array();	$rg_sql_struct[46] = array();

...	...	function rg_sql_struct_update($db, $flags)
865	865	}	}
866	866
867	867	/*	/*
868		* Update slaves (postgresql partitions)
	868		* Update partitions table
869	869	* Returns FALSE in case of error	* Returns FALSE in case of error
870	870	*/	*/
871		function rg_sql_struct_slaves_update($db)
	871		function rg_sql_struct_parts_update($db)
872	872	{	{
873		global $rg_sql_struct_slaves;
	873		global $rg_sql_struct_parts;
874	874
875		rg_prof_start('sql_struct_slaves_update');
876		rg_log_enter('sql_struct_slaves_update');
	875		rg_prof_start('sql_struct_parts_update');
	876		rg_log_enter('sql_struct_parts_update');
877	877
878	878	$ret = FALSE;	$ret = FALSE;
879	879	$doit = FALSE;	$doit = FALSE;
880	880	while (1) {	while (1) {
881		if (empty($rg_sql_struct_slaves)) {
	881		if (empty($rg_sql_struct_parts)) {
882	882	$ret = TRUE;	$ret = TRUE;
883	883	break;	break;
884	884	}	}
885	885
886		$last_list = rg_state_get($db, 'slaves_create_last_list');
	886		$last_list = rg_state_get($db, 'parts_create_last_list');
887	887	if ($last_list === FALSE)	if ($last_list === FALSE)
888	888	break;	break;
889	889
890		$last_ts = rg_state_get_uint($db, 'slaves_create_last_ts');
	890		$last_ts = rg_state_get_uint($db, 'parts_create_last_ts');
891	891	if ($last_ts === FALSE)	if ($last_ts === FALSE)
892	892	break;	break;
893	893	if ($last_ts == 0)	if ($last_ts == 0)
894	894	$last_ts = gmmktime(0, 0, 0, gmdate('m') - 1, 1, gmdate('Y'));	$last_ts = gmmktime(0, 0, 0, gmdate('m') - 1, 1, gmdate('Y'));
895	895	rg_log('DEBUG: last_ts=' . $last_ts . ' (' . gmdate('Y-m-d', $last_ts) . ')');	rg_log('DEBUG: last_ts=' . $last_ts . ' (' . gmdate('Y-m-d', $last_ts) . ')');
896	896
897		$cur_list = sha1(rg_serialize($rg_sql_struct_slaves));
	897		$cur_list = sha1(rg_serialize($rg_sql_struct_parts));
898	898	rg_log('DEBUG: last_list=[' . $last_list . '] cur_list=[' . $cur_list . ']');	rg_log('DEBUG: last_list=[' . $last_list . '] cur_list=[' . $cur_list . ']');
899	899	if (strcmp($last_list, $cur_list) != 0) {	if (strcmp($last_list, $cur_list) != 0) {
900	900	// We need to set last_ts to first day of the previous month	// We need to set last_ts to first day of the previous month
901	901	$last_ts = gmmktime(0, 0, 0, gmdate('m') - 1, 1, gmdate('Y'));	$last_ts = gmmktime(0, 0, 0, gmdate('m') - 1, 1, gmdate('Y'));
902		rg_log('DEBUG: list of the slaves changed. Set last_ts to ' . $last_ts);
	902		rg_log('DEBUG: list of the parts changed. Set last_ts to ' . $last_ts);
903	903	$doit = TRUE;	$doit = TRUE;
904	904	break;	break;
905	905	}	}

...	...	function rg_sql_struct_slaves_update($db)
926	926	$locked = FALSE;	$locked = FALSE;
927	927	while ($doit) {	while ($doit) {
928	928	// If we cannot lock, return error	// If we cannot lock, return error
929		if (rg_lock('slave_create.lock') === FALSE)
	929		if (rg_lock('parts_create.lock') === FALSE)
930	930	break;	break;
931	931	$locked = TRUE;	$locked = TRUE;
932	932

...	...	function rg_sql_struct_slaves_update($db)
946	946	$month++;	$month++;
947	947	$next_ts = gmmktime(0, 0, 0, $month, 1, $year);	$next_ts = gmmktime(0, 0, 0, $month, 1, $year);
948	948
949		foreach ($rg_sql_struct_slaves as $table => $ti) {
950		$slave_table = $table . '_' . gmdate('Y_m', $ts);
951		rg_log('Creating table ' . $slave_table);
	949		foreach ($rg_sql_struct_parts as $table => $ti) {
	950		$part_table = $table . '_' . gmdate('Y_m', $ts);
	951		rg_log('Creating table ' . $part_table);
952	952
953	953	// First, check if exists. It is possible that	// First, check if exists. It is possible that
954	954	// we did the update but we could not set the cache.	// we did the update but we could not set the cache.
955	955	$sql = 'SELECT 1 FROM pg_class'	$sql = 'SELECT 1 FROM pg_class'
956		. ' WHERE relname = \'' . $slave_table . '\'';
	956		. ' WHERE relname = \'' . $part_table . '\'';
957	957	$res = rg_sql_query($db, $sql);	$res = rg_sql_query($db, $sql);
958	958	if ($res === FALSE) {	if ($res === FALSE) {
959	959	$ok = FALSE;	$ok = FALSE;

...	...	function rg_sql_struct_slaves_update($db)
966	966	continue;	continue;
967	967	}	}
968	968
969		$sql = 'CREATE TABLE ' . $slave_table
	969		$sql = 'CREATE TABLE ' . $part_table
970	970	. ' (CHECK(itime >= ' . $ts . ' AND itime <= ' . ($next_ts - 1) . '))'	. ' (CHECK(itime >= ' . $ts . ' AND itime <= ' . ($next_ts - 1) . '))'
971	971	. ' INHERITS (' . $table . ')';	. ' INHERITS (' . $table . ')';
972	972	$res = rg_sql_query($db, $sql);	$res = rg_sql_query($db, $sql);

...	...	function rg_sql_struct_slaves_update($db)
976	976	}	}
977	977	rg_sql_free_result($res);	rg_sql_free_result($res);
978	978
979		$sql = 'CREATE INDEX ' . $slave_table . '_i_itime'
980		. ' ON ' . $slave_table
	979		$sql = 'CREATE INDEX ' . $part_table . '_i_itime'
	980		. ' ON ' . $part_table
981	981	. ' (itime)';	. ' (itime)';
982	982	$res = rg_sql_query($db, $sql);	$res = rg_sql_query($db, $sql);
983	983	if ($res === FALSE) {	if ($res === FALSE) {

...	...	function rg_sql_struct_slaves_update($db)
988	988
989	989	if (isset($ti['extra_index'])) {	if (isset($ti['extra_index'])) {
990	990	foreach ($ti['extra_index'] as $i) {	foreach ($ti['extra_index'] as $i) {
991		$sql = 'CREATE INDEX ' . $slave_table . '_i_' . $i
992		. ' ON ' . $slave_table
	991		$sql = 'CREATE INDEX ' . $part_table . '_i_' . $i
	992		. ' ON ' . $part_table
993	993	. ' (' . $i . ')';	. ' (' . $i . ')';
994	994	$res = rg_sql_query($db, $sql);	$res = rg_sql_query($db, $sql);
995	995	if ($res === FALSE) {	if ($res === FALSE) {

...	...	function rg_sql_struct_slaves_update($db)
1010	1010	if ($ok !== TRUE)	if ($ok !== TRUE)
1011	1011	break;	break;
1012	1012
1013		$r = rg_state_set($db, 'slaves_create_last_ts', $stop_ts);
	1013		$r = rg_state_set($db, 'parts_create_last_ts', $stop_ts);
1014	1014	if ($r !== TRUE) {	if ($r !== TRUE) {
1015		rg_log('Cannot set slaves_create_last_ts (' . rg_state_error() . ')');
	1015		rg_log('Cannot set parts_create_last_ts (' . rg_state_error() . ')');
1016	1016	break;	break;
1017	1017	}	}
1018	1018
1019		$r = rg_state_set($db, 'slaves_create_last_list', $cur_list);
	1019		$r = rg_state_set($db, 'parts_create_last_list', $cur_list);
1020	1020	if ($r !== TRUE) {	if ($r !== TRUE) {
1021		rg_log('Cannot set slaves_create_last_list (' . rg_state_error() . ')');
	1021		rg_log('Cannot set parts_create_last_list (' . rg_state_error() . ')');
1022	1022	break;	break;
1023	1023	}	}
1024	1024

...	...	function rg_sql_struct_slaves_update($db)
1034	1034	rg_sql_rollback($db);	rg_sql_rollback($db);
1035	1035
1036	1036	if ($locked === TRUE)	if ($locked === TRUE)
1037		rg_unlock('slave_create.lock');
	1037		rg_unlock('parts_create.lock');
1038	1038
1039	1039	rg_log_exit();	rg_log_exit();
1040		rg_prof_end('sql_struct_slaves_update');
	1040		rg_prof_end('sql_struct_parts_update');
1041	1041	return $ret;	return $ret;
1042	1042	}	}
1043	1043

...	...	function rg_struct_ok($db)
1070	1070	rg_cache_restart();	rg_cache_restart();
1071	1071	}	}
1072	1072
1073		$r = rg_sql_struct_slaves_update($db);
	1073		$r = rg_sql_struct_parts_update($db);
1074	1074	if ($r !== TRUE)	if ($r !== TRUE)
1075	1075	break;	break;
1076	1076

File scripts/cron.php changed (mode: 100644) (index 0a5a87e..e9e29e1)
...	...	$when = gmmktime(0, 0, 0, gmdate('m'), 1, gmdate('Y'));
212	212	while (($r < $today_00) && (rg_load() < 50)) {	while (($r < $today_00) && (rg_load() < 50)) {
213	213	rg_log_enter('Clean old part tables...');	rg_log_enter('Clean old part tables...');
214	214	$ok = TRUE;	$ok = TRUE;
215		foreach ($rg_sql_struct_slaves as $table => $ti) {
	215		foreach ($rg_sql_struct_parts as $table => $ti) {
216	216	// value is in 'months' = how many months we do want to keep the table	// value is in 'months' = how many months we do want to keep the table
217	217	$r = rg_state_get($db, 'part_clean_table_' . $table);	$r = rg_state_get($db, 'part_clean_table_' . $table);
218	218	if ($r === FALSE) {	if ($r === FALSE) {

File tests/common.php changed (mode: 100644) (index 163f603..eb72cf0)
...	...	if ($r !== TRUE) {
43	43	exit(1);	exit(1);
44	44	}	}
45	45
46		$r = rg_sql_struct_slaves_update($db);
	46		$r = rg_sql_struct_parts_update($db);
47	47	if ($r !== TRUE) {	if ($r !== TRUE) {
48		rg_log('Cannot create slaves: ' . rg_sql_error());
	48		rg_log('Cannot create parts: ' . rg_sql_error());
49	49	exit(1);	exit(1);
50	50	}	}
51	51

Hints:
Before first commit, do not forget to setup your git environment:

git config --global user.name "your_name_here"
git config --global user.email "your@email_here"

Clone this repository using HTTP(S):

git clone https://rocketgit.com/user/catalinux/rocketgit

Clone this repository using ssh (do not forget to upload a key first):

git clone ssh://rocketgit@ssh.rocketgit.com/user/catalinux/rocketgit

Clone this repository using git:

git clone git://git.rocketgit.com/user/catalinux/rocketgit

You are allowed to anonymously push to this repository.
This means that your pushed commits will automatically be transformed into a merge request:

... clone the repository ...
... make some changes and some commits ...
git push origin main

File TODO changed (mode: 100644) (index e90fc19..a1ad6b5)
88	88	machine reporting the load (see rg_stats_insert)!	machine reporting the load (see rg_stats_insert)!
89	89	[ ] @@if - if it is the only thing on a line, remove \n, else, do not.	[ ] @@if - if it is the only thing on a line, remove \n, else, do not.
90	90	[ ] Document worker proxy-tls connection type and nginx sample (check cloud.conf).	[ ] Document worker proxy-tls connection type and nginx sample (check cloud.conf).
91		[ ] Add admin Settings entries for slave_clean_table_* and for how much to keep
	91		[ ] Add admin Settings entries for parts_clean_table_* and for how much to keep
92	92	the logs.	the logs.
93	93	[ ] Add network statistics. Not clear how to obtain them easily.	[ ] Add network statistics. Not clear how to obtain them easily.
94	94	[ ] Discover: should we have the 'Public' column?	[ ] Discover: should we have the 'Public' column?

...	...	to add a worker? Also, no hint is presented.
1915	1915	[ ] Accessing a file with '"' inside, is not working.	[ ] Accessing a file with '"' inside, is not working.
1916	1916	See rocketgit.com/user/catalinux/test1/source/tree/blob/"xx\"yy"	See rocketgit.com/user/catalinux/test1/source/tree/blob/"xx\"yy"
1917	1917	[ ] Present bugs as a git repo, so we easily add a new one by push?	[ ] Present bugs as a git repo, so we easily add a new one by push?
1918		[ ] Configuration: a number of months to keep history (see slaves).
1919	1918	[ ] vagrant install?	[ ] vagrant install?
1920	1919	[ ] If I generated some activity on an object, do not notify myself.	[ ] If I generated some activity on an object, do not notify myself.
1921	1920	[ ] Show the age of a repo/user/bug/etc. Hm. This will prevent the caching.	[ ] Show the age of a repo/user/bug/etc. Hm. This will prevent the caching.

...	...	But, we have a problem with the expiration time!
2452	2451	[ ] Add a maximum time to keep logs.	[ ] Add a maximum time to keep logs.
2453	2452	[ ] Do not allow double ssh keys in database! Because when we output them	[ ] Do not allow double ssh keys in database! Because when we output them
2454	2453	in authorized_keys ssh will use the first one!	in authorized_keys ssh will use the first one!
2455		[ ] Move everything 1 month back to simulate next month for slave tables.
	2454		[ ] Move everything 1 month back to simulate next month for part tables.
2456	2455	Better, do a unit test.	Better, do a unit test.
2457	2456	[ ] To not wake up many times (for every sub-event), cache what was done	[ ] To not wake up many times (for every sub-event), cache what was done
2458	2457	(or max(id)) and ignore that wake-ups. Of course, W will become W<ev_id>.	(or max(id)) and ignore that wake-ups. Of course, W will become W<ev_id>.