catalinux / rocketgit (public) (License: Affero GPLv3) (since 2015-03-19) (hash sha1)
Main RocketGit repo
Clone URLs: https://rocketgit.com/user/catalinux/rocketgit ssh://rocketgit@ssh.rocketgit.com/user/catalinux/rocketgit git://git.rocketgit.com/user/catalinux/rocketgit
v0.0 v0.1 v0.10 v0.11 v0.12 v0.13 v0.14 v0.15 v0.17 v0.18 v0.19 v0.2 v0.20 v0.21 v0.22 v0.23 v0.24 v0.25 v0.27 v0.28 v0.29 v0.31 v0.32 v0.33 v0.34 v0.35 v0.36 v0.38 v0.39 v0.40 v0.41 v0.42 v0.43 v0.44 v0.45 v0.46 v0.47 v0.48 v0.49 v0.50 v0.51 v0.52 v0.53 v0.54 v0.55 v0.56 v0.57 v0.58 v0.60 v0.61 v0.62 v0.63 v0.64 v0.65 v0.66 v0.67 v0.68 v0.69 v0.70 v0.71 v0.72 v0.73 v0.77 v0.78 v0.79 v0.80 group1/branch1 group2/subgroup1/branch2 main master release
List of commits:
Subject Hash Author Date (UTC)
Allow cron to execute sub-tasks in parallel - we missed stats f65c76714cf349a53f584c8e65d9e89c24a4fa69 Catalin(ux) M. BOIE 2020-06-12 06:50:40
bug: search: fixed the override of some global variables 82e251ea883e50b8d1476530e0268fd5770fbf41 Catalin(ux) M. BOIE 2020-06-10 21:29:03
If a repo is non-existent, we should not try to use the id bcb08f74e21556aa750920335ebec2f965e72f36 Catalin(ux) M. BOIE 2020-06-10 08:05:43
If no activity, average elap is not computed correctly a7d4f0042c912d29f7943e0446820a477f42d6f7 Catalin(ux) M. BOIE 2020-06-09 07:03:21
When a user is requesting an e-mail confirmation operation, ignore if the e-mail was confirmed 62c387ed431d0c46d16e534cc1c954557ee389f7 Catalin(ux) M. BOIE 2020-06-08 06:42:22
Reverse test messed the creation of partition tables; more debugging d820816c2624549e0b0e1634523717cda0b7ba66 Catalin(ux) M. BOIE 2020-06-05 17:33:08
We need to prepare the partition tables before some inserts 07a12d1c79104b4707a9d43463fa9c11a4697293 Catalin(ux) M. BOIE 2020-06-05 15:55:43
.spec updates 5fe0aca1b4a3bfd8c1c8db95409faf512ad09b43 Catalin(ux) M. BOIE 2020-06-04 22:03:57
Switch to Podman (because of Fedora 32) and fix the worker service in the spec file 05b3866cd09b0aa8367d3b5755f45ea7274658de Catalin(ux) M. BOIE 2020-06-04 15:50:41
Added sample configuration for workers df36ccd20e40987a4ca2355062ee1312aa2ee3a2 Catalin(ux) M. BOIE 2020-06-04 15:14:52
Avoid complains from install for systemd daemon-reload 683a92542cc0608293ceefc111a05e1eb328a2d8 Catalin(ux) M. BOIE 2020-06-04 15:09:57
Small stuff e64a838d032e54587ab3110ab8bb04a0cde78231 Catalin(ux) M. BOIE 2020-06-04 14:49:56
Lots of changes b8164d9e1e263eb4db75e32430f0753b63d0028e Catalin(ux) M. BOIE 2020-06-04 07:14:35
Lots of small fixes and functional tests ad2c8f4ce214896cb5984031dbd916467878ccfd Catalin(ux) M. BOIE 2020-05-16 09:30:47
Lots of things, mostly worker fixes 8768227306ef89ecc51d392cf986e75a7044dde4 Catalin(ux) M. BOIE 2020-04-10 15:27:11
Added functional test for HTTP 304 code 93205ec14d1073c434360abf7ea525a75407b1ab Catalin(ux) M. BOIE 2020-04-02 14:05:49
Lots of changes all over the place 3ddbd9e686f514eed1e715d7702a910ca3f8b22c Catalin(ux) M. BOIE 2020-04-01 07:17:57
More tests updates d5d45ace725f6ab336998a328136a2f7f9474df1 Catalin(ux) M. BOIE 2020-04-01 05:30:54
Lots of tests updates 183c4af52e56ae4e4fd74a35e2b17ac79952aec6 Catalin(ux) M. BOIE 2020-03-20 16:10:33
Improved a little bit the mail reporting for admin 576f96d554b1e273391ac54404bf5f885325e5e8 Catalin(ux) M. BOIE 2020-03-03 19:25:17
Commit f65c76714cf349a53f584c8e65d9e89c24a4fa69 - Allow cron to execute sub-tasks in parallel - we missed stats
Author: Catalin(ux) M. BOIE
Author date (UTC): 2020-06-12 06:50
Committer name: Catalin(ux) M. BOIE
Committer date (UTC): 2020-06-12 06:50
Parent(s): 82e251ea883e50b8d1476530e0268fd5770fbf41
Signer:
Signing key:
Signing status: N
Tree: d8c3691b60148cb0189fa76f5f154544cd58a5a0
File Lines added Lines deleted
scripts/cron.php 85 64
File scripts/cron.php changed (mode: 100644) (index 128f52a..7984b1e)
1 1 <?php <?php
2 2 // This is called by cron // This is called by cron
3 3 error_reporting(E_ALL); error_reporting(E_ALL);
4 ini_set("track_errors", "On");
4 ini_set('track_errors', 'On');
5 5 set_time_limit(0); set_time_limit(0);
6 6
7 require_once("/etc/rocketgit/config.php");
8
9 $INC = dirname(__FILE__) . "/../inc";
10 require_once($INC . "/init.inc.php");
11 require_once($INC . "/log.inc.php");
12 require_once($INC . "/sql.inc.php");
13 require_once($INC . "/struct.inc.php");
14 require_once($INC . "/repo.inc.php");
15 require_once($INC . "/keys.inc.php");
16 require_once($INC . "/fixes.inc.php");
17 require_once($INC . "/mr.inc.php");
18 require_once($INC . "/admin.inc.php");
19 require_once($INC . "/stats.inc.php");
20 require_once($INC . "/ver.php");
7 require_once('/etc/rocketgit/config.php');
8
9 $INC = dirname(__FILE__) . '/../inc';
10 require_once($INC . '/init.inc.php');
11 require_once($INC . '/log.inc.php');
12 require_once($INC . '/sql.inc.php');
13 require_once($INC . '/struct.inc.php');
14 require_once($INC . '/repo.inc.php');
15 require_once($INC . '/keys.inc.php');
16 require_once($INC . '/fixes.inc.php');
17 require_once($INC . '/mr.inc.php');
18 require_once($INC . '/admin.inc.php');
19 require_once($INC . '/stats.inc.php');
20 require_once($INC . '/ver.php');
21 21
22 22 $now = time(); $now = time();
23 23
24 rg_log_set_file($rg_log_dir . "/cron.log");
25 rg_log_set_sid("000000"); // to spread the logs
24 rg_log_set_file($rg_log_dir . '/cron.log');
25 rg_log_set_sid('000000'); // to spread the logs
26 26
27 27 // locking // locking
28 rg_lock_or_exit("cron.lock");
28 rg_lock_or_exit('cron.lock');
29 29
30 rg_log("Start (ver=$rocketgit_version)...");
30 rg_log('Start (ver=' . $rocketgit_version . ')...');
31 31
32 if (gmdate("Hi") == "0305")
33 rg_clean_logs('/var/log/rocketgit');
34
35 rg_sql_app("rg-cron");
32 rg_sql_app('rg-cron');
36 33 $db = rg_sql_open($rg_sql); $db = rg_sql_open($rg_sql);
37 34 if ($db === FALSE) { if ($db === FALSE) {
38 rg_log("Cannot connect to database (" . rg_sql_error() . ")!");
35 rg_log('Cannot connect to database (' . rg_sql_error() . ')!');
39 36 // TODO: inform admin - already by e-mail? // TODO: inform admin - already by e-mail?
40 37 exit(1); exit(1);
41 38 } }
 
... ... $rg['base_url'] = rg_base_url();
61 58 rg_log('DEBUG: base_url=' . rg_base_url()); rg_log('DEBUG: base_url=' . rg_base_url());
62 59 $rg['debug'] = rg_state_get($db, 'debug'); $rg['debug'] = rg_state_get($db, 'debug');
63 60
61 // From now on, we give up the lock because some tasks take a long time to run.
62 rg_unlock('cron.lock');
64 63
65 64 rg_stats_insert($db); rg_stats_insert($db);
66 65
67 if (gmdate("Hi") == "0105") {
66 if (gmdate('Hi') == '0105') {
68 67 while (1) { while (1) {
69 68 if (rg_load() > 100) if (rg_load() > 100)
70 69 break; break;
71 70
72 rg_log("Compute repository sizes if dirty...");
71 rg_log('Compute repository sizes if dirty...');
73 72 // delete 'dirty' files // delete 'dirty' files
74 73 $sql = 'SELECT uid, repo_id, master, disk_used_mb FROM repos' $sql = 'SELECT uid, repo_id, master, disk_used_mb FROM repos'
75 74 . ' WHERE deleted = 0'; . ' WHERE deleted = 0';
76 75 $res = rg_sql_query($db, $sql); $res = rg_sql_query($db, $sql);
77 76 if ($res === FALSE) { if ($res === FALSE) {
78 77 // TODO: rg_internal_error? it must notify me in case of problems // TODO: rg_internal_error? it must notify me in case of problems
79 rg_log("Cannot run query (" . rg_sql_error() . ")!");
78 rg_log('Cannot run query (' . rg_sql_error() . ')!');
80 79 break; break;
81 80 } }
82 81
83 82 while (($row = rg_sql_fetch_array($res))) { while (($row = rg_sql_fetch_array($res))) {
84 rg_log("Processing repo " . $row['repo_id'] . "...");
83 rg_log('Processing repo ' . $row['repo_id'] . '...');
85 84 $repo_path = rg_repo_path_by_id($row['uid'], $row['repo_id']); $repo_path = rg_repo_path_by_id($row['uid'], $row['repo_id']);
86 85
87 86 $all_files = $row['master'] == 0 ? TRUE : FALSE; $all_files = $row['master'] == 0 ? TRUE : FALSE;
88 87 $disk_used = rg_repo_size($repo_path, $all_files); $disk_used = rg_repo_size($repo_path, $all_files);
89 88 if ($disk_used === FALSE) { if ($disk_used === FALSE) {
90 rg_log("Cannot compute the repo size: " . rg_repo_error());
89 rg_log('Cannot compute the repo size: ' . rg_repo_error());
91 90 continue; continue;
92 91 } }
93 $disk_used_mb = sprintf("%u", $disk_used / 1024 / 1024);
92 $disk_used_mb = sprintf('%u', $disk_used / 1024 / 1024);
94 93 if ($disk_used_mb != $row['disk_used_mb']) { if ($disk_used_mb != $row['disk_used_mb']) {
95 $sql = "UPDATE repos SET disk_used_mb = $disk_used_mb"
96 . " WHERE repo_id = " . $row['repo_id'];
94 $sql = 'UPDATE repos SET disk_used_mb = ' . $disk_used_mb
95 . ' WHERE repo_id = ' . $row['repo_id']
96 . ' AND disk_used_mb != ' . $disk_used_mb;
97 97 $res2 = rg_sql_query($db, $sql); $res2 = rg_sql_query($db, $sql);
98 98 if ($res2 === FALSE) { if ($res2 === FALSE) {
99 rg_log("Cannot run query!");
99 rg_log('Cannot run query!');
100 100 break; break;
101 101 } }
102 102 rg_sql_free_result($res2); rg_sql_free_result($res2);
 
... ... if (gmdate("Hi") == "0105") {
107 107 } }
108 108
109 109 while (1) { while (1) {
110 rg_log("Compute repository sizes per user...");
111 $sql = "SELECT SUM(disk_used_mb) AS disk_used_mb, uid"
112 . " FROM repos"
113 . " GROUP BY uid";
110 rg_log('Compute repository sizes per user...');
111 $sql = 'SELECT SUM(disk_used_mb) AS disk_used_mb, uid'
112 . ' FROM repos'
113 . ' GROUP BY uid';
114 114 $res = rg_sql_query($db, $sql); $res = rg_sql_query($db, $sql);
115 115 if ($res === FALSE) { if ($res === FALSE) {
116 rg_log("Cannot run query (" . rg_sql_error() . ")!");
116 rg_log('Cannot run query (' . rg_sql_error() . ')!');
117 117 break; break;
118 118 } }
119 119
120 120 while (($row = rg_sql_fetch_array($res))) { while (($row = rg_sql_fetch_array($res))) {
121 $sql = "UPDATE users"
122 . " SET disk_used_mb = " . $row['disk_used_mb']
123 . " WHERE uid = " . $row['uid']
124 . " AND disk_used_mb != " . $row['disk_used_mb'];
121 $sql = 'UPDATE users'
122 . ' SET disk_used_mb = ' . $row['disk_used_mb']
123 . ' WHERE uid = ' . $row['uid']
124 . ' AND disk_used_mb != ' . $row['disk_used_mb'];
125 125 $res2 = rg_sql_query($db, $sql); $res2 = rg_sql_query($db, $sql);
126 126 if ($res2 !== FALSE) if ($res2 !== FALSE)
127 127 rg_sql_free_result($res2); rg_sql_free_result($res2);
 
... ... if (gmdate("Hi") == "0105") {
131 131 } }
132 132 } }
133 133
134 if (gmdate("Hi") == "0300") {
135 rg_log_enter("Clean old forget_pass entries...");
136 $sql = "DELETE FROM forgot_pass WHERE expire < $now";
134 if (gmdate('Hi') == '0300') {
135 rg_log_enter('Clean old forget_pass entries...');
136 $sql = 'DELETE FROM forgot_pass WHERE expire < ' . $now;
137 137 $res = rg_sql_query($db, $sql); $res = rg_sql_query($db, $sql);
138 138 if ($res !== FALSE) if ($res !== FALSE)
139 139 rg_sql_free_result($res); rg_sql_free_result($res);
140 140 rg_log_exit(); rg_log_exit();
141 141 } }
142 142
143 if (gmdate("i") == "30") {
144 rg_log_enter("Clean old tokens...");
145 $sql = "DELETE FROM tokens WHERE expire < $now";
143 if (gmdate('i') == '30') {
144 rg_log_enter('Clean old tokens...');
145 $sql = 'DELETE FROM tokens WHERE expire < ' . $now;
146 146 $res = rg_sql_query($db, $sql); $res = rg_sql_query($db, $sql);
147 147 if ($res !== FALSE) if ($res !== FALSE)
148 148 rg_sql_free_result($res); rg_sql_free_result($res);
149 149 rg_log_exit(); rg_log_exit();
150 150 } }
151 151
152 if (gmdate("i") == "01") {
153 rg_log_enter("Clean old sess entries...");
154 $sql = "DELETE FROM sess WHERE expire < $now";
152 if (gmdate('i') == '01') {
153 rg_log_enter('Clean old sess entries...');
154 $sql = 'DELETE FROM sess WHERE expire < ' . $now;
155 155 $res = rg_sql_query($db, $sql); $res = rg_sql_query($db, $sql);
156 156 if ($res !== FALSE) if ($res !== FALSE)
157 157 rg_sql_free_result($res); rg_sql_free_result($res);
158 158 rg_log_exit(); rg_log_exit();
159 159 } }
160 160
161 if (gmdate("Hi") == "0605") {
162 rg_log_enter("Clean old login_tokens_ip entries...");
163 $sql = "DELETE FROM login_tokens_ip WHERE expire < $now";
161 if (gmdate('Hi') == '0605') {
162 rg_log_enter('Clean old login_tokens_ip entries...');
163 $sql = 'DELETE FROM login_tokens_ip WHERE expire < ' . $now;
164 164 $res = rg_sql_query($db, $sql); $res = rg_sql_query($db, $sql);
165 165 if ($res !== FALSE) if ($res !== FALSE)
166 166 rg_sql_free_result($res); rg_sql_free_result($res);
167 167 rg_log_exit(); rg_log_exit();
168 168 } }
169 169
170 while (gmdate("dHi") == "010610") {
170 while (gmdate('Hi') == '0610') {
171 171 if (rg_load() > 100) if (rg_load() > 100)
172 172 break; break;
173 173
174 rg_log_enter("Clean old empty slave tables...");
174 rg_log_enter('Clean old empty slave tables...');
175 $r = rg_state_get($db, 'slave_clean_last_run');
176 if ($r === FALSE)
177 break;
178 $when = gmmktime(0, 0, 0, gmdate('m'), 1, gmdate('Y'));
179 if ($r >= $when)
180 break;
175 181
176 $ts = time() - 3 * 31 * 24 * 3600;
177 $limit = gmdate('Y_m', $ts);
178 rg_log('limit=' . $limit);
179 foreach ($rg_sql_struct_slaves as $table) {
182 foreach ($rg_sql_struct_slaves as $table => $ti) {
180 183 rg_log('Slave: ' . $table); rg_log('Slave: ' . $table);
184
185 // value is in 'months'
186 $r = rg_state_get($db, 'slave_clean_table_' . $table);
187 if ($r === FALSE)
188 break;
189 if ($r === '')
190 continue;
191 if ($r == 0) // no expiration
192 continue;
193
194 $ts = gmmktime(0, 0, 0, gmdate('m') - $r, 1, gmdate('Y'));
195 $limit = gmdate('Y_m', $ts);
196 rg_log('limit=' . $limit);
197
181 198 $sql = 'SELECT relname FROM pg_class' $sql = 'SELECT relname FROM pg_class'
182 199 . ' WHERE relname LIKE \'' . $table . '_%\'' . ' WHERE relname LIKE \'' . $table . '_%\''
183 200 . ' AND relkind = \'r\'' . ' AND relkind = \'r\''
 
... ... while (gmdate("dHi") == "010610") {
209 226 rg_sql_free_result($res2); rg_sql_free_result($res2);
210 227 } }
211 228 rg_sql_free_result($res); rg_sql_free_result($res);
212 break;
213 229 } }
214 230
231 rg_stat_set($db, 'slave_clean_last_run', $when);
232
215 233 rg_log_exit(); rg_log_exit();
234 break;
216 235 } }
217 236
218 if (gmdate("Hi") == "0100")
237 if (gmdate('Hi') == '0100')
219 238 rg_admin_report1($db, $rg); rg_admin_report1($db, $rg);
220 239
221 240 // TODO: move it as an event after the push // TODO: move it as an event after the push
222 241 rg_mr_queue_process($db); rg_mr_queue_process($db);
223 242
243 if (gmdate('Hi') == '0305')
244 rg_clean_logs('/var/log/rocketgit');
245
224 246 rg_log_cron(); rg_log_cron();
225 247
226 rg_log("Done!");
227 ?>
248 rg_log('Done!');
Hints:
Before first commit, do not forget to setup your git environment:
git config --global user.name "your_name_here"
git config --global user.email "your@email_here"

Clone this repository using HTTP(S):
git clone https://rocketgit.com/user/catalinux/rocketgit

Clone this repository using ssh (do not forget to upload a key first):
git clone ssh://rocketgit@ssh.rocketgit.com/user/catalinux/rocketgit

Clone this repository using git:
git clone git://git.rocketgit.com/user/catalinux/rocketgit

You are allowed to anonymously push to this repository.
This means that your pushed commits will automatically be transformed into a merge request:
... clone the repository ...
... make some changes and some commits ...
git push origin main