RocketGit

hahattpro / crowd_counting_framework (public) (License: Unspecified) (since 2020-12-23) (hash sha1)

backupfrom https://github.com/ttpro1995/crowd_counting_framework

Clone URLs: https://rocketgit.com/user/hahattpro/crowd_counting_framework ssh://rocketgit@ssh.rocketgit.com/user/hahattpro/crowd_counting_framework git://git.rocketgit.com/user/hahattpro/crowd_counting_framework

List of commits:

Subject	Hash	Author	Date (UTC)
data_flow fixed 256, we ignore too small image in Batch for now	34a56f59674dee5267dba356b3ebe469baa91baf	Thai Thien	2020-04-08 15:46:34
fix typo	e9e5fe46043233c703e0a872afd1510434d4b5b0	Thai Thien	2020-04-07 18:49:20
tryting to reproduce ccnn again	b3681bbf85bb50eb5430581d8779415520d78dc2	Thai Thien	2020-04-07 18:25:30
fix bug, new dataflow	28706ac8dd35ff6f7d17c7d01656b047dd95cffe	Thai Thien	2020-04-07 17:54:57
ccnn_v2_t2_sha_scheduler	0bd4b83a3e2adb4d054537fde38c56ec3eca0d68	Thai Thien	2020-04-07 02:16:53
lower ccnn sgd lr	9f10c1c664cf9be2b3093ba72815740b31521287	Thai Thien	2020-04-06 17:41:39
5	07b51e18899506183b197366b8a2e6319dec0215	Thai Thien	2020-04-06 17:30:33
3 more test	03cca5576a44e925db8626c905f324d42c205923	Thai Thien	2020-04-06 17:22:40
modify bigtail2 t2	fcd49caa43303c11049cb6c40b5e5c62f1fcdab3	Thai Thien	2020-04-06 17:06:15
try to reproduce	0946ba05287dcbc00d7743347f6f75d7f7601636	Thai Thien	2020-04-06 16:55:50
custom ccnn v2 t4 sha lower lr	b2381872bc284fc16af9c7d3129f1bee418f8915	Thai Thien	2020-04-06 02:11:55
ccnn v2 t5	d2c4a1182c5ca93b9e6b2bc4dadc4ddc8c9c521b	Thai Thien	2020-04-05 10:41:18
try to reproduce that 86	4a715c2e62e1f92051d5a4abbba983821e6bb422	Thai Thien	2020-04-05 10:30:47
add c5	e6c82e35a4920700840bde0e891ca5d6cf285a7c	Thai Thien	2020-04-05 10:05:44
some big tail	3705300d179b004c1049ec402461de6c9f67bbd1	Thai Thien	2020-04-05 09:49:06
t5	b704da887280b721e0402db32919a1e7b8753269	Thai Thien	2020-04-04 11:18:46
verify if it work	9aa7f6d734d1fb13b9fb02e6d77d813139b6122c	Thai Thien	2020-04-04 11:15:25
m3 t123 exp	f76cec015c34648ec45f02865562f6d05d507eb2	Thai Thien	2020-04-03 15:27:22
m1 shb	35455b092280af7e7f3140601b909dc58b5c2590	Thai Thien	2020-04-02 10:42:43
m2	765cc07c3d3645b897f53b54f905b8a9c3aa0ac3	Thai Thien	2020-04-02 10:37:47

Commit 34a56f59674dee5267dba356b3ebe469baa91baf - data_flow fixed 256, we ignore too small image in Batch for now
Author: Thai Thien
Author date (UTC): 2020-04-08 15:46
Committer name: Thai Thien
Committer date (UTC): 2020-04-08 15:46
Parent(s): e9e5fe46043233c703e0a872afd1510434d4b5b0
Signer:
Signing key:
Signing status: N
Tree: 8877897b9aea257aa9674b9188efd9993bf800d7

File	Lines added	Lines deleted
data_flow.py	52	2
sanity_check_dataloader.py	7	3
train_compact_cnn_lrscheduler.py	1	1
train_script/CCNN/ccnn_v2_t5_sha_schedule.sh	4	4
train_script/CCNN/ccnn_v2_t7_sha.sh	9	0

File data_flow.py changed (mode: 100644) (index c50eac3..a9da806)
...	...	def load_data_shanghaitech_180(img_path, train=True):
255	255	target1 = np.expand_dims(target1, axis=0) # make dim (batch size, channel size, x, y) to make model output	target1 = np.expand_dims(target1, axis=0) # make dim (batch size, channel size, x, y) to make model output
256	256	return img, target1	return img, target1
257	257
	258
	259		def load_data_shanghaitech_256(img_path, train=True):
	260		"""
	261		crop fixed 256, allow batch in non-uniform dataset
	262		:param img_path:
	263		:param train:
	264		:return:
	265		"""
	266		gt_path = img_path.replace('.jpg', '.h5').replace('images', 'ground-truth-h5')
	267		img = Image.open(img_path).convert('RGB')
	268		gt_file = h5py.File(gt_path, 'r')
	269		target = np.asarray(gt_file['density'])
	270		target_factor = 8
	271		crop_sq_size = 256
	272		if train:
	273		crop_size = (crop_sq_size, crop_sq_size)
	274		dx = int(random.random() * (img.size[0] - crop_sq_size))
	275		dy = int(random.random() * (img.size[1] - crop_sq_size))
	276		if dx < 0 or dy < 0: # we crop more than we can chew, so...
	277		return None, None
	278		img = img.crop((dx, dy, crop_size[0] + dx, crop_size[1] + dy))
	279		target = target[dy:crop_size[1] + dy, dx:crop_size[0] + dx]
	280
	281		if random.random() > 0.8:
	282		target = np.fliplr(target)
	283		img = img.transpose(Image.FLIP_LEFT_RIGHT)
	284
	285		target1 = cv2.resize(target, (int(target.shape[1] / target_factor), int(target.shape[0] / target_factor)),
	286		interpolation=cv2.INTER_CUBIC) * target_factor * target_factor
	287		# target1 = target1.unsqueeze(0) # make dim (batch size, channel size, x, y) to make model output
	288		target1 = np.expand_dims(target1, axis=0) # make dim (batch size, channel size, x, y) to make model output
	289		return img, target1
	290
258	291	def load_data_shanghaitech_same_size_density_map(img_path, train=True):	def load_data_shanghaitech_same_size_density_map(img_path, train=True):
259	292	gt_path = img_path.replace('.jpg', '.h5').replace('images', 'ground-truth-h5')	gt_path = img_path.replace('.jpg', '.h5').replace('images', 'ground-truth-h5')
260	293	img = Image.open(img_path).convert('RGB')	img = Image.open(img_path).convert('RGB')

...	...	class ListDataset(Dataset):
555	588	self.load_data_fn = load_data_shanghaitech_20p_rnd	self.load_data_fn = load_data_shanghaitech_20p_rnd
556	589	elif dataset_name == "shanghaitech_180":	elif dataset_name == "shanghaitech_180":
557	590	self.load_data_fn = load_data_shanghaitech_180	self.load_data_fn = load_data_shanghaitech_180
558
	591		elif dataset_name == "shanghaitech_256":
	592		self.load_data_fn = load_data_shanghaitech_256
559	593	elif dataset_name == "ucf_cc_50":	elif dataset_name == "ucf_cc_50":
560	594	self.load_data_fn = load_data_ucf_cc50	self.load_data_fn = load_data_ucf_cc50
561	595	elif dataset_name == "ucf_cc_50_pacnn":	elif dataset_name == "ucf_cc_50_pacnn":

...	...	class ListDataset(Dataset):
574	608	if self.debug:	if self.debug:
575	609	print(img_path)	print(img_path)
576	610	img, target = self.load_data_fn(img_path, self.train)	img, target = self.load_data_fn(img_path, self.train)
	611		if img is None or target is None:
	612		return None
577	613	if self.transform is not None:	if self.transform is not None:
578	614	img = self.transform(img)	img = self.transform(img)
579	615	return img, target	return img, target
580	616
581	617
	618		def my_collate(batch): # batch size 4 [{tensor image, tensor label},{},{},{}] could return something like G = [None, {},{},{}]
	619		"""
	620		collate that ignore None
	621		However, if all sample is None, we have problem, so, set batch size bigger
	622		https://stackoverflow.com/questions/57815001/pytorch-collate-fn-reject-sample-and-yield-another
	623		:param batch:
	624		:return:
	625		"""
	626		batch = list(filter (lambda x:x is not None, batch)) # this gets rid of nones in batch. For example above it would result to G = [{},{},{}]
	627		# I want len(G) = 4
	628		# so how to sample another dataset entry?
	629		return torch.utils.data.dataloader.default_collate(batch)
	630
582	631	def get_dataloader(train_list, val_list, test_list, dataset_name="shanghaitech", visualize_mode=False, batch_size=1):	def get_dataloader(train_list, val_list, test_list, dataset_name="shanghaitech", visualize_mode=False, batch_size=1):
583	632	if visualize_mode:	if visualize_mode:
584	633	transformer = transforms.Compose([	transformer = transforms.Compose([

...	...	def get_dataloader(train_list, val_list, test_list, dataset_name="shanghaitech",
599	648	num_workers=0,	num_workers=0,
600	649	dataset_name=dataset_name),	dataset_name=dataset_name),
601	650	batch_size=batch_size,	batch_size=batch_size,
602		num_workers=4)
	651		num_workers=4,
	652		collate_fn=my_collate)
603	653
604	654	if val_list is not None:	if val_list is not None:
605	655	val_loader = torch.utils.data.DataLoader(	val_loader = torch.utils.data.DataLoader(

File sanity_check_dataloader.py changed (mode: 100644) (index a1885b5..9f28698)
...	...	if __name__ == "__main__":
13	13	TRAIN_PATH = os.path.join(DATA_PATH, "train_data")	TRAIN_PATH = os.path.join(DATA_PATH, "train_data")
14	14	TEST_PATH = os.path.join(DATA_PATH, "test_data")	TEST_PATH = os.path.join(DATA_PATH, "test_data")
15	15	dataset_name = args.datasetname	dataset_name = args.datasetname
	16		dataset_name = "shanghaitech_256"
16	17
17
	18		count_below_256 = 0
18	19	# create list	# create list
19	20	train_list, val_list = get_train_val_list(TRAIN_PATH)	train_list, val_list = get_train_val_list(TRAIN_PATH)
20	21	test_list = None	test_list = None
21	22
22	23	# create data loader	# create data loader
23		train_loader, val_loader, test_loader = get_dataloader(train_list, val_list, test_list, dataset_name=dataset_name)
	24		train_loader, val_loader, test_loader = get_dataloader(train_list, val_list, test_list, dataset_name=dataset_name, batch_size=5)
24	25
25	26	print("============== TRAIN LOADER ====================================================")	print("============== TRAIN LOADER ====================================================")
26	27	min_1 = 500	min_1 = 500

...	...	if __name__ == "__main__":
33	34	min_1 = size_1	min_1 = size_1
34	35	if min_2 > size_2:	if min_2 > size_2:
35	36	min_2 = size_2	min_2 = size_2
	37		if size_1 < 256 or size_2 < 256:
	38		count_below_256+=1
36	39	# example: img shape:torch.Size([1, 3, 716, 1024]) == label shape torch.Size([1, 1, 89, 128])	# example: img shape:torch.Size([1, 3, 716, 1024]) == label shape torch.Size([1, 1, 89, 128])
37	40
38	41	print("============== VAL LOADER ====================================================")	print("============== VAL LOADER ====================================================")
39	42	for img, label in val_loader:	for img, label in val_loader:
40	43	print("img shape:" + str(img.shape) + " == " + "label shape " + str(label.shape))	print("img shape:" + str(img.shape) + " == " + "label shape " + str(label.shape))
41	44	print(min_1)	print(min_1)
42		print(min_2)
	45		print(min_2)
	46		print("count < 256 = ", count_below_256)

File train_compact_cnn_lrscheduler.py changed (mode: 100644) (index 176ca05..fdec7cd)
...	...	if __name__ == "__main__":
62	62	optimizer = torch.optim.Adam(model.parameters(), args.lr,	optimizer = torch.optim.Adam(model.parameters(), args.lr,
63	63	weight_decay=args.decay)	weight_decay=args.decay)
64	64
65		milestones_values = [(200, 1e-4), (300, 2e-5)]
	65		milestones_values = [(300, 1e-4), (400, 2e-5)]
66	66	experiment.log_parameter("milestones_values", str(milestones_values))	experiment.log_parameter("milestones_values", str(milestones_values))
67	67	lr_scheduler = PiecewiseLinear(optimizer, param_name="lr", milestones_values=milestones_values)	lr_scheduler = PiecewiseLinear(optimizer, param_name="lr", milestones_values=milestones_values)
68	68

File train_script/CCNN/ccnn_v2_t5_sha_schedule.sh copied from file train_script/CCNN/ccnn_v2_t4_sha_schedule.sh (similarity 53%) (mode: 100644) (index adcc3c5..69de937)
1		CUDA_VISIBLE_DEVICES=6 HTTPS_PROXY="http://10.30.58.36:81" nohup python train_compact_cnn_lrscheduler.py \
2		--task_id ccnn_v2_t4_sha_schedule \
	1		CUDA_VISIBLE_DEVICES=2 HTTPS_PROXY="http://10.30.58.36:81" nohup python train_compact_cnn_lrscheduler.py \
	2		--task_id ccnn_v2_t5_sha_schedule \
3	3	--note "ccnnv2 with decay and 20p augment, much lower lr" \	--note "ccnnv2 with decay and 20p augment, much lower lr" \
4	4	--input /data/rnd/thient/thient_data/ShanghaiTech/part_A \	--input /data/rnd/thient/thient_data/ShanghaiTech/part_A \
5	5	--lr 1e-4 \	--lr 1e-4 \
6	6	--decay 1e-4 \	--decay 1e-4 \
7	7	--batch_size 8 \	--batch_size 8 \
8		--datasetname shanghaitech_180 \
9		--epochs 1000 > logs/ccnn_v2_t4_sha_schedule.log &
	8		--datasetname shanghaitech_256 \
	9		--epochs 1500 > logs/ccnn_v2_t5_sha_schedule.log &

File train_script/CCNN/ccnn_v2_t7_sha.sh added (mode: 100644) (index 0000000..54cd4b6)
	1	CUDA_VISIBLE_DEVICES=1 HTTPS_PROXY="http://10.30.58.36:81" nohup python train_compact_cnn.py \
	2	--task_id ccnn_v2_t7_sha \
	3	--note "ccnnv2 with decay and 20p augment, much lower lr" \
	4	--input /data/rnd/thient/thient_data/ShanghaiTech/part_A \
	5	--lr 1e-5 \
	6	--batch_size 8 \
	7	--decay 1e-4 \
	8	--datasetname shanghaitech_256 \
	9	--epochs 1500 > logs/ccnn_v2_t7_sha.log &

Hints:
Before first commit, do not forget to setup your git environment:

git config --global user.name "your_name_here"
git config --global user.email "your@email_here"

Clone this repository using HTTP(S):

git clone https://rocketgit.com/user/hahattpro/crowd_counting_framework

Clone this repository using ssh (do not forget to upload a key first):

git clone ssh://rocketgit@ssh.rocketgit.com/user/hahattpro/crowd_counting_framework

Clone this repository using git:

git clone git://git.rocketgit.com/user/hahattpro/crowd_counting_framework

You are allowed to anonymously push to this repository.
This means that your pushed commits will automatically be transformed into a merge request:

... clone the repository ...
... make some changes and some commits ...
git push origin main