# PointCNN **Repository Path**: deep_learning_workpiece/PointCNN ## Basic Information - **Project Name**: PointCNN - **Description**: No description available - **Primary Language**: Unknown - **License**: MIT - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 0 - **Created**: 2018-06-22 - **Last Updated**: 2024-06-06 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # PointCNN Created by Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, and Baoquan Chen from Shandong University. **Yangyan, Rui, Mingchao and Wei are (being) hired by Alibaba AI Lab, working on 3D perception for autonomous driving. Join us for making the world a better place!** ## Introduction PointCNN is a simple and general framework for feature learning from point cloud, which refreshed five benchmark records in point cloud processing (as of Jan. 23, 2018), including: * classification accuracy on ModelNet40 (**91.7%**, with 1024 input points only) * classification accuracy on ScanNet (**77.9%**) * segmentation part averaged IoU on ShapeNet Parts (**86.13%**) * segmentation mean IoU on S3DIS (**62.74%**) * per voxel labelling accuracy on ScanNet (**85.1%**) PointCNN achieved 84.4% accuracy on ModelNet40 classification with only 32 input points, which outperforms PointNet and PointNet++ with a 18.3% accuracy gap, making PointCNN quite promising for real time recognition applications with low resolution point cloud input, such as **autonomous driving**, as well as **robotics** in general. See our PointCNN paper on arXiv for more details. **We highly welcome issues, rather than emails, for PointCNN related questions.** **We are working actively on Semantic3D dataset, stay tuned.** ## License Our code is released under MIT License (see LICENSE file for details). ## Code Organization The core X-Conv and PointCNN architecture are defined in [pointcnn.py](pointcnn.py). The network/training/data augmentation hyper parameters for classification tasks are defined in [pointcnn_cls](pointcnn_cls), for segmentation tasks are defined in [pointcnn_seg](pointcnn_seg). ### Explanation of X-Conv and X-DeConv Parameters Take the xconv_params and xdconv_params from [shapenet_x8_2048_fps.py](pointcnn_seg/shapenet_x8_2048_fps.py) for example: ``` xconv_param_name = ('K', 'D', 'P', 'C', 'links') xconv_params = [dict(zip(xconv_param_name, xconv_param)) for xconv_param in [(8, 1, -1, 32 * x, []), (12, 2, 768, 32 * x, []), (16, 2, 384, 64 * x, []), (16, 6, 128, 128 * x, [])]] xdconv_param_name = ('K', 'D', 'pts_layer_idx', 'qrs_layer_idx') xdconv_params = [dict(zip(xdconv_param_name, xdconv_param)) for xdconv_param in [(16, 6, 3, 2), (12, 6, 2, 1), (8, 6, 1, 0), (8, 4, 0, 0)]] ``` Each element in xconv_params is a tuple of (K, D, P, C, links), where K is the neighborhood size, D is the dilation rate, P is the representative point number in the output (-1 means all input points are output representative points), and C is the output channel number. The links are used for adding DenseNet style links, e.g., [-1, -2] will tell the current layer to receive inputs from the previous two layers. from Each element specifies the parameters of one X-Conv layer, and they are stacked to create a deep network. Each element in xdconv_params is a tuple of (K, D, pts_layer_idx, qrs_layer_idx), where K and D have the same meaning as that in xconv_params, pts_layer_idx specifies the output of which X-Conv layer (from the xconv_params) will be the input of this X-DeConv layer, and qrs_layer_idx specifies the output of which X-Conv layer (from the xconv_params) will be forwarded and fused with the output of this X-DeConv layer. The P and C parameters of this X-DeConv layer is also determined by qrs_layer_idx. Similarly, each element specifies the parameters of one X-DeConv layer, and they are stacked to create a deep network. ## PointCNN Usage PointCNN is implemented and tested with Tensorflow 1.6 in python3 scripts. **Tensorflow before 1.5 version is not recommended, because of API.** It has dependencies on some python packages such as transforms3d, h5py, plyfile, and maybe more if it complains. Install these packages before the use of PointCNN. If you can only use Tensorflow 1.5 because of OS factor(UBUNTU 14.04),please modify "isnan()" to "std::nan()" in "/usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/numeric_types.h" line 49 Here we list the commands for training/evaluating PointCNN on classification and segmentation tasks on multiple datasets. * ### Classification * #### ModelNet40 ``` cd data_conversions python3 ./download_datasets.py -d modelnet cd ../pointcnn_cls ./train_val_modelnet.sh -g 0 -x modelnet_x3_l4 ``` * #### ScanNet Please refer to for downloading ScanNet task data and scannet_labelmap, and refer to https://github.com/ScanNet/ScanNet/tree/master/Tasks/Benchmark for downloading ScanNet benchmark files: scannet_dataset_download |_ data |_ scannet_labelmap |_ benchmark ``` cd ../data/scannet/scannet_dataset_download/ mv ./scannet_labelmap/scannet-labels.combined.tsv ../benchmark/ #./pointcnn_root cd ../../../pointcnn/data_conversions python extract_scannet_objs.py -f ../../data/scannet/scannet_dataset_download/data/ -b ../../data/scannet/scannet_dataset_download/benchmark/ -o ../../data/scannet/cls/ python prepare_scannet_cls_data.py -f ../../data/scannet/cls/ cd ../pointcnn_cls/ ./train_val_scannet.sh -g 0 -x scannet_x3_l4 ``` * #### tu_berlin ``` cd data_conversions python3 ./download_datasets.py -d tu_berlin python3 ./prepare_tu_berlin_data.py -f ../../data/tu_berlin/ -a cat ../../data/tu_berlin/fold_1_*.txt ../../data/tu_berlin/fold_0_*.txt > ../../data/tu_berlin/train_files.txt cat ../../data/tu_berlin/fold_2_files.txt > ../../data/tu_berlin/test_files.txt cd ../pointcnn_cls ./train_val_tu_berlin.sh -g 0 -x tu_berlin_x3_l4 ``` * #### quick_darw Note that the training/evaluation of quick_draw requires LARGE RAM, as we load all stokes into RAM and converting them into point cloud on-the-fly. ``` cd data_conversions python3 ./download_datasets.py -d quick_draw cd ../pointcnn_cls ./train_val_quick_draw.sh -g 0 -x quick_draw_full_x4_l4 ``` * #### MNIST ``` cd data_conversions python3 ./download_datasets.py -d mnist python3 ./prepare_mnist_data.py -f ../../data/mnist cd ../pointcnn_cls ./train_val_mnist.sh -g 0 -x mnist_x2_l4 ``` * #### CIFAR-10 ``` cd data_conversions python3 ./download_datasets.py -d cifar10 python3 ./prepare_cifar10_data.py cd ../pointcnn_cls ./train_val_cifar10.sh -g 0 -x cifar10_x3_l4 ``` * ### Segmentation We use farthest point sampling (the implementation from PointNet++) in segmentation tasks. Compile FPS before the training/evaluation: ``` cd sampling bash tf_sampling_compile.sh ``` * #### ShapeNet ``` cd data_conversions python3 ./download_datasets.py -d shapenet_partseg python3 ./prepare_partseg_data.py -f ../../data/shapenet_partseg cd ../pointcnn_seg ./train_val_shapenet.sh -g 0 -x shapenet_x8_2048_fps ./test_shapenet.sh -g 0 -x shapenet_x8_2048_fps -l ../../models/seg/pointcnn_seg_shapenet_x8_2048_fps_xxxx/ckpts/iter-xxxxx -r 10 cd ../evaluation python3 eval_shapenet_seg.py -g ../../data/shapenet_partseg/test_label -p ../../data/shapenet_partseg/test_data_pred_10 -a ``` * #### S3DIS Please refer to [data_conversions](data_conversions/README.md) for downloading S3DIS, then: ``` cd data_conversions python3 prepare_s3dis_label.py python3 prepare_s3dis_data.py python3 prepare_s3dis_filelists.py mv S3DIS_files/* ../../data/S3DIS/out_part_rgb/ ./train_val_s3dis.sh -g 0 -x s3dis_x8_2048_k16_fps -a 1 ./test_s3dis.sh -g 0 -x s3dis_x8_2048_k16_fps -a 1 -l ../../models/seg/s3dis_x8_2048_fps_k16_xxxx/ckpts/iter-xxxxx -r 4 cd ../evaluation python3 s3dis_merge.py -d python3 eval_s3dis.py ``` Please notice that these command just for Area 1 (specified by -a 1 option) validation. Results on other Areas can be computed by iterating -a option. * #### ScanNet Please refer to [data_conversions](data_conversions/README.md) for downloading ScanNet, then: ``` cd data_conversions python3 prepare_scannet_seg_data.py python3 prepare_scannet_seg_filelists.py cd ../pointcnn_seg ./train_val_scannet.sh -g 0 -x scannet_x8_2048_k8_fps ./test_scannet.sh -g 0 -x scannet_x8_2048_k8_fps -l ../../models/seg/pointcnn_seg_scannet_x8_2048_k8_fps_xxxx/ckpts/iter-xxxxx -r 4 cd ../evaluation python3 eval_scannet.py ``` * #### Semantic3D ``` cd data_conversions bash download_semantic3d.sh bash un7z_semantic3d.sh mkdir ../../data/semantic3d/val mv ../../data/semantic3d/train/bildstein_station3_xyz_intensity_rgb.* ../../data/semantic3d/train/domfountain_station2_xyz_intensity_rgb.* ../../data/semantic3d/train/sg27_station4_intensity_rgb.* ../../data/semantic3d/train/untermaederbrunnen_station3_xyz_intensity_rgb.* ../../data/semantic3d/val python3 prepare_semantic3d_data.py python3 prepare_semantic3d_filelists.py cd ../pointcnn_seg ./train_val_semantic3d.sh -g 0 -x semantic3d_x8_2048_k16 ./test_semantic3d.sh -g 0 -x semantic3d_x8_2048_k16 -l cd ../evaluation python3 semantic3d_merge.py -d -v ``` * ### Tensorboard If you want to moniter your train step, we recommand you use following command ``` cd /PointCNN tensorboard --logdir=../models/ <--port=6006> ``` ## More PointCNN Implementations * MXNet implementation  * Pytorch implementation