最佳實務
Linux Basics
The default operating system for instances rented on WebCal is a Linux Ubuntu distribution, so familiarity with basic Linux commands is essential for training models. The following is an introduction to commonly used essential commands:
List files/folders
Commands: ls (list)
user@host:/tmp/test_dir# ls # 列出当前目录下的文件和文件夹
a.txt b
user@host:/tmp/test_dir# ls -l # 列出文件和文件夹的详细信息：权限，Owner，Group和创建/更新时间
total 4
-rw-rw-r-- 1 root root 0 11月 9 10:50 a.txt
drwxrwxr-x 2 root root 4096 11月 9 10:50 b
Create/Switch Path
New command: mkdir (create directory)
Change directory command: cd (change working directory)
user@host:/tmp# mkdir test_dir # 新建一个叫test_dir的路径
user@host:/tmp# cd test_dir/ # 进入 test_dir 路径
user@host:/tmp/test_dir#
Two special directories: .. and ., or written as ../ and ./, where ../ represents the parent directory and ./ represents the current directory.
user@host:/tmp/test_dir# cd ../test_dir/ # 进入上一级目录下的test_dir目录
user@host:/tmp#
View Current Path
Commands: pwd (list)
user@host:~# pwd
/root/
user@host:~#
Renaming and Moving Files/Folders
Command: mv (move)
user@host:/tmp# mv test_dir/ test_directory # 将test_dir目录重命名为test_directory，文件重命名同样适用
user@host:/tmp# cd test_directory/
daiauser@hostb@seeta:/tmp/test_directory#
user@host:/tmp/test_directory# mkdir a b # 创建两个文件夹a和b
user@host:/tmp/test_directory# ls
a b
user@host:/tmp/test_directory# mv a b/ # 将a移动到b目录下。如果b目录不存在的话，这条命令相当于将a重命名为b
user@host:/tmp/test_directory# tree
.
└── b
└── a
Copy Files/Folders
Command: cp (copy)
Parameters: -r (where -r stands for recursion)
user@host:/tmp/test_directory# mkdir a b # 创建两个文件夹a和b
user@host:/tmp/test_directory# ls
a b
user@host:/tmp/test_directory# cp -r a b # 将a文件夹拷贝到b文件夹下，-r代表递归拷贝
user@host:/tmp/test_directory# tree
.
└── a
└── b
└── a
Delete files/folders
Command: rm (remove)
Parameters: -rf (where -r stands for recursive and -f stands for force)
user@host:/tmp/test_directory# ls
a.txt folder
user@host:/tmp/test_directory# rm -rf folder
user@host:/tmp/test_directory# rm -rf folder/* # *是通配符号，这样代表folder文件夹下所有文件/文件夹
Set Environment Variables
Command: export
以常见的两个环境变量：PATH和LD_LIBRARY_PATH为例
1. PATH
如果你有自己安装的命令，希望暴露出来直接使用。比如miniconda中的python，如果不加环境变量一般需要写完整的路径：/x/x/x/miniconda3/bin/python，如果希望直接写python就能用到调用conda中的python指令，那么可以：
export PATH=/x/x/x/miniconda3/bin:$PATH
先解释上述命令的格式，右侧的路径可以写多个，以:分隔，$PATH表示求PATH变量的值，因为PATH环境变量以前可能已经有值，需要保留那些值不影响其他命令的使用，其次当输入了python命令时，会从PATH变量的路径下去找python可执行文件，先找到哪个就用哪个，因此:前后路径的先后顺序也很重要。
2. LD_LIBRARY_PATH
和PATH路径一样，只不过LD_LIBRARY_PATH是设置动态链接库的搜索路径。比如安装了CUDA以后，一般需要设置：
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
如果要查看是否设置成功可以使用命令：env | grep PATH 进行确认
最后以上设置的环境变量只在当前terminal的上下文中有效，如果希望全局有效，那么将export命令写入文件: ~/.bashrc，然后执行：source ~/.bashrc 生效或新打开终端
Editing Text Files
Command: vim
Advanced use of Vim is relatively complex; please refer to other documentation for guidance.
Compression and Decompression
Command: zip、unzip、tar
zip and unzip are used for compressing and decompressing ZIP archives, respectively, while tar is another, more general-purpose compression and decompression tool in Linux.
# zip和unzip。如果没有zip请使用apt-get update && apt-get install -y zip安装
user@host:/tmp/$ zip -r dir.zip test_directory/ # 将test_directory文件夹压缩为dir.zip文件
user@host:/tmp/$ unzip dir.zip # 将dir.zip文件解压
# tar. 以下参数c代表压缩，x表示解压，z代表压缩/解压为gz格式的压缩包
user@host:/tmp/$ tar czf dir.tar.gz test_directory/ # 将test_directory文件夹压缩为dir.tar.gz文件
user@host:/tmp/$ tar xzf dir.tar.gz # 将dir.tar.gz文件解压
# tar还可以用于压缩和解压其他格式的压缩文件，比如bz2
user@host:/tmp/$ tar cjf dir.tar.bz2 test_directory/ # 将test_directory文件夹压缩为dir.tar.bz2文件
user@host:/tmp/$ tar xjf dir.tar.bz2 # 将dir.tar.bz2文件解压
View GPU Information
Command: nvidia-smi
user@host:/tmp/test_directory# nvidia-smi
Mon Nov 8 11:55:26 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.82 Driver Version: 440.82 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 TITAN X (Pascal) Off | 00000000:01:00.0 On | N/A |
| 31% 57C P0 66W / 250W | 408MiB / 12194MiB | 2% Default |
+-------------------------------+----------------------+----------------------+
| 1 TITAN X (Pascal) Off | 00000000:04:00.0 Off | N/A |
| 93% 27C P8 11W / 250W | 2MiB / 12196MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1450 G /usr/lib/xorg/Xorg 32MiB |
| 0 2804 G /usr/lib/xorg/Xorg 351MiB |
+-----------------------------------------------------------------------------+
Memory-Usage # 显存的使用情况
408MiB / 12194MiB # 前者408MiB代表已使用的显存，后者12194MiB代表总显存
GPU-Util # GPU的使用率
2% # 使用率百分比
If you need to view real-time GPU usage information, you can do so under Console > Container Instances > Instance Monitoring.
View/Kill Processes
View process commands: ps
Command to kill a process: kill
root@container-5e3e11aeb4-948a17b1:~# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 14:04 ? 00:00:00 bash /init/boot/boot.sh
root 58 48 8 14:04 ? 00:00:03 /root/miniconda3/bin/python /root/miniconda3/bin/jupyter-lab --allow-root
root 60 48 0 14:04 ? 00:00:00 /usr/sbin/sshd -D
root 61 48 10 14:04 ? 00:00:04 /root/miniconda3/bin/python /root/miniconda3/bin/tensorboard --host 0.0.0.0 --port 6006 --logdir /root/tf-logs
root 146 61 0 14:04 ? 00:00:00 /root/miniconda3/lib/python3.8/site-packages/tensorboard_data_server/bin/server --logdir=/root/tf-logs
root 402 338 99 14:05 pts/0 00:00:06 python tensorflow2.x-test.py
Identify the process to terminate from the ps output based on the command name. For example, the process ID for the command python tensorflow2.x-test.py at the end is 402, so you can:
root@container-5e3e11aeb4-948a17b1:~# kill -9 402
root@container-5e3e11aeb4-948a17b1:~#
After running kill, you can use ps -ef again to verify that the process has terminated.
View a process’s CPU and memory usage
Command: top
Or use the platform’s “实例监控” feature for a simpler way to view the information.
Tasks: 11 total, 2 running, 9 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.3 us, 1.3 sy, 0.0 ni, 96.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 52801571+total, 45453059+free, 7807904 used, 65677196 buff/cache
KiB Swap: 2074620 total, 2074620 free, 0 used. 51678192+avail Mem
PID user@host PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2316 root 20 0 21.846g 1.796g 244664 R 101.4 0.4 0:05.56 python
58 root 20 0 372352 84804 15540 S 1.4 0.0 0:05.40 jupyter-lab
59 root 20 0 713796 11288 7668 S 1.4 0.0 0:01.31 proxy
2395 root 20 0 45920 3940 3444 R 1.4 0.0 0:00.01 top
1 root 20 0 25368 3724 3404 S 0.0 0.0 0:00.07 bash
48 root 20 0 55060 24328 9728 S 0.0 0.0 0:00.33 supervisord
60 root 20 0 72304 5872 5140 S 0.0 0.0 0:00.01 sshd
61 root 20 0 9756148 315032 156124 S 0.0 0.1 0:04.16 tensorboard
146 root 20 0 1582996 6964 5296 S 0.0 0.0 0:00.04 server
338 root 20 0 25824 4312 3800 S 0.0 0.0 0:00.18 bash
481 root 20 0 25824 4544 4040 S 0.0 0.0 0:00.18 bash
If there is a high load (high CPU usage), the relevant process will typically appear at the top of the list, and you can identify it by its name. You can determine the CPU usage of this process by checking the %CPU field. Memory usage is a bit more complex, but generally, checking the RES field is sufficient.For example, the CPU usage of the first Python process shown above is 101.4%, and its memory usage is 1.796g (Tip: If the memory unit displayed differs from the one shown above, press the e key to switch it).
Redirect Logs
Command: >
user@host:/tmp# python train.py # 一般情况下日志会输出到stdout/stderr中
Epoch.1 Iter 20
Epoch.1 Iter 40
Epoch.1 Iter 50
...
user@host:/tmp# python train.py > ./train.log 2>&1 # 把stdout/stderr中的日志重定向到train.log文件中，最后的2>&1中，2代表stderr, 1代表stdout，&1可以理解成像c语言中的求地址
user@host:/tmp# cat ./train.log # 将train.log文件中的内容打印在stdout。cat（Concatenate FILE(s) to standard output.）
Epoch.1 Iter 20
Epoch.1 Iter 40
Epoch.1 Iter 50
...
user@host:/tmp$ python train.py > ./train.log 2>&1 & # 如果最后再加一个&的效果是后台运行，还可以参考nohup的配合使用
Scenario 1
Scenario: The program has stopped, but GPU memory is still being used.
In general, this situation indicates that the process is frozen—it appears to have stopped but is actually still running. You can use ps -ef to check if the process still exists. If it does, use the kill command to kill the process, and finally use nvidia-smi to verify that the GPU memory has been released.
Scenario 2
Scenario: You want to save a copy of the models and data from an instance to file storage so that other instances can use them.
user@host:~# pwd
/root/
user@host:~# ls
train.py webcal-tmp webcal-fs
user@host:~# cp -r train.py webcal-fs/ # 把 train.py 文件存入文件存储中
Scenario 3
Scenario: It was discovered that a process’s memory usage exceeded the limit, causing the process to be terminated.
You can use the top command to view a process’s memory usage and confirm whether memory usage remains at a constant value rather than continuing to increase. If it continues to increase, this indicates a memory release issue in the program; you can analyze variable references in the Python code to optimize it.
Scenario 4
Scenario: Running training via a daemon process in a JupyterLab terminal, with concerns that logs might be lost if the browser window is closed.
You can use the log redirection feature to write logs to a file.