FAQ
Frequently Asked Questions
Q: Why is the program stuck and not producing any output?
A: First, use the top and nvidia-smi commands to check CPU and GPU usage, respectively. If the CPU is consistently at 100% and the GPU is idle, it is highly likely that the program is stuck on a GPU call; in that case, please refer to the answer to the previous question.If the issue is not caused by the reasons mentioned above, you’ll need to debug the code. We recommend adding print statements to log output at key lines of code, then running the program to pinpoint the exact line where it’s getting stuck. Once you’ve identified the line, search Google to determine the cause. Since these situations are often related to the code itself, each case requires a specific analysis; avoid making assumptions without reviewing the code.
Q: What causes a GPU memory out-of-memory (CUDA OOM) error?
A: If the program reports an OOM (out of memory) error due to insufficient GPU memory, the simplest way to troubleshoot is to set the batch size to 1 and then gradually increase it, observing at what batch size the OOM occurs. Use this information to decide whether to upgrade to a multi-GPU setup or switch to a card with more GPU memory.In another scenario, if the program runs without an OOM error the first time but triggers an OOM error upon subsequent runs, first use the nvidia-smi command to check the GPU memory usage while the program is not running. If memory is already in use, this indicates that residual processes from previous runs need to be cleaned up. To do this, run: ps -ef 找到进程PID，然后kill -9 PID.If there is no GPU memory usage, it means this is a characteristic of the program itself. For example, in dynamic deep learning frameworks, GPU memory usage can fluctuate, and the memory demand during computation may have exceeded the limit.
Q: What should I do if there aren’t enough idle GPUs on the host?
A: Option 1: You can use cardless startup to download important data from the instance. Option 2: Migrate the instance. Alternatively, you can wait until the GPU on that host becomes available.
Q: Why can't I connect to VSCode, SSH, etc., after changing the instance image?
A: For Linux/Mac users, delete the local known_hosts file and run the command: rm ~/.ssh/known_hosts. For Windows users, delete the C:/Users/当前user名字/.ssh/konwn_hosts file. Try again after deleting the file.
Q: Can vouchers be used for annual or monthly subscriptions?
A: Some coupons are eligible; please check the coupon usage scope. Coupons can be used in combination, and the system will apply them before deducting from your balance.
Q: Will the GPU be reserved after an annual or monthly subscription instance is shut down?
A: As long as you are within your annual or monthly subscription period, the resources will remain reserved, and you can restart them at any time without worrying that the GPU will be occupied by other users.
Q: Do multiple GPUs in a single instance support parallel processing?
A: Multiple GPUs within the same instance are located on the same physical host and support parallel processing. If you require parallel processing across multiple hosts and GPUs, please contact customer service.
Q: For pay-as-you-go instances, how is billing calculated if the price of the host GPU changes during the instance’s runtime?
A: Billing for pay-as-you-go instances is based on the price in effect at the time the instance is started; price changes that occur during the instance’s runtime do not affect billing. If the instance is restarted, billing will be based on the latest price.
Q: Can data be recovered from an instance that has already been released?
A: It cannot be recovered.
Q: What should I do if the host where the instance is located experiences a failure, such as a hard drive or GPU failure?
A: There are two options: migrate the instance to another host or wait for the machine to be repaired and brought back online. The platform will provide appropriate compensation in such cases.
Q: Can data on an instance be unexpectedly corrupted or lost?
A: Most of the local data disks used in instances are physical disks with no redundant backups; therefore, data stored on local disks is at risk of loss. Please back up important data promptly. Data stored on shared cloud disks uses multi-replica redundancy, ensuring very high data reliability.
Q: Will closing the browser or logging out affect programs running in JupyterLab (Notebook)?
A: It will not be affected, but you must ensure that logs are properly saved, such as by redirecting them to a log file. For details, refer to the documentation
Q: How can I ensure that a program running via SSH won’t terminate if the connection is lost?
A: We recommend running commands in the JupyterLab terminal or using them in conjunction with tools like screen or tmux. Please refer to the Daemon Subsection
Q: Why does the program display "Killed" and stop?
A: The program was terminated by the system because it exceeded the memory limit. You can verify this by checking the changes in memory usage through the instance monitoring interface. To resolve this, you can upgrade the instance—memory size is allocated linearly based on the number of GPUs—or switch to a host with more memory.