Remote conversion server and remote storage server problem

[chown]

I currently have 1 remote conversion server that is working properly or seems to be at least that is using FTP. I have the local conversion server disabled as it is my web server as well. The remote conversion server is also a storage server and is treated like a CDN, but is running Nginx and seems to serve content fine. The problem seems to be that once I disabled my local conversion server it no longer is able to move files from the remote conversion server to the remote storage server (they are the same server) using FTP. I think it was all working fine when I was using NFS, but I switched to FTP because of the latency between data centers is around ~100ms and NFS was very slow. Should this setup work? Is it supposed to pull the video from the remote conversion server back to the local server then upload it again to the FTP server running on the remote storage server? I don't know what the problem is. I may look into switching to SSHFS instead of FTP. Any suggestions on making this work? All my background tasks are failing with the message "Failed to put XXXXX.mp4 file to storage server" and error code "04 - Storage server connection error".

 

Example of one of the many failed task logs:

 

[2023-11-04 04:03:11] INFO  Starting conversion task 6470 for video 4784 [PH-C]
[2023-11-04 04:03:11] INFO  Downloading source files [PH-C-1]
[2023-11-04 04:03:11] INFO  ....Downloading source file from: https://somedomain/get_file/0/ef497997f62551dcadd20402e807c221/4000/4784/4784.tmp/
[2023-11-04 04:03:14] INFO  ....Downloaded 67088176 bytes
[2023-11-04 04:03:14] INFO  Pre-processing source file [PH-C-3]
[2023-11-04 04:03:14] INFO  Creating video format "MP4  480p" [PH-C-5:MP4  480p]
[2023-11-04 04:03:14] INFO  ....Creating video file [PH-C-5-2]
[2023-11-04 04:04:29] INFO  ........Created video file 4784.mp4: 670 seconds, 67235407 bytes
[2023-11-04 04:04:29] INFO  ....Moving metadata to the file beginning [PH-C-5-3]
[2023-11-04 04:04:29] INFO  ........Skipped, already done by FFmpeg
[2023-11-04 04:04:29] INFO  ....Copying video file to "Remote" [PH-C-5-4:Remote]
[2023-11-04 04:04:29] ERROR ........Failed to put 4784.mp4 file to storage server "Remote"
[2023-11-04 04:04:29] INFO  Conversion task 6470 for video 4784 is finished in 78 sec [PH-CE]
[2023-11-04 04:04:29] INFO  Removing 6481 task dir as it is being marked to be deleted
[2023-11-03 23:05:12] ERROR(4)  Failed to put 4784.mp4 file to storage server "Remote" [PH-E]

[chown]

I moved to SSHFS and it's working fine now.

[Tech Support]

On 11/4/2023 at 4:17 AM, chown said:

Is it supposed to pull the video from the remote conversion server back to the local server then upload it again to the FTP server running on the remote storage server?

No, the conversion server will push file directly to the storage server via FTP. If this push is not working, likely related to FTP limitation (firewall, or something like that). But you have already said that this was resolved.