# Slow ftp but fast fetch or curl



## athan (Apr 13, 2018)

Putting or getting files using amd64 FreeBSD 11.1rel stock cli ftp client I'm experiencing very slow speed (4MB/s).
Same files using either fetch, curl or wget are transfered at almost 50MB/sec, still using exactly the same settings and ftp protocol.

Tried with various other ftp servers getting exactly the same results. Builtin ftp client is 10x slower than either fetch, curl or wget.

Anybody has any idea why is that?

PS: Generic kernel and no firewall or packet filter


----------



## SirDice (Apr 13, 2018)

It might be caused by a difference in active or passive FTP. I'm not entirely sure but I believe the ftp(1) command uses active FTP and fetch(1) passive FTP by default. It's worth trying to see if that makes a difference.


----------



## athan (Apr 13, 2018)

Thanks SirDice but I'm afraid it's not that simple.
All ftp servers I tried are passive - only thus ftp is not even connecting when using -A option

Btw no other ftp client I tried so far has this speed issue, only the stock one. Really weird situation...


----------



## PacketMan (Apr 16, 2018)

I've never tried to performance test FTP, but I wonder, do you log in once, or per file.  When I transfer files using a utility the utility logs in once and the file transfer rate seems to be pretty fast.  However some IP cameras I have that use FTP, authenticate on a per file basis I'm pretty sure. I gotta belive that will hurt the throughput rate.  I have this vague memory of using a utility program one time that did the same thing, authentication on a per file basis, and I was thinking "well now that's just yucky".


----------



## VladiBG (Apr 16, 2018)

Search the problem somewhere else it's not in the native ftp client. It utulize the 100Mbps line (11.25MiB/s). Check your router for ftp inspection.


----------



## athan (Apr 17, 2018)

VladiBG said:


> Check your router for ftp inspection.


I think that if router was the problem any ftp client using same ports and same mode should be affected not only the stock one.
Try using the stock client with some distant server with enough latency and compare results with other ftp clients.


----------



## Datapanic (Apr 17, 2018)

I tried `/usr/bin/ftp` with FreeBSD 11.1-p9 amd64 between a VM and my NAS box (all FreeBSD) on the same LAN and got 52 MiB/s for a 620M file.  

But going from the same VM to ftp.freebsd.org and downloading FreeBSD-11.1-RELEASE-amd64-dvd1.iso, I am only getting about 2 MiB/s.


----------



## athan (Apr 18, 2018)

Datapanic said:


> I tried `/usr/bin/ftp` But going from the same VM to ftp.freebsd.org and downloading FreeBSD-11.1-RELEASE-amd64-dvd1.iso, I am only getting about 2 MiB/s.



Could you please try the same download from ftp.freebsd.org using fetch this time and tell us the difference if any.


----------



## Datapanic (Apr 18, 2018)

athan said:


> Could you please try the same download from ftp.freebsd.org using fetch this time and tell us the difference if any.



With `/usr/bin/fetch`, it was only a little better with an average of 3200 kBps which should be around 3.2 MiB/s:


```
[honkycat@chateau01 ~]$ fetch -h ftp.freebsd.org -f /pub/FreeBSD/releases/ISO-IMAGES/11.1/FreeBSD-11.1-RELEASE-amd64-dvd1.iso
FreeBSD-11.1-RELEASE-amd64-dvd1.iso            31% of 3193 MB 3600 kBps 11m02s
```


----------



## SirDice (Apr 18, 2018)

Are these different?

`fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/11.1/FreeBSD-11.1-RELEASE-amd64-disc1.iso`
`fetch http://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/11.1/FreeBSD-11.1-RELEASE-amd64-disc1.iso`


----------



## athan (Apr 18, 2018)

Yes they're different, only the first uses ftp protocol. The second one is http..

BTW, 2MB/s to 3.2MB is not only "a little better", it's about 60% faster; a huge difference considering that both use the same protocol.


----------



## VladiBG (Apr 19, 2018)

If you want to test the speed then you have to make a test setup between two computers connected on the same switch. On the public ftp/http servers there's a lot of QoS polices that restrict the bandwidth per connection and protocol to ensure that one client won't take the entire bandwidth and to protect itself from DoS. That's why those test are irrelevant. Also there's a difference in the protocols and send/receive buffers that also have some impact of the transfer speed.

So you are testing nothing more than your Internet connection speed not the ftp client itself. If you still want to test it over the Internet then you can use some of the test servers that can provide bigger throughput per client. For example you can try the speedtest.tele2.net that claim to support up to 10Gbps. 
the addresses are:
ftp://speedtest.tele2.net
http://speedtest.tele2.net

where you can select to download/upload a test file.


----------



## SirDice (Apr 19, 2018)

athan said:


> Yes they're different, only the first uses ftp protocol. The second one is http..


Yes, I know that. I was wondering if _you_ experience a difference in speed between HTTP and FTP.


----------



## VladiBG (Apr 20, 2018)

I don't see any difference in the download speed between HTTP and FTP. 
Here's the test with ftp and fetch:


----------



## athan (Apr 21, 2018)

VladiBG said:


> So you are testing nothing more than your Internet connection speed not the ftp client itself. If you still want to test it over the Internet then you can use some of the test servers that can provide bigger throughput per client. For example you can try the speedtest.tele2.net that claim to support up to 10Gbps.
> the addresses are:
> ftp://speedtest.tele2.net
> http://speedtest.tele2.net



I know that VladiBG and it's exactly what I'm doing.

Take a look


----------



## VladiBG (Apr 21, 2018)

can you try in active mode?


----------



## athan (Apr 21, 2018)

98 MBps in active mode, much better but still less than half of fetch throughput


----------



## Sensucht94 (Apr 22, 2018)

athan said:


> Putting or getting files using amd64 FreeBSD 11.1rel stock cli ftp client I'm experiencing very slow speed (4MB/s).
> Same files using either fetch, curl or wget are transfered at almost 50MB/sec, still using exactly the same settings and ftp protocol.
> 
> Tried with various other ftp servers getting exactly the same results. Builtin ftp client is 10x slower than either fetch, curl or wget.
> ...



Hi, I had never realized it, but after performing some tests it looks like you're right: the ftp client included in base system is ~ 4 times slower than any other ftp/http download client I tried. Keeping in mind that FTP as proto should be slightly faster in download than HTTP given same hardware and configuration (I assume both FTP and HTTP FreeBSD servers physically reside on the same machine), here's my tests (speed peaks) performed on FreeBSD 11.1-p9 RELEASE, custom kernel, IPFW enabled (http, ftp, ftp-data, dns ports naturally opened), Ethernet stable dsl connection with Intel (R) PRO/1000 Gigabit (2016) using em(4) driver:

- FreeBSD's ftp(1) getting ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/11.1/FreeBSD-11.1-RELEASE-amd64-disc1.iso:  *1,1 Mb/s*

- FreeBSD's fetch(1) getting http://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/11.1/FreeBSD-11.1-RELEASE-amd64-disc1.iso: *4,1 Mb/s*

At which point i thought there was rather an issue with the FreeBSD server. However:

- ftp/lftp getting disc1.iso from ftp FreeBSD server: *4,5 Mb/s*

- ftp/axel download accelerator getting disc1.iso from http FreeBSD server: *4,45 Mb/s*

- ftp/axel getting disc1.iso from ftp FreeBSD server: *4,57 Mb/s*


----------

