/wsr/ - Worksafe Requests » Thread #1324325

[11 / 3 / 4]

17KiB, 206x282, file.png

External hard drive poor performance

Anonymous Mon 27 Feb 2023 04:21:59 No.1324325 View View Reply Original Report

Quoted By: >>1324327 >>1324349 >>1325221 >>1325658

/g/ wizards I can't figure this out, the write speeds to my external hdd seem too slow.

I'm using a WD Elements 1 TB USB 3.0 as external hdd, the partition table is GPT and the partition is has a Ext4 LUKS encrypted filesystem
KDE Plasma 5.27.1
Kernel 6.1.12-zen1-1-zen (64-bit)
Garuda Linux (Arch based distro for noobs)

I'm using rsync -avh --progress to copy a 20~ GB file from my internal nvme to it and rsync shows a speed rate of about 500 MB/s thenrapidly drops to 5-20 MB/s shortly after copying 2 GB of the file. This sounds off to me.
I tried Dolphin (Plasma's file manager) and Thunar (booting a session with Xfce4 instead of KDE Plasma) and got the same results.
I tried updating packages and rebooting and got the same results

The relevant output of lsusb -t shows
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
|__ Port 4: Dev 7, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
|__ Port 6: Dev 8, If 0, Class=Hub, Driver=hub/4p, 5000M

Is my system using the correct driver? What the hell is going on, shouldn't it copy faster? Any help is appreciated.

Anonymous

Anonymous Mon 27 Feb 2023 04:23:49 No.1324327 Report

Quoted By:

>>1324325
I forgot to add, SMART doesn't show any problems on the external hdd. It has little over a 100 hours of being powered on and it doesn't record any bad sectors.

Anonymous

Anonymous Mon 27 Feb 2023 05:27:24 No.1324349 Report

Quoted By: >>1324352 >>1325658

>>1324325
I don't know but this sounds a lot like SMR slowdown. Once a SMR drive's small CMR buffer is filled the drive has to start writing directly to the SMR section, which is most of the drive, and write speed does slow down substantially. Speeds lower than 1 MB/s are reported by some users.

If this is the cause of you slowdown you can anticipate several hours of leaving the drive on with no drive activity for the drive's internal logic to flush the buffer, ie, move data from its small CMR buffer to the main SMR storage.

Small home user drives like you have (currently 8TB or less) are very often SMR because they are cheaper for manufacturers to make. Look up "SMR vs CMR" for details on what SMR and CMR mean.

Anonymous

Anonymous Mon 27 Feb 2023 05:57:21 No.1324352 Report

Quoted By: >>1324365

>>1324349
I'm not going to lie this is more than I can understand without reading about it, but thanks for the reply. Just today I learnt that Seagate, WD and more sold a lot of SMR drives without labeling them as such (the Wikipedia SMR article goes over this briefly) and that they have worse performance.

I'll look up data about the differences, thanks anon. Idk if it's just that (I should check for more info on the hard drive model I have) or if I can tweak something in my configs. Thanks again.

Maybe I just started noticing the slow performance because now I'm writing to it (torrents, soulseek and direct downloads) and reading (listening to lossless music, uploading torrents and soulseek) all at the same time, when before I'd just read one file from it.

Anonymous

Anonymous Mon 27 Feb 2023 06:40:13 No.1324365 Report

Quoted By: >>1324785 >>1325658

>>1324352
Here's a maybe overly simplified version.

Lets say drive tracks are normally written two units wide.

When you write track 1 it's the first track all by itself, it's fine, it's two units wide. But when your SMR writes track 2 it overlaps track 1. Track 2 is on top, it's two units wide, but track 1 is now only one unit wide. Track 3 overlaps track 2, so now tracks 1 and 2 are one unit wide.

You can still read the data in tracks 1 and 2 just fine even if narrow.

Three tracks of data in four units of space instead of six units of space makes manufacturer happy because they sell you less material and they get more profit.

If you want to make a change to a file in track 1, the disk has to rewrite the whole section of tracks 1 2 and 3 because writing to track 1 will overlap the space track 2 occupies. So it sort of has to move all this section. This takes a lot more more time than it would if there was no shingle overlap.

Conventional Magnetic Recording doesn't have this overlap. If you want to change a file in track 1, tracks 2 and 3 are not affected.

When you get a new SMR drive, it's empty, and you copy stuff to and it will just fill up space sort of sequentially 1, 2, 3 etc. and you get pretty good speeds.

But after you've used it for a while, you'll eventually start needing to overwrite existing copies of files.

Your SMR drive has a small section of CMR. When you do only a little copying, the CMR section will fill up just a little. When the drive is idle, it will move files off the CMR section to the SMR section. You won't even know this is happening so the drive gives you a good speed.

But when this CMR buffer is full, and especially when you copy lots of little files which have to be individually journalled, your drive writes directly to the SMR and this is slow.

Read speeds between SMR and CMR drives are effectively unchanged, just SMR writes are slower.

Downloading torrents --> many non-sequential small writes --> bad for SMR speed

Anonymous

Anonymous Tue 28 Feb 2023 12:28:26 No.1324785 Report

Quoted By: >>1325629 >>1325660

>>1324365
Anon thanks for the detailed reply! I spent some time reading about this and your explanation is the best one I found so far.

I found that sector size is very important when partitioning a SMR drive:
>In particular shingled magnetic recording (SMR) drives that are firmware-managed are severely and negatively impacted if using a logical sector size of 512 bytes if their physical sector size is of 4096 bytes. Those drives have different performance writing zones and remapping reallocation occurs while being idle, but during heavy active writes (e.g., RAID resilvering, backups, writing many small files, rsync, etc.), a different file system sector size could drop write speed to single digit megabytes/second, as the higher performance write areas get depleted, and the sector translation layer gets overworked on the shingled areas.
Source: https://wiki.archlinux.org/title/Advanced_Format

And to no surprise, my drive reports 512 bytes logical sectors when the physical sectors are 4096 bytes. I'll try moving all my data to another drive and reformat it.

Anonymous

View Same Google ImgOps iqdb SauceNAO 1491451341631.png, 760KiB, 760x839

Anonymous Wed 01 Mar 2023 17:17:29 No.1325221 Report

Quoted By: >>1325629

>>1324325
>external hard drive
bro just buy small laptop drives and hook through usb converter they're far more reliable

Anonymous

View Same Google ImgOps iqdb SauceNAO file.png, 336KiB, 637x477

Anonymous Thu 02 Mar 2023 19:22:03 No.1325629 Report

Quoted By: >>1325632

>>1324785
Just to clarify for anyone going through the archive like I did to learn more...
>https://desuarchive.org/g/thread/76657636/#q76664867
Compare the output of
>cat /sys/block/sdb/queue/physical_block_size
>cat /sys/block/sdb/queue/minimum_io_size
If they match even though the logical block size might be different
>cat /sys/block/sdb/queue/logical_block_size
your drive is partitioned just fine and you should do as >>1325221 says. I got an enclouse for an old 7200 RPM 2.5 inch Sata 3.0 laptop drive (CMR) and it works wonders. I wish I knew all about this before buying 8 FUCKING TB of MSR 5200-5400 RPM storage like a dumbass.

Anonymous

Anonymous Thu 02 Mar 2023 19:23:39 No.1325632 Report

Quoted By:

>>1325629
SMR, I was seething I can't even speak correctly
/thread
Thanks to everyone involved

Anonymous

Anonymous Thu 02 Mar 2023 20:38:57 No.1325658 Report

Quoted By:

>>1324325
>rsync shows a speed rate of about 500 MB/s thenrapidly drops to 5-20 MB/s shortly after copying 2 GB of the file. This sounds off to me.
Linux has a large write cache.
>>1324349
>>1324365
No.
There's no hard disk in the world that can do >>1324325
>500 MB/s
, that's simply ridiculous, it's close to the theoretical maximum of SATA 3. The only thing that's that fast is the NVMe's read speed, and it's only free to read that fast so long as there's space in the write cache for it to read into. Once the cache is full, you see the real speed of the device.

Anonymous

Anonymous Thu 02 Mar 2023 20:40:43 No.1325660 Report

Quoted By:

>>1324785
>I'll try moving all my data to another drive and reformat it.
It's not possible to have unaligned sectors with GPT, because GPT forces partitions to start on multiples of 1MB, and 1MB is 256*4k.

Capcode	All Only User Posts Only Verified Posts Only Moderator Posts Only Manager Posts Only Admin Posts Only Developer Posts Only Founder Posts
Show Posts	All Only With Images Only Without Images Only Spoiler Images Only Non-Spoiler Images
Deleted Posts	All Only Deleted Posts Only Non-Deleted Posts
Ghost Posts	All Only Ghost Posts Only Non-Ghost Posts
Post Type	All Only Sticky Threads Only Opening Posts Only Reply Posts
Results	All Grouped By Threads
Order	Latest Posts First Oldest Posts First

On these archives

Your latest searches

External hard drive poor performance