Jan 252013
 

In this post, we’re going to demonstrate why aligning your disk partitions can be a good idea. For starters, we’re going to go with the default alignment, do a test, then align, and redo the test.

But first, in case you have no idea what I’m talking about, this randomly discovered post may be a useful primer.

Create the scheme on the disk

This command creates a GPT scheme on the drive.

$ sudo gpart create -s GPT ada1
ada1 created
$ gpart show ada1
=>       34  488397101  ada1  GPT  (232G)
         34  488397101        - free -  (232G)
$

Create a FreeBSD partition

This command creates a FreeBSD partition, and I am purposely not aligning it.

$ sudo gpart add -t freebsd -a 574 ada1
ada1s1 added, but partition is not aligned on 4096 bytes
$ gpart show ada1
=>       34  488397101  ada1  GPT  (232G)
         34        540        - free -  (270k)
        574  488396510     1  freebsd  (232G)
  488397084         51        - free -  (25k)

$ 

As you can see, this partition starts 574 blocks in on the disk, or 574 * 512 = 293888 bytes in. Clearly, not 4K block aligned, as stated in the output of the command, and as shown in the math: 293888 / 4096 = 71.75.

Create a BSD scheme

Here, we create a scheme, into which we will create a partition and newfs it.

$ sudo gpart create -s BSD ada1s1
ada1s1 created

$ gpart show ada1s1
=>        0  488396510  ada1s1  BSD  (232G)
          0  488396510          - free -  (232G)

Create a partition

Let’s create a 90GB partition. That’s big enough for some writes, but smaller than using the whole drive, because we don’t need it all.

$ sudo gpart add -t freebsd-ufs -s 90g ada1s1
ada1s1a added
$ gpart show ada1s1
=>        0  488396510  ada1s1  BSD  (232G)
          0          2          - free -  (1.0k)
          2  188743680       1  freebsd-ufs  (90G)
  188743682  299652828          - free -  (142G)

$ 

newfs

Let’s create a filesystem on there, so we can write to it.

$ sudo newfs -U /dev/ada1s1a
/dev/ada1s1a: 92160.0MB (188743680 sectors) block size 32768, fragment size 4096
	using 148 cylinder groups of 626.09MB, 20035 blks, 80256 inodes.
	with soft updates
super-block backups (for fsck_ffs -b #) at:
 192, 1282432, 2564672, 3846912, 5129152, 6411392, 7693632, 8975872, 10258112, 11540352, 12822592, 14104832, 15387072, 16669312, 17951552, 19233792, 20516032, 21798272, 23080512, 24362752, 25644992,
 26927232, 28209472, 29491712, 30773952, 32056192, 33338432, 34620672, 35902912, 37185152, 38467392, 39749632, 41031872, 42314112, 43596352, 44878592, 46160832, 47443072, 48725312, 50007552, 51289792,
 52572032, 53854272, 55136512, 56418752, 57700992, 58983232, 60265472, 61547712, 62829952, 64112192, 65394432, 66676672, 67958912, 69241152, 70523392, 71805632, 73087872, 74370112, 75652352, 76934592,
 78216832, 79499072, 80781312, 82063552, 83345792, 84628032, 85910272, 87192512, 88474752, 89756992, 91039232, 92321472, 93603712, 94885952, 96168192, 97450432, 98732672, 100014912, 101297152, 102579392,
 103861632, 105143872, 106426112, 107708352, 108990592, 110272832, 111555072, 112837312, 114119552, 115401792, 116684032, 117966272, 119248512, 120530752, 121812992, 123095232, 124377472, 125659712,
 126941952, 128224192, 129506432, 130788672, 132070912, 133353152, 134635392, 135917632, 137199872, 138482112, 139764352, 141046592, 142328832, 143611072, 144893312, 146175552, 147457792, 148740032,
 150022272, 151304512, 152586752, 153868992, 155151232, 156433472, 157715712, 158997952, 160280192, 161562432, 162844672, 164126912, 165409152, 166691392, 167973632, 169255872, 170538112, 171820352,
 173102592, 174384832, 175667072, 176949312, 178231552, 179513792, 180796032, 182078272, 183360512, 184642752, 185924992, 187207232, 188489472

Simple test

Let’s write a 60GB file into there.

But first, mount it:

$ sudo mount /dev/ada1s1a /mnt
$ ls -l /mnt
total 0
$ cd /mnt
$ sudo mkdir test1
$ sudo chown dan:dan test1
$ cd test1

Now, we can write:

$ dd if=/dev/zero of=FirstTest bs=32k count=15728640

/mnt: write failed, filesystem is full
dd: FirstTest: No space left on device
2626900+0 records in
2626899+0 records out
86078226432 bytes transferred in 647.749411 secs (132888159 bytes/sec)

That filled up the space, but that’s OK. The goal is the rate. We got about 126 MB/s. Let’s try the other HDD. We will do the same steps as above, but for ada2, instead of ada1.

$ gpart show ada2
=>       34  488397101  ada2  GPT  (232G)
         34        540        - free -  (270k)
        574  488396510     1  freebsd  (232G)
  488397084         51        - free -  (25k)

$ gpart show ada2s1
=>        0  488396510  ada2s1  BSD  (232G)
          0          2          - free -  (1.0k)
          2  188743680       1  freebsd-ufs  (90G)
  188743682  299652828          - free -  (142G)

$ dd if=/dev/zero of=FirstTest bs=32k count=15728640

/mnt: write failed, filesystem is full
dd: FirstTest: No space left on device
2626900+0 records in
2626899+0 records out
86078226432 bytes transferred in 692.483754 secs (124303604 bytes/sec)

And that is 118MB/s. Hmmm, this is very interesting.

Now let’s try 4K aligned partitions

First, we destroy what we already created:

$ sudo gpart delete -i 1 ada2s1
ada2s1a deleted

$ sudo gpart destroy ada2s1
ada2s1 destroyed

$ sudo gpart delete -i 1 ada2  
ada2s1 deleted

$ sudo destroy ada2
ada2 destroy 

$ sudo gpart delete -i 1 ada1s1
ada1s1a deleted

$ sudo gpart destroy ada1s1
ada1s1 destroyed

$ sudo gpart delete -i 1 ada1
ada1s1 deleted

$ sudo gpart destroy ada1
ada1 destroyed

Aligned

Now we do the same steps as above, but align the partitions.

$ sudo gpart create -s GPT ada1
ada1 created
$ sudo gpart add -t freebsd -a 4k ada1
ada1s1 added
$ gpart show ada1
=>       34  488397101  ada1  GPT  (232G)
         34          6        - free -  (3.0k)
         40  488397088     1  freebsd  (232G)
  488397128          7        - free -  (3.5k)

$ sudo gpart create -s BSD ada1s1
ada1s1 created
$ sudo gpart add -t freebsd-ufs -s 90g -a 4k ada1s1
ada1s1a added

$ gpart show ada1s1
=>        0  488397088  ada1s1  BSD  (232G)
          0  188743680       1  freebsd-ufs  (90G)
  188743680  299653408          - free -  (142G)

$ sudo newfs -U /dev/ada1s1a
/dev/ada1s1a: 92160.0MB (188743680 sectors) block size 32768, fragment size 4096
	using 148 cylinder groups of 626.09MB, 20035 blks, 80256 inodes.
	with soft updates
super-block backups (for fsck_ffs -b #) at:
 192, 1282432, 2564672, 3846912, 5129152, 6411392, 7693632, 8975872, 10258112, 11540352, 12822592, 14104832, 15387072, 16669312, 17951552, 19233792, 20516032, 21798272, 23080512, 24362752, 25644992,
 26927232, 28209472, 29491712, 30773952, 32056192, 33338432, 34620672, 35902912, 37185152, 38467392, 39749632, 41031872, 42314112, 43596352, 44878592, 46160832, 47443072, 48725312, 50007552, 51289792,
 52572032, 53854272, 55136512, 56418752, 57700992, 58983232, 60265472, 61547712, 62829952, 64112192, 65394432, 66676672, 67958912, 69241152, 70523392, 71805632, 73087872, 74370112, 75652352, 76934592,
 78216832, 79499072, 80781312, 82063552, 83345792, 84628032, 85910272, 87192512, 88474752, 89756992, 91039232, 92321472, 93603712, 94885952, 96168192, 97450432, 98732672, 100014912, 101297152, 102579392,
 103861632, 105143872, 106426112, 107708352, 108990592, 110272832, 111555072, 112837312, 114119552, 115401792, 116684032, 117966272, 119248512, 120530752, 121812992, 123095232, 124377472, 125659712,
 126941952, 128224192, 129506432, 130788672, 132070912, 133353152, 134635392, 135917632, 137199872, 138482112, 139764352, 141046592, 142328832, 143611072, 144893312, 146175552, 147457792, 148740032,
 150022272, 151304512, 152586752, 153868992, 155151232, 156433472, 157715712, 158997952, 160280192, 161562432, 162844672, 164126912, 165409152, 166691392, 167973632, 169255872, 170538112, 171820352,
 173102592, 174384832, 175667072, 176949312, 178231552, 179513792, 180796032, 182078272, 183360512, 184642752, 185924992, 187207232, 188489472

$ dd if=/dev/zero of=FirstTest bs=32k count=15728640

/mnt: write failed, filesystem is full
dd: FirstTest: No space left on device
2626900+0 records in
2626899+0 records out
86078226432 bytes transferred in 649.784040 secs (132472054 bytes/sec)

That’s pretty much the same rate.

Again, for ada2:

$ dd if=/dev/zero of=FirstTest bs=32k count=15728640640

/mnt: write failed, filesystem is full
dd: FirstTest: No space left on device
2626900+0 records in
2626899+0 records out
86078226432 bytes transferred in 693.786576 secs (124070182 bytes/sec)

Hmm. Am I doing this wrong? Are these HDD not subject to a 4K alignment issue?

Sequential doesn’t count

Sequential sequence doesn’t count. “Alignment doesn’t matter much for sequential access, because the disk can coalesce requests. Only the first and last block will require a read-modify-write” – Dag-Erling Smørgrav

Instead, let’s try bonnie++ on ada1 with aligned partitions:

$ bonnie++ -s 66000
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
floater.unix 66000M   532  99 113838  20 40954  36  1049  99 116269  19 132.0   5
Latency             16232us     755ms   15295ms   19001us    1624ms    4518ms
Version  1.96       ------Sequential Create------ --------Random Create--------
floater.unixathome. -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 31292  58 +++++ +++ +++++ +++ 23477  46 +++++ +++ +++++ +++
Latency               158ms      72us      38us     125ms      35us      37us
1.96,1.96,floater.unixathome.org,1,1359208317,66000M,,532,99,113838,20,40954,36,1049,99,116269,19,132.0,5,16,,,,,31292,58,+++++,+++,+++++,+++,23477,46,+++++,+++,+++++,+++,16232us,755ms,15295ms,19001us,1624ms,4518ms,158ms,72us,38us,125ms,35us,37us

Then the same drive, unaligned:

first on ada1:

$ bonnie++ -s 66000
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
floater.unix 66000M   519  99 109704  20 41924  33  1028  98 114528  18 120.0   7
Latency             16477us     448ms   15903ms   58855us     902ms    4473ms
Version  1.96       ------Sequential Create------ --------Random Create--------
floater.unixathome. -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 31825  70 +++++ +++ +++++ +++ 24005  57 +++++ +++ +++++ +++
Latency               111ms      50us      41us     252ms      34us      41us
1.96,1.96,floater.unixathome.org,1,1359185135,66000M,,519,99,109704,20,41924,33,1028,98,114528,18,120.0,7,16,,,,,31825,70,+++++,+++,+++++,+++,24005,57,+++++,+++,+++++,+++,16477us,448ms,15903ms,58855us,902ms,4473ms,111ms,50us,41us,252ms,34us,41us

And now on ada2 (aligned):

$ bonnie++ -s 66000
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
floater.unix 66000M   524  99 121793  23 39647  43  1053  99 120336  20 160.2  12
Latency             16407us     795ms   10873ms   22076us    1623ms     579ms
Version  1.96       ------Sequential Create------ --------Random Create--------
floater.unixathome. -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency             62188us      36us      61us   62548us      37us      41us
1.96,1.96,floater.unixathome.org,1,1359202461,66000M,,524,99,121793,23,39647,43,1053,99,120336,20,160.2,12,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,16407us,795ms,10873ms,22076us,1623ms,579ms,62188us,36us,61us,62548us,37us,41us

Now, again with unaligned partitions:

Then on ada2:

$ bonnie++ -s 66000
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done                
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
floater.unix 66000M   526  99 98514  18 37055  35  1048  99 99222  16 132.4   5
Latency             16232us     588ms   17524ms   47938us    1624ms    4466ms
Version  1.96       ------Sequential Create------ --------Random Create--------
floater.unixathome. -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency             63286us      33us      45us   65937us      33us      46us
1.96,1.96,floater.unixathome.org,1,1359213425,66000M,,526,99,98514,18,37055,35,1048,99,99222,16,132.4,5,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,16232us,588ms,17524ms,47938us,1624ms,4466ms,63286us,33us,45us,65937us,33us,46us
Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

  One Response to “Aligned versus not aligned”

  1. Ahhh, yes, as was pointed out, today, and early (had I recalled it), alignment does not matter much for sequential access.

    I’m running some bonnie++ tests now.