Sunday 13 April 2014

Create large files quickly on Linux



For testing its really useful to create files of an exact file size. The following details some quick ways to create files on Linux with related considerations. 


fallocate - preallocate space to a file


  • Recommended on Linux due to speed
  • Example to create 1GB
$ time fallocate -l 1G 1GBFile.tmp

real    0m0.051s
user    0m0.008s
sys    0m0.004s

  • Note : you can remove the 'time' command 
  • -l :  Specifies the length of the allocation, in bytes
  • The file was created in less than 1 second!
  • Consideration - A fast alternative to dd

dd - /dev/zero


  • /dev/zero is a Linux file providing null characters when read
  • Example to create 1GB
$ time dd if=/dev/zero of=1GBFile.tmp bs=1024 count=$((1024 * 1024 ))
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 18.4725 s, 58.1 MB/s

real    0m18.479s
user    0m0.300s
sys    0m8.248s

  • Note : you can remove the 'time' command
  • bs : byte size
  • count : copy only this amount of input blocks
  • The 1GB file was created in 18 seconds 
  • Consideration - file cannot be read from but is fast to create

dd - /dev/urandom

  • /dev/urandom is a Linux file that serves as a random number generator 
  • Uses entropy collected from device drivers and uses the pseudo-random generator when more entropy is needed than has been collected 
$ time dd if=/dev/urandom of=1GBFile.tmp bs=1024 count=$((1024 * 1024 ))
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 132.903 s, 8.1 MB/s

real    2m12.910s
user    0m0.272s
sys    2m0.952s

  • Note : you can remove the 'time' command 
  • This took 2 minutes & 12 seconds significantly longer than using /dev/zero
  • Consideration - No readable lines but some content information. Can take time to create


No comments:

Post a Comment