The best way to store a large number of files on the server
As you know, you should not store a large number of files in one folder since A system crash can occur very quickly, or simply the files will be read very slowly.
To solve this problem, many programmers take the md5 file name f789f789abc898d6892df98d09a8f8, and then break the name like this:
/f7/89/f789abc898d6892df98d09a8f8.jpg The
math here is very simple - one character is 16 options.
Thus, 2 characters is already 16 * 16 = 256 options.
In our case, we have 2 nestings of 2 characters each, so the maximum number of folders will be 256 * 256 = 65536 folders.
If we need to save 1,000,000 files, then the number of files in each folder will not exceed 1,000,000 / 65536 = 15 files.
Yes, the option is simple, but what if we need to not only save files well, but also quickly find them?
For example, we have a social network and we want for each user to create a separate folder with its id number and store files in it, which in turn also have their own id.
And for us it is important not only to save the file but also quickly find where it lies by its id.
To solve this problem, I wrote a class that allows you to save a large number of files or folders on the server in the tree structure of folders.
Here is the structure the class creates:
To calculate the maximum number of files that fit in this structure, you need to raise the maximum number of files in a folder to the degree of the number of branches plus one.
In the image we see 2 branches and 3 files in each folder.
Thus, 3 needs to be raised to the power of 2 + 1 = 3 * 3 * 3 = 27 files.
To save no more than 1,000,000 files in such a structure, 2 branches of 100 files in each folder (100 * 100 * 100) are enough for us.
You need to pass an array of parameters to the class - the path to the folder where the tree will be built, the maximum number of files in the folder, the number of branches, or you can apply the pattern (parameter) of the maximum number of files that has already been calculated in advance - bigint, int, mediumint, smallint:
array ('upload_dir' => Q_PATH. '/ uploads /', 'max_file_count' => 1000, 'branches' => 2, 'pattern' => '')
The class itself:
Download in the archive
For the variant of the social network described above, you need to use the class 2 times: first, to build a tree of folders, then to build a tree in each folder for files.
I also draw your attention to the fact that in this post I omitted (but did not know) the topic "Maximum allowable number of files on a hard disk".
To solve this problem, many programmers take the md5 file name f789f789abc898d6892df98d09a8f8, and then break the name like this:
/f7/89/f789abc898d6892df98d09a8f8.jpg The
math here is very simple - one character is 16 options.
Thus, 2 characters is already 16 * 16 = 256 options.
In our case, we have 2 nestings of 2 characters each, so the maximum number of folders will be 256 * 256 = 65536 folders.
If we need to save 1,000,000 files, then the number of files in each folder will not exceed 1,000,000 / 65536 = 15 files.
Yes, the option is simple, but what if we need to not only save files well, but also quickly find them?
For example, we have a social network and we want for each user to create a separate folder with its id number and store files in it, which in turn also have their own id.
And for us it is important not only to save the file but also quickly find where it lies by its id.
To solve this problem, I wrote a class that allows you to save a large number of files or folders on the server in the tree structure of folders.
Here is the structure the class creates:
To calculate the maximum number of files that fit in this structure, you need to raise the maximum number of files in a folder to the degree of the number of branches plus one.
In the image we see 2 branches and 3 files in each folder.
Thus, 3 needs to be raised to the power of 2 + 1 = 3 * 3 * 3 = 27 files.
To save no more than 1,000,000 files in such a structure, 2 branches of 100 files in each folder (100 * 100 * 100) are enough for us.
You need to pass an array of parameters to the class - the path to the folder where the tree will be built, the maximum number of files in the folder, the number of branches, or you can apply the pattern (parameter) of the maximum number of files that has already been calculated in advance - bigint, int, mediumint, smallint:
array ('upload_dir' => Q_PATH. '/ uploads /', 'max_file_count' => 1000, 'branches' => 2, 'pattern' => '')
The class itself:
$value) {
if(array_key_exists($key, $def_arr) && is_array($value)) {
$def_arr[$key]=self::arr_union($def_arr[$key], $new_arr[$key]);
}
else {
$def_arr[$key]=$value;
}
}
return $def_arr;
}
}
/**
* Класс построения дерева
*/
class Upload {
public $id;
private $upload_dir;
private $max_file_count;
private $branches;
public function __construct(array $param=array()) {
$def_param=array('upload_dir'=>Q_PATH.'/uploads/','max_file_count'=>1000,'branches'=>2,'pattern'=>'');
$upload_param=Functions::arr_union($def_param,$param);
$this->upload_dir=$upload_param['upload_dir'];
$this->max_file_count=$upload_param['max_file_count'];
$this->branches=$upload_param['branches'];
//сложность надумана, все зависит от инодов df -i и tune2fs -l /dev/hda1 и df -Ti
switch($upload_param['pattern']) {
case 'bigint':
$this->max_file_count=512;
$this->branches=6;
break;
case 'int':
$this->max_file_count=216;
$this->branches=3;
break;
case 'mediumint':
$this->max_file_count=204;
$this->branches=2;
break;
case 'smallint':
$this->max_file_count=182;
$this->branches=1;
break;
}
$this->del_id();
}
public function set_id($id) {
$this->id=$id;
}
public function del_id() {
$this->id=0;
}
public function find_upload($url) {
if(is_file($url)) {
return true;
}
else {
return false;
}
}
public function get_upload($id,$fl) {
$this->set_id($id);
for($i=$this->branches;$i>=1;$i--) {
$dir=ceil($this->id/pow($this->max_file_count,$i))%$this->max_file_count;
$dir_file_arr[]=$dir>0?$dir:$this->max_file_count;
}
$dir_file_str=implode("/", $dir_file_arr);
return $this->upload_dir.$dir_file_str.'/'.$this->id.$fl;
}
public function put_upload($id,$fl,$data) {
$this->set_id($id);
for($i=$this->branches;$i>=1;$i--) {
$dir=ceil($this->id/pow($this->max_file_count,$i))%$this->max_file_count;
$dir_file_arr[]=$dir>0?$dir:$this->max_file_count;
$dir_file_str=implode("/", $dir_file_arr);
if(!is_dir($this->upload_dir.$dir_file_str)) {
mkdir($this->upload_dir.$dir_file_str, 0777);
//chmod($this->upload_dir.$dir_file_str, 0777);
}
}
file_put_contents($this->upload_dir.$dir_file_str.'/'.$this->id.$fl, $data);
return $this->upload_dir.$dir_file_str.'/'.$this->id.$fl;
}
public function set_upload($id,$fl) {
$this->set_id($id);
for($i=$this->branches;$i>=1;$i--) {
$dir=ceil($this->id/pow($this->max_file_count,$i))%$this->max_file_count;
$dir_file_arr[]=$dir>0?$dir:$this->max_file_count;
$dir_file_str=implode("/", $dir_file_arr);
if(!is_dir($this->upload_dir.$dir_file_str)) {
mkdir($this->upload_dir.$dir_file_str, 0777);
//chmod($this->upload_dir.$dir_file_str, 0777);
}
}
return $this->upload_dir.$dir_file_str.'/'.$this->id.$fl;
}
public function get_upload_dir($id) {
$this->set_id($id);
for($i=$this->branches;$i>=1;$i--) {
$dir=ceil($this->id/pow($this->max_file_count,$i))%$this->max_file_count;
$dir_file_arr[]=$dir>0?$dir:$this->max_file_count;
}
$dir_file_str=implode("/", $dir_file_arr);
return $this->upload_dir.$dir_file_str.'/'.$this->id;
}
public function set_upload_dir($id) {
$this->set_id($id);
for($i=$this->branches;$i>=1;$i--) {
$dir=ceil($this->id/pow($this->max_file_count,$i))%$this->max_file_count;
$dir_file_arr[]=$dir>0?$dir:$this->max_file_count;
$dir_file_str=implode("/", $dir_file_arr);
if(!is_dir($this->upload_dir.$dir_file_str)) {
mkdir($this->upload_dir.$dir_file_str, 0777);
//chmod($this->upload_dir.$dir_file_str, 0777);
}
}
if(!is_dir($this->upload_dir.$dir_file_str.'/'.$this->id)) {
mkdir($this->upload_dir.$dir_file_str.'/'.$this->id, 0777);
//chmod($this->upload_dir.$dir_file_str.'/'.$this->id, 0777);
}
return $this->upload_dir.$dir_file_str.'/'.$this->id;
}
}
Download in the archive
For the variant of the social network described above, you need to use the class 2 times: first, to build a tree of folders, then to build a tree in each folder for files.
I also draw your attention to the fact that in this post I omitted (but did not know) the topic "Maximum allowable number of files on a hard disk".