November 26, 2012 at 14:30Large file storage for a small such company
I think that any group of developers will sooner or later face such a seemingly primitive task as
And if in the case of the first and second, many excellent tools are offered, in particular for bug tracking there are Redmine , Trac known to everyone , and for version control Subversion , Git , Mercurial , then for the competent organization of file storage you have to reinvent the wheel again.
What do I mean by a file server? Good question. Ideally, this should be a directory-type system capable of storing large volumes of binary files (pdf, doc, xls, msi, avi, etc.), allowing you to tag files for flexible search by storage, automatically index their contents and metadata, and search by to many criteria, to provide access to files both locally and through a browser (web-interface), jointly edit files, have clients for different OS, synchronizing versions from the server, but something I noticed ... Of course, such a system does not exist . Now, probably, many will object to me and point to Microsoft SharePoint Server , however, for a small company this product is unbearable due to the complexity of maintenance and the astronomical price.
Options such as Dropbox, SkyDrive and others are immediately discarded, because the management does not want to distribute the company's projects to no one knows where, again, there are restrictions on the access speed (in the case of a file server on the local network, most requests come through high-speed connections and only a part via the web -interface), as well as the size of the stored data.
What free analogues exist that partially solve the problems listed above? I have done a thorough study of this issue and am ready to share my bitter experience with you.
I will say right away that I am still in search of the best option, and those tools that I will discuss below only partially satisfy the requirements. Combining several means, maybe there is a chance to get the desired effect.
Recently, a new version (4.5) of the OwnCloud storage, synchronization and data exchange system has been released. Earlier, a wonderful article about OwnCloud already appeared on Habrahabr , which motivated me to get to know the cloud better.
An excellent article was written about the features of the installation process of OwnCloud by the BlackIce13 Habraiser . Experience installing ownCloud 6 on Debian 7 wheezy .
The first impression was simply magical: a modern web interface, the ability to view (txt, PDF, ODF) and edit (txt) files online, calendar, tasks, address book, WebDAV synchronization, search by content, and most importantly - the ability mounting local folders and external storage via FTP, Samba, etc. However, after a closer acquaintance, a huge number of bugs and glitches were discovered, as quite harmless and even funny, so a complete and irrevocable system crash. With OwnCloud, enemies do not need to use tricky DDoS attacks because this function was originally laid down by the developers in the core of the system.
Most interested in the function of mounting a remote file system through Samba. That is, a web server with a project management system and OwnCloud stood out on the local network, as well as a simple Windows file server, from which the web interface would pull up data for remote access. However, mounting through Samba did not want to function under any conditions, despite the officially announced functionality and examples on the OwnCloud website. An attempt was made to trick OwnCloud and slip its "local" folder with a mounted Samba-ball, but this drove OwnCloud into an inadequate state.
In general, the tool is quite good for home use, or if you have enough server hard disk space for the entire company. It is worth noting the developed community of OwnCloud developers, in which you will always be prompted how to deal with the next file. Add a link to their bug tracker to your favorites, as you often have to talk with these guys.
Developed by Novell, the iFolder tool also provides the capabilities we need so much — distribution to several servers, synchronization between clients, and a web interface to the repository.
The bitter fact is that iFolder on the Linux side supports openSUSE, for which the installation process consists of a single command.
For other Linux distributions, installing iFolder is an almost impossible task, despite many manuals, such as iFolderInstall . Unfortunately, I was not able to install iFolder due to the specific versions of packages that are required for the system to work and which are no longer available on the iFolder Novell website. Perhaps the readers of Habr are more successful and advanced and they will manage to find a common language with iFolder.
In addition to iFolder on the same site, Novell offers two more projects for collaboration (Kablink Teaming) and for messaging (Kablink Conferencing), but they were not so interesting to me, so I leave you to get to know them for dessert.
A rather interesting tool , written according to some hackers who are tired of synchronizing their files. It is based on the git version control system and is essentially an add-on. A separate repository is created for the file server and then SparkleShare is hung over it. Clients, working with files, synchronize them between themselves and the server by analogy with the client for Dropbox. In addition to the official site , there is a good instruction for installing and working with SparkleShare . Thanks to the foundation in the form of git, there are also disadvantages inherent in systems of this class: clients store a full local copy of the repository, which is simply impossible in case of large volumes. There is a way to "lazy" accessto the repository using git-fs, but only in read mode. Again, for non-programmers (economists, human resources), this is too high-tech solution and they will rather send documentation to each other with endless e-mails than git will use. Again, the repositories' hatred of binary files completely excludes SparkleShare from the list of possible solutions.
It would seem like a dream : a cloud file storage with support for FTP, IMAP, WebDAV, Windows NetBIOS / CIFS, SFTP / SSH, data encryption, etc. But the project has been under development for two years and there have been no official releases of the system. The authors kindly offer to join the ranks of the developers or donate something that is not a pity ... So, Habrachitateli dreaming to contribute to Cloud Storage, there is a great option to realize themselves.
Perform functions similar to Dropbox, that is, synchronization of the local and remote folders. This is not exactly what I was looking for, so I will not dwell on this decision too much. I note only the lack of a graphical interface and clients for Windows, which automatically excludes Rsync and Lsyncd from the list.
If previous products could be modestly called cloudy, then AeroFS uses this concept to the fullest. In fact, AeroFS is a p2p network that collectively stores files not necessarily using a central server! The system is fully distributed and uses sophisticated data replication algorithms. There is an opportunity to allocate a central server that would bring two positive aspects - a web-interface and additional data duplication (suddenly everyone will go on a business trip and the network will begin to experience exhaustion). Until now, I have many questions for AeroFS, the answers to which have yet to be obtained. Download release is available only on invites, so we are waiting ... Be sure to unsubscribe from the result of the deployment of AeroFS.
Thanks to umcherrel, we can get to know one more tool: AjaXplorer . Impression, like from OwnCloud, is the most positive. The developers site has the opportunity to test the demo repositoryfor which they are a huge plus. It is also worth noting the ease of installing and adding repositories. From a technical point of view, AjaXplorer is characterized by the following properties: online viewing (txt, pdf, zip, graphics, multimedia) and file editing (txt), differentiation of rights, adaptable for iOS and Android browsers, search (it’s better not to use external storage with Unfortunately), a lot of plugins for any occasion. It should also be noted that AjaXplorer can be friends with various version control systems through a plug-in, which is also important for us. External storage can be connected via Samba, FTP (S), WebDAV, IMAP, POP. And that is wonderful. Of the shortcomings, only resource intensity can be noted. On the other hand, you have to pay for everything ...
Thanks to srs2k , we learned about Amahi . What kind of beast is this? In fact, Amahi is a great platform for a home media center in the concept of a “smart home”. Media orientation immediately catches your eye: Squeezebox server, DLNA server, Gallery 2, uShare UPnP server, Jinzora and Ampache media streamers, OpenDB and VCD-db media libraries, Home Inventory household accounting, storage of phpRecipeBook recipes, torrent clients, wikis , forums, etc. It is also worth noting the “free” service of tracking dynamic IP, that is, in some cases (worth mentioning, very limited), you do not need to buy a white static IP address. It is also possible to extend functionality with plugins.
Although the installation is performed from the terminal, they will persistently and very unceremoniously install the graphic configuration panel, change the IP address of the server , enable DHCP , restart the server and generally feel at home. Of course, for inexperienced users, such a solution "out of the box" is simply necessary, but in most cases, I think this is unacceptable.
From a technical point of view, Amahi supports Samba, VPN, WebDAV (Outlook, iCal), etc. I ask for more details on the site for colorful presentations.
From the point of view of Amahi's applicability to corporate needs, the question remains open.
To summarize, I’ll say that so far we have chosen OwnCloud as a temporary alternative. According to the saying, there is nothing more permanent than something temporary, but we hope to find an alternative, since the probability that OwnCloud will get rid of absolutely all the shortcomings in the near future, unfortunately, is striving for 0. Most likely, we will switch to AjaXplorer, if the best an alternative will not appear.
This is the situation with the means for organizing file storage. Turning to the Habrasociety, I ask you to share your experience in solving such an important task for the state. I understand that there are not many absolutely free funds and, meanwhile, the authorities are even (!!!) ready to pay reasonable money for a really functional thing.
I would like to know how this problem is solved in larger companies with a set development process?
Post scriptum: Has anyone encountered employee alert systems (via email, sms, etc.) about planned or cyclical tasks with a web interface? This is probably the topic of another article. Therefore, to be continued ....
- wiki, task accounting, tickets, defects;
- version control system / repository;
- file server.
And if in the case of the first and second, many excellent tools are offered, in particular for bug tracking there are Redmine , Trac known to everyone , and for version control Subversion , Git , Mercurial , then for the competent organization of file storage you have to reinvent the wheel again.
What is needed?
What do I mean by a file server? Good question. Ideally, this should be a directory-type system capable of storing large volumes of binary files (pdf, doc, xls, msi, avi, etc.), allowing you to tag files for flexible search by storage, automatically index their contents and metadata, and search by to many criteria, to provide access to files both locally and through a browser (web-interface), jointly edit files, have clients for different OS, synchronizing versions from the server, but something I noticed ... Of course, such a system does not exist . Now, probably, many will object to me and point to Microsoft SharePoint Server , however, for a small company this product is unbearable due to the complexity of maintenance and the astronomical price.
Options such as Dropbox, SkyDrive and others are immediately discarded, because the management does not want to distribute the company's projects to no one knows where, again, there are restrictions on the access speed (in the case of a file server on the local network, most requests come through high-speed connections and only a part via the web -interface), as well as the size of the stored data.
What free analogues exist that partially solve the problems listed above? I have done a thorough study of this issue and am ready to share my bitter experience with you.
I will say right away that I am still in search of the best option, and those tools that I will discuss below only partially satisfy the requirements. Combining several means, maybe there is a chance to get the desired effect.
Owncloud
Recently, a new version (4.5) of the OwnCloud storage, synchronization and data exchange system has been released. Earlier, a wonderful article about OwnCloud already appeared on Habrahabr , which motivated me to get to know the cloud better.
An excellent article was written about the features of the installation process of OwnCloud by the BlackIce13 Habraiser . Experience installing ownCloud 6 on Debian 7 wheezy .
The first impression was simply magical: a modern web interface, the ability to view (txt, PDF, ODF) and edit (txt) files online, calendar, tasks, address book, WebDAV synchronization, search by content, and most importantly - the ability mounting local folders and external storage via FTP, Samba, etc. However, after a closer acquaintance, a huge number of bugs and glitches were discovered, as quite harmless and even funny, so a complete and irrevocable system crash. With OwnCloud, enemies do not need to use tricky DDoS attacks because this function was originally laid down by the developers in the core of the system.
Most interested in the function of mounting a remote file system through Samba. That is, a web server with a project management system and OwnCloud stood out on the local network, as well as a simple Windows file server, from which the web interface would pull up data for remote access. However, mounting through Samba did not want to function under any conditions, despite the officially announced functionality and examples on the OwnCloud website. An attempt was made to trick OwnCloud and slip its "local" folder with a mounted Samba-ball, but this drove OwnCloud into an inadequate state.
In general, the tool is quite good for home use, or if you have enough server hard disk space for the entire company. It is worth noting the developed community of OwnCloud developers, in which you will always be prompted how to deal with the next file. Add a link to their bug tracker to your favorites, as you often have to talk with these guys.
iFolder
Developed by Novell, the iFolder tool also provides the capabilities we need so much — distribution to several servers, synchronization between clients, and a web interface to the repository.
The bitter fact is that iFolder on the Linux side supports openSUSE, for which the installation process consists of a single command.
For other Linux distributions, installing iFolder is an almost impossible task, despite many manuals, such as iFolderInstall . Unfortunately, I was not able to install iFolder due to the specific versions of packages that are required for the system to work and which are no longer available on the iFolder Novell website. Perhaps the readers of Habr are more successful and advanced and they will manage to find a common language with iFolder.
In addition to iFolder on the same site, Novell offers two more projects for collaboration (Kablink Teaming) and for messaging (Kablink Conferencing), but they were not so interesting to me, so I leave you to get to know them for dessert.
Sparkleshare
A rather interesting tool , written according to some hackers who are tired of synchronizing their files. It is based on the git version control system and is essentially an add-on. A separate repository is created for the file server and then SparkleShare is hung over it. Clients, working with files, synchronize them between themselves and the server by analogy with the client for Dropbox. In addition to the official site , there is a good instruction for installing and working with SparkleShare . Thanks to the foundation in the form of git, there are also disadvantages inherent in systems of this class: clients store a full local copy of the repository, which is simply impossible in case of large volumes. There is a way to "lazy" accessto the repository using git-fs, but only in read mode. Again, for non-programmers (economists, human resources), this is too high-tech solution and they will rather send documentation to each other with endless e-mails than git will use. Again, the repositories' hatred of binary files completely excludes SparkleShare from the list of possible solutions.
Syncany
It would seem like a dream : a cloud file storage with support for FTP, IMAP, WebDAV, Windows NetBIOS / CIFS, SFTP / SSH, data encryption, etc. But the project has been under development for two years and there have been no official releases of the system. The authors kindly offer to join the ranks of the developers or donate something that is not a pity ... So, Habrachitateli dreaming to contribute to Cloud Storage, there is a great option to realize themselves.
Rsync and lsyncd
Perform functions similar to Dropbox, that is, synchronization of the local and remote folders. This is not exactly what I was looking for, so I will not dwell on this decision too much. I note only the lack of a graphical interface and clients for Windows, which automatically excludes Rsync and Lsyncd from the list.
AeroFS
If previous products could be modestly called cloudy, then AeroFS uses this concept to the fullest. In fact, AeroFS is a p2p network that collectively stores files not necessarily using a central server! The system is fully distributed and uses sophisticated data replication algorithms. There is an opportunity to allocate a central server that would bring two positive aspects - a web-interface and additional data duplication (suddenly everyone will go on a business trip and the network will begin to experience exhaustion). Until now, I have many questions for AeroFS, the answers to which have yet to be obtained. Download release is available only on invites, so we are waiting ... Be sure to unsubscribe from the result of the deployment of AeroFS.
UPD: AjaXplorer
Thanks to umcherrel, we can get to know one more tool: AjaXplorer . Impression, like from OwnCloud, is the most positive. The developers site has the opportunity to test the demo repositoryfor which they are a huge plus. It is also worth noting the ease of installing and adding repositories. From a technical point of view, AjaXplorer is characterized by the following properties: online viewing (txt, pdf, zip, graphics, multimedia) and file editing (txt), differentiation of rights, adaptable for iOS and Android browsers, search (it’s better not to use external storage with Unfortunately), a lot of plugins for any occasion. It should also be noted that AjaXplorer can be friends with various version control systems through a plug-in, which is also important for us. External storage can be connected via Samba, FTP (S), WebDAV, IMAP, POP. And that is wonderful. Of the shortcomings, only resource intensity can be noted. On the other hand, you have to pay for everything ...
UPD: Amahi
Thanks to srs2k , we learned about Amahi . What kind of beast is this? In fact, Amahi is a great platform for a home media center in the concept of a “smart home”. Media orientation immediately catches your eye: Squeezebox server, DLNA server, Gallery 2, uShare UPnP server, Jinzora and Ampache media streamers, OpenDB and VCD-db media libraries, Home Inventory household accounting, storage of phpRecipeBook recipes, torrent clients, wikis , forums, etc. It is also worth noting the “free” service of tracking dynamic IP, that is, in some cases (worth mentioning, very limited), you do not need to buy a white static IP address. It is also possible to extend functionality with plugins.
Although the installation is performed from the terminal, they will persistently and very unceremoniously install the graphic configuration panel, change the IP address of the server , enable DHCP , restart the server and generally feel at home. Of course, for inexperienced users, such a solution "out of the box" is simply necessary, but in most cases, I think this is unacceptable.
From a technical point of view, Amahi supports Samba, VPN, WebDAV (Outlook, iCal), etc. I ask for more details on the site for colorful presentations.
From the point of view of Amahi's applicability to corporate needs, the question remains open.
Conclusion
To summarize, I’ll say that so far we have chosen OwnCloud as a temporary alternative. According to the saying, there is nothing more permanent than something temporary, but we hope to find an alternative, since the probability that OwnCloud will get rid of absolutely all the shortcomings in the near future, unfortunately, is striving for 0. Most likely, we will switch to AjaXplorer, if the best an alternative will not appear.
This is the situation with the means for organizing file storage. Turning to the Habrasociety, I ask you to share your experience in solving such an important task for the state. I understand that there are not many absolutely free funds and, meanwhile, the authorities are even (!!!) ready to pay reasonable money for a really functional thing.
I would like to know how this problem is solved in larger companies with a set development process?
Post scriptum: Has anyone encountered employee alert systems (via email, sms, etc.) about planned or cyclical tasks with a web interface? This is probably the topic of another article. Therefore, to be continued ....