Another my entry from the sandbox, if I have time, I will translate the remaining parts.

This is the translation of the first part of the article about writing VFS (virtual file system) in c ++ which I found a long time ago. I hope you will like it. :)

Introduction

When I started developing my 3D engine, I realized that I needed something like a file system. Not a simple archive, but its own virtual file system that supports compression, encryption, has fast access times, and so on.

And I decided to lay out my achievements so that you would not have to reinvent the wheel. This article will be divided into 2 parts. The first is what you are reading now and the structure of VFS will be described here. In the second part, we will actually write the VFS itself (it will be quite large).

So what is VFS?

VFS is a file system similar to those in Windows (fat32, ntfs, etc.). The main difference between VFS and the real file system is that VFS uses the real file system within itself.

Functionality

Let's name some of the features of VFS:
Quick access time.
Several archives instead of a huge number of small files
.
Pluggable Encryption and Compression (PEC) debugging.
Security (MD5 key is stored inside the vfs file so any changes to the archive will be noticed right there)
Several root paths

Now, as we have listed list of features we can go to the design phase

Basic design

Let's get started: We will have a main interface with 16 functions. I will show you these functions first, then we will discuss what they are for and in the next part we will write them. These functions actually do nothing but start / unload some structures that we will need later

#define VFS_VERSION 0x0100

#define VFS_PATH_SEPARATOR '\\'



void VFS_Init();

void VFS_Shutdown();




Filters

typedef BOOL (* VFS_FilterProc )( LPCBYTE pIn, DWORD dwInCount, LPBYTE* ppOut, DWORD* pOutCount );



struct VFS_Filter

{

string strName;


string strDescription;

VFS_FilterProc pfnEncodeProc;

VFS_FilterProc pfnDecodeProc;

};



void VFS_RegisterFilter( VFS_Filter* pFilter );

void VFS_UnregisterFilter( VFS_Filter* pFilter );

void VFS_UnregisterFilter( DWORD dwIndex );


DWORD VFS_GetNumFilters();

const VFS_Filter* VFS_GetFilter( DWORD dwIndex );


Well, that is already a little trickier. VFS_Filter is a kind of routine that processes data. You can for example write VFS_CryptFilter which encrypts / decrypts data. Do you have that? The filter consists of something like a preliminary processor (pfnEncodeProc procedure) for data written to the archive and a post-processor (pfnDecodeProc procedure) for reading data from the archive. These filters implement the Pluggable Encryption and Compression mentioned above, so you can assign one or more filters for each vfs file that you use. If you are a little confused, then look at Figure 1, which is a filter diagram.



You see, coding / decoding procedures manipulate data flow in both directions: from archive to memory and from memory to archive.

For a better understanding of filters, let's write a simple filter (in any case, keep in mind that we will not be able to check the filter, because we will implement VFS later). Our filter will add 1 to each byte. The only point of this filter is that opening a vfs file can be made more difficult.

VFS_Filter ONEADD_Filter =

{

"ONEADD",

"This Filter adds 1 to each Byte of the Data. It doesn't really make sense, "

" but anyway, this is just a test, you know",


ONEADD_EncodeProc,

ONEADD_DecodeProc

};



BOOL ONEADD_EncodeProc( LPCBYTE pIn, DWORD dwInCount, LPBYTE* ppOut, DWORD* pOutCount )

{

assert( ppOut );

assert( pOutCount );




// Allocate the Memory.

*ppOut = new BYTE[ dwInCount ];



// Perform a For-Loop through each Byte.

for( DWORD dwIndex = 0; dwIndex < dwInCount; dwIndex++ )

{


( *ppOut )[ dwIndex ] = ( BYTE )( pIn[ dwIndex ] + 1 );

}



// Set the Output Count.

*pOutCount = dwInCount;

}



BOOL ONEADD_DecodeProc( LPCBYTE pIn, DWORD dwInCount, LPBYTE* ppOut, DWORD* pOutCount )


{

assert( ppOut );

assert( pOutCount );



// Allocate the Memory.

*ppOut = new BYTE[ dwInCount ];



// Perform a For-Loop through each Byte.


for( DWORD dwIndex = 0; dwIndex < dwInCount; dwIndex++ )

{

( *ppOut )[ dwIndex ] = ( BYTE )( pIn[ dwIndex ] - 1 );

}



// Set the Output Count.


*pOutCount = dwInCount;

}



Root path functions

We discussed root path functions not so long ago, remember? If not, no problem. I said that we want a function to use several root paths, that is, several search paths, such as the program installation directory, optical disk and network drive. The following functions will be used to accomplish this: Everything is easy enough, isn't it?

void VFS_AddRootPath( LPCTSTR pszRootPath );

void VFS_RemoveRootPath( LPCTSTR pszRootPath );

void VFS_RemoveRootPath( DWORD dwIndex );

DWORD VFS_GetNumRootPaths();

LPCTSTR VFS_GetRootPath( DWORD dwIndex );

A few simple but necessary things

The following 4 functions are quite simple:

void VFS_Flush();

This function will close all open vfs files that are not accessed. You may wonder why a vfs file that is not accessed does not close automatically, but if we do, we had to reanalyze the vfs file every time we open or close the file. Look at this code for a further explanation: You see, we would have to open the file archive twice. A good place to call VFS_Flush () in a game may be when all level data has loaded. But here are the last 3 main functions:

// Reference Count is 1. -> Load + Parse!!!

DWORD dwHandle = VFS_File_Open( "Bla\\Bla.Txt" );




// Reference Count is 0. -> Close!!!

VFS_File_Close( dwHandle );



// Reference Count is 1. -> Load + Parse!!!

dwHandle = VFS_File_Open( "Bla\\Bla.Txt" );




// Reference Count is 0. -> Close!!!

VFS_File_Close( dwHandle );





struct VFS_EntityInfo

{

BOOL bIsDir; // Is the Entity a Directory.


BOOL bArchived; // True if the Entity is located in an Archive.

string strDir; // like Models/Sarge/Textures

string strPath; // like Models/Sarge/Textures/Texture1.Jpg

string strName; // like Texture1.Jpg

DWORD dwSize; // The Number of Files and Subdirectories for a

Directory.


};



BOOL VFS_Exists( LPCTSTR pszPath );

void VFS_GetEntityInfo( LPCTSTR pszPath, VFS_EntityInfo* pInfo );

DWORD VFS_GetVersion();



The first function is currently checking if an object exists with the pszPath path. You see that this is a fairly simple thing, but in C (+ +) the standard library does not contain such a function (I know they have functions like stat (), but I just want things like exists ()). The second function is something like stat (), it returns information about the

object. The last function has nothing to do with information about the object, this function simply returns the current version of VFS. Nothing special (Seriously, it just returns the VFS_VERSION constant ;-)

File interface

We looked at simple things. But don’t worry, there are still a couple of easy things ahead. In fact, everything described in this part of the article is easy. Unfortunately, if you want something more complicated, you have to wait for the next part of this article ... ;-)

Well, here they are, the functions of the file interface:

#define VFS_INVALID_HANDLE ( ( DWORD ) -1 )



// The VFS_File_Open/Create() Flags.

#define VFS_READ 0x01

#define VFS_WRITE 0x02




// The VFS_File_Seek() Flags.

#define VFS_SET 0x00

#define VFS_CURRENT 0x01

#define VFS_END 0x02



// Create / Open / Close a File.

DWORD VFS_File_Create( LPCTSTR pszFile, DWORD dwFlags );

DWORD VFS_File_Open( LPCTSTR pszFile, DWORD dwFlags );

void VFS_File_Close( DWORD dwHandle );




// Read / Write from / to the File.

void VFS_File_Read( DWORD dwHandle, LPBYTE pBuffer, DWORD dwToRead, DWORD* pRead = NULL );

void VFS_File_Write( DWORD dwHandle, LPCBYTE pBuffer, DWORD dwToWrite, DWORD* pWritten = NULL );



// Direct Data Access.

LPCBYTE VFS_File_GetData( DWORD dwHandle );



// Positioning.

void VFS_File_Seek( DWORD dwHandle, LONG dwPos, DWORD dwOrigin = VFS_SET );


LONG VFS_File_Tell( DWORD dwHandle );

DWORD VFS_File_GetSize( DWORD dwHandle );



// Information.

BOOL VFS_File_Exists( LPCTSTR pszFile );

void VFS_File_GetInfo( LPCTSTR pszFile, VFS_EntityInfo* pInfo );

void VFS_File_GetInfo( DWORD dwHandle, VFS_EntityInfo* pInfo );



There are just a few things worth noting. Firstly, the dwFlags parameter for VFS_File_Create () and VFS_File_Open () can be either VFS_READ or VFS_WRITE or both at once, which means read, write or read / write access. Secondly, these two functions return a handle, which is used by almost all other functions, as a kind of pointer. We will not use pointers, but we will use handle as they provide one more level of abstraction. I would also like to mention the fact that our functions will load the entire file into memory. This is necessary due to the peculiarity of the filters (since they need memory to process). You can access this memory directly with VFS_File_GetData (). Well, the rest in things you should know, thanks to the standard I / O library.

The interface of our library

This may be the place you were expecting, starting from some lines or it would be better to say pages (and when we talk about expectation: what I did not expect is the fact that this is already page 7 or so. WOW!).

Anyway, let's continue: A very simple interface, isn't it? Just the usual things for an archive file. And now you finally see an application for the filter functions that we saw before. You can apply filters using VFS_Archive_Set / GetUsedFilters ().

// Create / Open / Close an Archive.

DWORD VFS_Archive_Create( LPCTSTR pszArchive, const VFS_FilterNameList& Filters, DWORD dwFlags );

DWORD VFS_Archive_CreateFromDirectory( LPCTSTR pszArchive, LPCTSTR pszSrcDir,


const VFS_FilterNameList& Filters, DWORD dwFlags );

DWORD VFS_Archive_Open( LPCTSTR pszArchive, DWORD dwFlags );

void VFS_Archive_Close( DWORD dwHandle );



// Set the Filters used by this Archive.

void VFS_Archive_SetUsedFilters( DWORD dwHandle, const VFS_FilterNameList& Filters );

void VFS_Archive_GetUsedFilters( DWORD dwHandle, VFS_FilterNameList& Filters );




// Add / Remove Files to / from the Archive.

void VFS_Archive_AddFile( DWORD dwHandle, LPCTSTR pszFile );

void VFS_Archive_RemoveFile( DWORD dwHandle, LPCTSTR pszFile );



// Extract the Archive.

void VFS_Archive_Extract( DWORD dwHandle, LPCTSTR pszTarget );



// Information.

void VFS_Archive_GetInfo( DWORD dwHandle, VFS_EntityInfo* pInfo );

void VFS_Archive_GetInfo( LPCTSTR pszArchive, VFS_EntityInfo* pInfo );




Folder Interface

This is the latest VFS interface. It contains 3 functions that should be understood without explanation as I think. Functions 1 and 2 are quite simple (as if they are used for files). Function 3 acts like a DOS command. ;-)

// Information.

BOOL VFS_Dir_Exists( LPCTSTR pszDir );

BOOL VFS_Dir_GetInfo( LPCTSTR pszDir, VFS_EntityInfo* pInfo );




// Get the Contents of a Directory.

vector< VFS_EntityInfo > VFS_Dir_GetContents( LPCTSTR pszDir, BOOL bRecursive = FALSE );

Have a little chat

That's all. We have finished the first part of the article. I do not believe (me too :) approx. translator). But the most difficult is yet to come:

We need to WRITE VFS !!!

Download article_vfs_header.h