PostgreSQL Virtual File Descriptor — VFD Mechanism
The number of files a process can open in an operating system is limited, and the file descriptors a process can obtain are finite. For database processes, which frequently open many files, they may easily exceed the OS limit (typically 1024, configurable).
postgres@slpc:~$ ulimit -n
1024To ensure the database complies with the OS limit on file descriptors per process, you can configure the GUC parameter
max_files_per_processto set the maximum number of files a database process (e.g., a backend process) can open simultaneously.
-- max_files_per_process default value is 1000
postgres=# show max_files_per_process ;
max_files_per_process
-----------------------
1000
(1 row)To address this problem, the Virtual File Descriptor (VFD) mechanism was introduced. VFD uses an LRU (Least Recently Used) pool management strategy, dynamically opening and closing actual operating system file descriptors as needed. In PostgreSQL, all VFDs opened by a process are stored in the VfdCache array, where each element represents a virtual file descriptor owned by that process.
/*
* VFD array dynamically expands as needed
* VfdCache[0] is an unusable VFD, serving only as the list head.
*/
static Vfd *VfdCache; // virtual file descriptor array
static Size SizeVfdCache = 0; // size of VfdCache array
static int nfile = 0; // number of currently open real physical file descriptorsnfile records the number of open physical file descriptors in the VFD array. When this count exceeds the specified limit, the least recently used VFD is released (by closing its corresponding physical file) so that a new physical file descriptor can be opened.
Virtual file descriptor definition:
typedef struct vfd
{
int fd; /* real physical file descriptor, or VFD_CLOSED if not open */
unsigned short fdstate; /* VFD flag bits */
ResourceOwner resowner; /* owner, for automatic cleanup */
File nextFree; /* points to the next free VFD; the File type is an integer
* representing the index in the VfdCache array */
File lruMoreRecently; /* points to a VFD more recently used than this one */
File lruLessRecently; /* points to a VFD less recently used than this one */
off_t fileSize; /* file size (0 if not temporary) */
char *fileName; /* file name, or NULL for unused VFD */
int fileFlags; /* flags used when opening the file,
* e.g., read-only, write-only, etc. */
mode_t fileMode; /* mode specified when creating the file */
} Vfd;Each process has its own private LRU pool and a set of VFDs. When a process needs to open a file, it must request a VFD from the LRU pool. When the LRU pool is not full — i.e., the number of files opened by the process has not exceeded the system limit — the process can normally request a VFD to open a physical file. When the LRU pool is full, however, the process must first close a VFD so that opening a new file does not cause unpredictable errors by exceeding the OS limit. The LRU pool uses a replacement strategy that evicts the VFD unused for the longest time.
int BasicOpenFilePerm(const char *fileName, int fileFlags, mode_t fileMode)
{
int fd;
tryAgain:
fd = open(fileName, fileFlags, fileMode); // system call
if (fd >= 0)
return fd; /* success! */
// EMFILE: Too many open files — number of open files exceeds the limit
// ENFILE: The global file table is exhausted; no new file descriptors
// can be allocated to any process
if (errno == EMFILE || errno == ENFILE)
{
int save_errno = errno;
ereport(LOG,
(errcode(ERRCODE_INSUFFICIENT_RESOURCES),
errmsg("out of file descriptors: %m; release and retry")));
errno = 0;
// release the least recently used VFD from the LRU pool
if (ReleaseLruFile())
goto tryAgain;
errno = save_errno;
}
return -1; /* failure */
}File open call stack:
libc.so.6!__libc_open64(const char * file, int oflag) (open64.c:30)
BasicOpenFilePerm(const char * fileName, int fileFlags, mode_t fileMode) (fd.c:1068)
PathNameOpenFilePerm(const char * fileName, int fileFlags, mode_t fileMode) (fd.c:1529)
PathNameOpenFile(const char * fileName, int fileFlags) (fd.c:1494)
mdopenfork(SMgrRelation reln, ForkNumber forknum, int behavior) (md.c:494)
mdnblocks(SMgrRelation reln, ForkNumber forknum) (md.c:772)
smgrnblocks(SMgrRelation reln, ForkNumber forknum) (smgr.c:557)
table_block_relation_size(Relation rel, ForkNumber forkNumber) (tableam.c:639)
table_relation_size(Relation rel, ForkNumber forkNumber) (tableam.h:1840)
RelationGetNumberOfBlocksInFork(Relation relation, ForkNumber forkNum) (bufmgr.c:2979)How to open different segment files of a table file? Use the _mdfd_openseg function. It returns an MdfdVec structure representing the segment number of this segment file and the virtual file descriptor number (the index in the VfdCache array).
typedef struct _MdfdVec
{
File mdfd_vfd; /* virtual file descriptor number */
BlockNumber mdfd_segno; /* segment file number */
} MdfdVec;The _mdfd_openseg function implementation:
static MdfdVec *_mdfd_openseg(SMgrRelation reln, ForkNumber forknum,
BlockNumber segno, int oflags)
{
MdfdVec *v;
// get the full path of the segment file: psprintf("%s.%u", relpath, segno)
char *fullpath = _mdfd_segpath(reln, forknum, segno);
/* open the file */
File fd = PathNameOpenFile(fullpath, O_RDWR | PG_BINARY | oflags);
pfree(fullpath);
if (fd < 0)
return NULL;
// adjust SMgrRelationData->md_seg_fds array length
// update md_num_open_segs value
_fdvec_resize(reln, forknum, segno + 1);
/* fill the entry */
v = &reln->md_seg_fds[forknum][segno];
v->mdfd_vfd = fd; // virtual file descriptor number
v->mdfd_segno = segno; // segment number
/* all done */
return v;
}How to iterate over all segment files of a table? For example, when obtaining the page count of a table, you need to know how many segment files the table has. The database does not have metadata about the number of segment files; it must determine this based on the opened segment files: get the last opened segment number (segno), check if it is the last segment (if the segment size is less than 1G, it is the last segment). If not, it may not be the last segment, so continue checking: increment segno, get the size of the next segment, and repeat until a segment’s size is less than 1G, which indicates the last segment. The segno value at that point is the number of segment files.
The core principle of the VFD mechanism: dynamically open and close actual operating system file descriptors as needed. In reality, the maximum number of files a process can open is bounded by the OS limit. However, under the VFD mechanism, when a process requests to open a file and the LRU pool is full (reaching the process’s maximum file descriptor limit), the VFD mechanism selects the least recently used VFD based on the LRU strategy, closes the corresponding physical file, releases the resources, and then retries opening the new file.
For example, in the following file read operation, FileAccess is first called to reopen the physical file descriptor because it may have been closed due to the LRU strategy. When executing FileAccess, the LRU eviction strategy may be triggered because the maximum file descriptor count has been exceeded, choosing the least recently used VFD to close.
This closing does not cause anomalies because PostgreSQL uses a process model: when a physical file descriptor is closed in the current process, read/write operations on that file will not happen simultaneously. A single postgres process can only read/write one file at any given moment, but it can have multiple open physical file descriptors at the same time.
int FileRead(File file, char *buffer, int amount, off_t offset,
uint32 wait_event_info)
{
int returnCode;
Vfd *vfdP;
DO_DB(elog(LOG, "FileRead: %d (%s) " INT64_FORMAT " %d %p",
file, VfdCache[file].fileName,
(int64) offset,
amount, buffer));
returnCode = FileAccess(file); // re-acquire the file descriptor
// in case it was closed by LRU
if (returnCode < 0)
return returnCode;
vfdP = &VfdCache[file];
retry:
pgstat_report_wait_start(wait_event_info);
returnCode = pg_pread(vfdP->fd, buffer, amount, offset);
pgstat_report_wait_end();
if (returnCode < 0)
{
/* OK to retry if interrupted */
if (errno == EINTR)
goto retry;
}
return returnCode;
}
/* returns 0 on success, -1 on re-open failure (with errno set) */
static int FileAccess(File file)
{
int returnValue;
DO_DB(elog(LOG, "FileAccess %d (%s)",
file, VfdCache[file].fileName));
/*
* Is the file open? If not, open it and put it at the head of the LRU
* ring (possibly closing the least recently used file to get an FD).
*/
if (FileIsNotOpen(file)) // if the physical file is not open
{
returnValue = LruInsert(file); // open the file and insert into the LRU
if (returnValue != 0)
return returnValue;
}
else if (VfdCache[0].lruLessRecently != file)
{
/*
* We now know that the file is open and that it is not the last one
* accessed, so we need to move it to the head of the Lru ring.
*/
// recently accessed, update its position in the LRU
Delete(file);
Insert(file);
}
return 0;
}