OS Lab 3 - Files. Memory mapping
In the last lab, we saw how a user-space program asks the kernel for services through system calls, and what the kernel places on the stack before main() runs. In this lab, we will use system calls to work with files: open them, map them into memory, read and write them, and list the contents of directories. As an application, we will look inside the executable file of our own program.
Learning objectives
By the end of this lab you should be able to:
- explain what a file descriptor is, and open and close files with
open(2)andclose(2); - map a file into memory with
mmap(2), and allocate memory with anonymous mappings; - read and write files with
read(2)andwrite(2), and move the file offset withlseek(2); - list the entries of a directory with
readdir(3)and relate them to inode numbers; - describe the layout of an ELF executable and find a program header, such as
PT_INTERP, in it.
System call wrappers
In the last lab, we called every system call through the generic syscall(2) function, to see the system call numbers at work. In practice, this is very rarely done: for each system call, the GNU C library (glibc) provides a small C function with the same name and the same arguments, which only calls the syscall function with the appropriate number.
| System call | glibc wrapper |
|---|---|
syscall(SYS_write, fd, buf, count) |
write(fd, buf, count)
|
syscall(SYS_read, fd, buf, count) |
read(fd, buf, count)
|
syscall(SYS_close, fd) |
close(fd)
|
syscall(SYS_mmap, addr, length, prot, flags, fd, offset) |
mmap(addr, length, prot, flags, fd, offset)
|
The wrappers handle errors in the same way as syscall(2): on failure they return -1 (or another special value, as we will see for mmap(2)) and set errno. From now on, we will use the wrappers.
printf(3), puts(3) and fopen(3) belong to the C standard I/O library (stdio), which keeps its own buffers in user space and calls write(2) and read(2) only from time to time. In the labs, we will aim to talk to the kernel directly, and will use only the thin wrappers.Files
A file is an array of bytes, identified by an inode, and found by a name in a directory. The same system calls are used for regular files, for devices (such as the terminal) and for other kinds of files, which we will meet in future lessons.
open(2) and close(2)
A program cannot use a file by its name directly. It first asks the kernel to open it:
// In: <fcntl.h>
int open(const char *pathname, int flags, ... /* mode_t mode */);
The kernel finds the file, checks the permissions, and records the open file in a table of the process. It returns a small integer, the file descriptor, which identifies the open file in all the following system calls. When the program is done with it, it releases the file descriptor with:
// In: <unistd.h>
int close(int fd);
Every process starts with three file descriptors already open, inherited from the program that started it (usually the shell):
| File descriptor | Constant | Name |
|---|---|---|
| 0 | STDIN_FILENO |
Standard input |
| 1 | STDOUT_FILENO |
Standard output |
| 2 | STDERR_FILENO |
Standard error |
The flags argument must contain exactly one of the access modes O_RDONLY, O_WRONLY or O_RDWR, optionally combined (with |) with other flags:
| Flag | Description |
|---|---|
O_RDONLY |
Open for reading only |
O_WRONLY |
Open for writing only |
O_RDWR |
Open for reading and writing |
O_CREAT |
Create the file if it does not exist. Requires the third argument, mode, with the permissions of the new file (for example 0644)
|
O_EXCL |
Together with O_CREAT: fail with EEXIST if the file already exists
|
O_TRUNC |
If the file exists, cut its size to 0 |
O_APPEND |
Every write goes to the end of the file |
For example, the shell runs echo hi > notes.txt by opening the file with open("notes.txt", O_WRONLY | O_CREAT | O_TRUNC, 0644), and echo hi >> notes.txt with O_APPEND instead of O_TRUNC.
1#include <errno.h>
2#include <fcntl.h>
3#include <stdio.h>
4#include <stdlib.h>
5#include <string.h>
6#include <unistd.h>
7
8static void print_error(const char *what)
9{
10 char msg[256];
11 snprintf(msg, sizeof msg, "%s: %s\n", what, strerror(errno));
12 write(STDERR_FILENO, msg, strlen(msg));
13}
14
15int main(int argc, char **argv)
16{
17 if (argc != 2) {
18 const char usage[] = "Usage: app FILE\n";
19 write(STDERR_FILENO, usage, strlen(usage));
20 return EXIT_FAILURE;
21 }
22
23 int fd = open(argv[1], O_RDONLY);
24 if (fd == -1) {
25 print_error(argv[1]);
26 return EXIT_FAILURE;
27 }
28
29 char msg[256];
30 snprintf(msg, sizeof msg, "Opened %s as file descriptor %d\n", argv[1], fd);
31 write(STDOUT_FILENO, msg, strlen(msg));
32
33 close(fd);
34 return EXIT_SUCCESS;
35}
$ ./build/debug/app notes.txt
Opened notes.txt as file descriptor 3
$ ./build/debug/app nope
nope: No such file or directory
mmap(2)
There are two ways for a program to get to the data in a file. The first is the transparent one: the kernel makes the file appear in the virtual memory of the process, and the program uses it like any other memory. This is done with mmap(2):
// In: <sys/mman.h>
void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);
int munmap(void *addr, size_t length);
| Parameter | Description |
|---|---|
addr |
A hint for where to place the mapping. Use nullptr, and let the kernel choose
|
length |
The number of bytes to map. The kernel maps whole pages, so it is rounded up to a multiple of the page size |
prot |
, or PROT_NONE
|
flags |
MAP_PRIVATE: writes to the memory stay in this process and never reach the file. MAP_SHARED: writes go to the file, and other processes mapping it see them
|
fd |
The open file to map. It must have been opened for reading |
offset |
Where in the file the mapping starts. Must be a multiple of the page size |
On success, mmap(2) returns the address of the first mapped byte. On failure, it returns MAP_FAILED (which is (void *) -1, not nullptr) and sets errno.
mmap(2) makes the file appear in virtual memory: p[i] is the byte at offset i in the file. The pages are read from the disk only when the program first touches them.To map the whole file, we need to know its size. We will look at lseek(2) in detail below, together with read(2) and write(2). For now, it is enough to know that lseek(fd, 0, SEEK_END) returns the size of the file.
The following program prints a file, by mapping it into memory and writing the memory to the standard output:
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <unistd.h>
static void print_error(const char *what)
{
char msg[256];
snprintf(msg, sizeof msg, "%s: %s\n", what, strerror(errno));
write(STDERR_FILENO, msg, strlen(msg));
}
int main(int argc, char **argv)
{
if (argc != 2) {
const char usage[] = "Usage: app FILE\n";
write(STDERR_FILENO, usage, strlen(usage));
return EXIT_FAILURE;
}
int fd = open(argv[1], O_RDONLY);
if (fd == -1) {
print_error(argv[1]);
return EXIT_FAILURE;
}
// The offset of the end of the file is its size.
off_t size = lseek(fd, 0, SEEK_END);
if (size == -1) {
print_error("lseek");
return EXIT_FAILURE;
}
char *p = mmap(nullptr, size, PROT_READ, MAP_PRIVATE, fd, 0);
if (p == MAP_FAILED) {
print_error("mmap");
return EXIT_FAILURE;
}
// The mapping stays valid after the file descriptor is closed.
close(fd);
// The whole file is now an array of bytes in memory.
write(STDOUT_FILENO, p, size);
munmap(p, size);
return EXIT_SUCCESS;
}
mmap(2) of an empty file fails with EINVAL (Invalid argument).truncate(2), given its path, or ftruncate(2), given a file descriptor open for writing:
// In: <unistd.h>
int truncate(const char *path, off_t length);
int ftruncate(int fd, off_t length);
Mapping a file is especially convenient for files with a binary structure: a pointer into the mapping can be cast to a pointer to a C struct, and the fields read directly, as we will do with the ELF file below.
Anonymous mmap(2)
With the MAP_ANONYMOUS flag, mmap(2) maps no file at all. The fd argument must be -1 and offset must be 0, and the kernel gives the process new memory, filled with zeros:
1#include <errno.h>
2#include <stdio.h>
3#include <stdlib.h>
4#include <string.h>
5#include <sys/mman.h>
6#include <unistd.h>
7
8int main(void)
9{
10 size_t size = 1024 * 1024; // 1 MiB
11
12 char *buf = mmap(nullptr, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
13 if (buf == MAP_FAILED) {
14 char msg[256];
15 snprintf(msg, sizeof msg, "mmap: %s\n", strerror(errno));
16 write(STDERR_FILENO, msg, strlen(msg));
17 return EXIT_FAILURE;
18 }
19
20 // The new memory is filled with zeros.
21 snprintf(buf, size, "Got %zu bytes at %p, the last one is %d\n", size, (void *) buf, buf[size - 1]);
22 write(STDOUT_FILENO, buf, strlen(buf));
23
24 munmap(buf, size);
25 return EXIT_SUCCESS;
26}
$ ./build/debug/app
Got 1048576 bytes at 0xffff98360000, the last one is 0
This is how the program gets memory from the kernel: malloc(3) itself uses anonymous mappings for large allocations (by default, 128 KiB and more), and carves the smaller ones out of bigger regions.
read(2), write(2) and lseek(2)
The second way to get to the data in a file is the explicit one: the program provides a buffer in its own memory, and asks the kernel to copy data from the file into it, or from it into the file.
// In: <unistd.h>
ssize_t read(int fd, void *buf, size_t count);
ssize_t write(int fd, const void *buf, size_t count);
read(2) copies at most count bytes from the file into buf, and returns the number of bytes it copied:
- a positive number, which may be less than
count: for a regular file, only near its end; for a terminal or a pipe, whenever less data is available; 0, at the end of the file;-1, on error.
write(2) works the same way in the other direction. It may also write fewer bytes than asked (for example, when the disk is full), in which case the program must call it again for the rest.
Each open file has a current offset: the position in the file where the next read(2) or write(2) starts. It begins at 0 when the file is opened, and every read(2) or write(2) moves it forward by the number of bytes it copied. The program can also move it explicitly:
// In: <unistd.h>
off_t lseek(int fd, off_t offset, int whence);
whence |
The new offset is |
|---|---|
SEEK_SET |
offset
|
SEEK_CUR |
the current offset + offset
|
SEEK_END |
the size of the file + offset
|
lseek(2) returns the new offset. This is why lseek(fd, 0, SEEK_END), which we used in the mmap(2) example above, returns the size of the file.
open(2) calls on the same file give two independent offsets.The following program prints a file, like the mmap(2) example above, this time copying it in pieces of up to 4 KiB, until read(2) reports the end of the file:
1#include <errno.h>
2#include <fcntl.h>
3#include <stdio.h>
4#include <stdlib.h>
5#include <string.h>
6#include <unistd.h>
7
8static void print_error(const char *what)
9{
10 char msg[256];
11 snprintf(msg, sizeof msg, "%s: %s\n", what, strerror(errno));
12 write(STDERR_FILENO, msg, strlen(msg));
13}
14
15int main(int argc, char **argv)
16{
17 if (argc != 2) {
18 const char usage[] = "Usage: app FILE\n";
19 write(STDERR_FILENO, usage, strlen(usage));
20 return EXIT_FAILURE;
21 }
22
23 int fd = open(argv[1], O_RDONLY);
24 if (fd == -1) {
25 print_error(argv[1]);
26 return EXIT_FAILURE;
27 }
28
29 char buf[4096];
30 ssize_t n;
31 while ((n = read(fd, buf, sizeof buf)) > 0) {
32 if (write(STDOUT_FILENO, buf, n) == -1) {
33 print_error("write");
34 return EXIT_FAILURE;
35 }
36 }
37 if (n == -1) {
38 print_error("read");
39 return EXIT_FAILURE;
40 }
41
42 close(fd);
43 return EXIT_SUCCESS;
44}
Unlike the mmap(2) version, this one also works for an empty file. Like the mmap(2) version, it fails on a directory: read: Is a directory.
readdir(3)
As we saw in the course, a directory is a special file which maps names to inode numbers. Its entries are not read with read(2), but with the getdents64(2) system call, which fills a buffer with as many entries as fit. glibc provides the following interface on top of it:
// In: <dirent.h>
DIR *opendir(const char *name);
struct dirent *readdir(DIR *dirp);
int closedir(DIR *dirp);
opendir(3) opens the directory and returns a DIR structure, which holds the file descriptor and a buffer of entries. Each call to readdir(3) returns the next entry, calling getdents64(2) again when the buffer runs out, and returns nullptr after the last one.
// In: <dirent.h> (simplified)
struct dirent {
ino_t d_ino; // The inode number
off_t d_off; // Used internally
unsigned short d_reclen; // Used internally
unsigned char d_type; // The type of the file
char d_name[256]; // The name, '\0'-terminated
};
readdir(3) returns the next one, in the order the file system keeps them.The entries come in the order the file system stores them, which is not alphabetical, and include . (the directory itself) and .. (its parent). The d_type field holds one of DT_REG (regular file), DT_DIR (directory), DT_LNK (symbolic link), DT_FIFO (pipe), DT_CHR (character device), DT_BLK (block device), DT_SOCK (socket), or DT_UNKNOWN if the file system does not provide it.
The following program prints the names in the current directory:
#include <dirent.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
static void print_error(const char *what)
{
char msg[256];
snprintf(msg, sizeof msg, "%s: %s\n", what, strerror(errno));
write(STDERR_FILENO, msg, strlen(msg));
}
int main(void)
{
DIR *dir = opendir(".");
if (dir == nullptr) {
print_error(".");
return EXIT_FAILURE;
}
struct dirent *entry;
errno = 0;
while ((entry = readdir(dir)) != nullptr) {
write(STDOUT_FILENO, entry->d_name, strlen(entry->d_name));
write(STDOUT_FILENO, "\n", 1);
}
// nullptr means either "no more entries" or an error: only an error sets errno.
if (errno != 0) {
print_error("readdir");
return EXIT_FAILURE;
}
closedir(dir);
return EXIT_SUCCESS;
}
ls sorts the names before printing them. ls -f prints them unsorted, in the same order as the program above.To check inode numbers from the shell, use ls -i or stat:
$ ls -i notes.txt
645260 notes.txt
$ stat -c '%i %n' notes.txt
645260 notes.txt
The ELF file
Executable files and shared libraries on Linux use the ELF format (Executable and Linkable Format: elf(5)). It is a binary format, described by the structures in <elf.h>. We will only look at the parts the kernel needs to start a program.
$ readelf -h build/debug/app # the ELF header
$ readelf -l build/debug/app # the program headers
The ELF header
Every ELF file starts with the ELF header, at offset 0:
// In: <elf.h> (simplified)
typedef struct {
unsigned char e_ident[16]; // "\x7f" "ELF", then the class, byte order, ...
uint16_t e_type; // Executable, shared library, ...
uint16_t e_machine; // The architecture: x86-64, AArch64, RISC-V, ...
uint32_t e_version;
uint64_t e_entry; // The entry point (see the last lab)
uint64_t e_phoff; // File offset of the program header table
uint64_t e_shoff; // File offset of the section header table
uint32_t e_flags;
uint16_t e_ehsize; // Size of this header
uint16_t e_phentsize; // Size of one program header
uint16_t e_phnum; // Number of program headers
uint16_t e_shentsize;
uint16_t e_shnum;
uint16_t e_shstrndx;
} Elf64_Ehdr;
The first 4 bytes are always 0x7f, 'E', 'L', 'F'. The next byte, e_ident[EI_CLASS], is ELFCLASS64 for 64-bit programs.
$ readelf -h build/debug/app
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
...
Machine: AArch64
...
Start of program headers: 64 (bytes into file)
...
Size of program headers: 56 (bytes)
Number of program headers: 10
...
The program headers
The program header table is an array of e_phnum entries, each e_phentsize bytes long, starting at offset e_phoff in the file. Each entry describes a segment: a range of bytes in the file and what the kernel should do with it.
// In: <elf.h>
typedef struct {
uint32_t p_type; // The kind of segment: PT_LOAD, PT_INTERP, ...
uint32_t p_flags; // Permissions: PF_R, PF_W, PF_X
uint64_t p_offset; // Where the segment starts in the file
uint64_t p_vaddr; // Where it goes in virtual memory
uint64_t p_paddr; // Unused
uint64_t p_filesz; // Its size in the file
uint64_t p_memsz; // Its size in memory (larger for .bss)
uint64_t p_align;
} Elf64_Phdr;
Common segment types:
p_type |
Value | Description |
|---|---|---|
PT_LOAD |
1 | A range of the file to map into memory (.text, .rodata, .data, .bss, ...)
|
PT_DYNAMIC |
2 | Information for the dynamic loader: which shared libraries to load, ... |
PT_INTERP |
3 | The path of the program interpreter (the dynamic loader) |
PT_NOTE |
4 | Auxiliary information (for example, a unique build ID) |
PT_PHDR |
6 | The program header table itself |
PT_INTERP
As we saw in the last lab, most programs are dynamically linked: the code of the libraries they use (such as glibc) is not in the executable file, and must be loaded when the program starts. When the kernel finds a PT_INTERP segment in a program, it also loads the program named there, the dynamic loader, and starts it instead of the program. The loader then loads the libraries and jumps to the program's entry point.
The PT_INTERP segment is a '\0'-terminated string of p_filesz bytes (the '\0' included), at offset p_offset in the file:
$ readelf -l build/debug/app
...
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
PHDR 0x0000000000000040 0x0000000000000040 0x0000000000000040
0x0000000000000230 0x0000000000000230 R 0x8
INTERP 0x0000000000000294 0x0000000000000294 0x0000000000000294
0x000000000000001b 0x000000000000001b R 0x1
[Requesting program interpreter: /lib/ld-linux-aarch64.so.1]
...
On x86-64, the dynamic loader is /lib64/ld-linux-x86-64.so.2.
PT_INTERP segment: the kernel jumps directly to their entry point.Exercises
Write each program in its own copy of the project template. As a reminder from the first lab:
mkdir -p ~/oslab && cd ~/oslab
curl -OL https://wiki.dcae.pub.ro/images/9/93/Os_lab_skeleton.zip
unzip -d lab03 Os_lab_skeleton.zip
Use a new directory name instead of interp_mmap for each exercise.
For this lab:
- Use the glibc wrappers for the system calls (
open(2),read(2),write(2),lseek(2),mmap(2), ...), instead ofsyscall(2). - Do not use the standard I/O library: no
FILE,printf(3),puts(3),fopen(3),fread(3)and so on.snprintf(3)is still fine, and recommended, for building a string that you thenwrite(2). opendir(3),readdir(3)andclosedir(3)should be used for exercise C.
A. PT_INTERP with mmap(2)
Print the path of the dynamic loader of your own program. Open the executable file, using argv[0] as its path, map the whole file into memory with mmap(2), then walk the program headers to find the PT_INTERP segment, and print its string.
$ ./build/debug/app
/lib/ld-linux-aarch64.so.1
$ readelf -l build/debug/app | grep interpreter
[Requesting program interpreter: /lib/ld-linux-aarch64.so.1]
Your output will be different on x86-64.
- Cast the start of the mapping to
const Elf64_Ehdr *. The program headers are an array ofElf64_Phdrate_phoffbytes from the start. - Check that the file starts with
[0x7f, 'E', 'L', 'F']before looking further. - If there is no
PT_INTERPsegment, say so on standard error and exit with status 1.
Extra: argv[0] is not always trustworthy as the location of the executable. Where does the kernel offer us this information direcly?
B. PT_INTERP with read(2) and lseek(2)
Write the same program again, without mmap(2)-ing the file: read the ELF header with read(2), then use lseek(2) to move to the program headers and read them, and finally move to the PT_INTERP segment and read the string. The two programs must print the same thing.
- The size of the program header table is only known at run time (
e_phnum * e_phentsizebytes). Allocate the buffer for it with an anonymousmmap(2), andmunmap(2)it when you are done. read(2)may return fewer bytes than you asked for. Check that you got everything you need: for a regular file, getting less means the file is shorter than its headers say.
Extra: compare the system calls the two versions make, with strace ./build/debug/app (install it with sudo apt install strace). The ones at the start belong to the dynamic loader and to glibc; look for openat and what comes after it.
C. Inode numbers with readdir(3)
Write a command which takes the name of a file in the current working directory and prints its inode number. Find it by reading the directory with readdir(3), not with stat(2).
$ echo hi > notes.txt
$ ./build/debug/app notes.txt
645260
$ ls -i notes.txt
645260 notes.txt
$ ./build/debug/app nope
nope: No such file or directory
$ echo $?
1
Some things to look at:
- Make a hard link with
ln notes.txt other.txtand a symbolic link withln -s notes.txt link.txt. What inode numbers do they have, and why? - What are the inode numbers of
.and..?
Extra: also print the type of the file, from d_type: regular file, directory, symbolic link, ...
