mmap with files >2GB

[Eric99]

I've been trying to get mmap to work on a 3GB file (Linux Ubuntu 12.04), but it keeps returning MAP_FAILED. If I ignore it, I eventually get a segmentation fault, so I'm sure something is wrong somewhere.

I'm compling with 64 bit file options:
gcc -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 test.c

Here's an example I made for testing:

#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <inttypes.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <stdint.h>


int main(int argc, char **argv)
{
    uint8_t *base;

    int writable=0;
 
    if(argc != 2)    
        return 1;
 
    int file=0;
        if((file=open(argv[1],O_RDONLY)) < -1)
            return 1;
 
    struct stat fileStat;
    if(fstat(file,&fileStat) < 0)    
        return 1;
 
    printf("Information for %s\n",argv[1]);
    printf("---------------------------\n");
    printf("File Size: \t\t%" PRIuPTR " bytes\n", (uintptr_t) fileStat.st_size);
    printf("Number of Links: \t%d\n",fileStat.st_nlink);
    printf("File inode: \t\t%" PRIuPTR " bytes\n", (uintptr_t) fileStat.st_ino);
 
    printf("File Permissions: \t");
    printf( (S_ISDIR(fileStat.st_mode)) ? "d" : "-");
    printf( (fileStat.st_mode & S_IRUSR) ? "r" : "-");
    printf( (fileStat.st_mode & S_IWUSR) ? "w" : "-");
    printf( (fileStat.st_mode & S_IXUSR) ? "x" : "-");
    printf( (fileStat.st_mode & S_IRGRP) ? "r" : "-");
    printf( (fileStat.st_mode & S_IWGRP) ? "w" : "-");
    printf( (fileStat.st_mode & S_IXGRP) ? "x" : "-");
    printf( (fileStat.st_mode & S_IROTH) ? "r" : "-");
    printf( (fileStat.st_mode & S_IWOTH) ? "w" : "-");
    printf( (fileStat.st_mode & S_IXOTH) ? "x" : "-");
    printf("\n\n");
 
    printf("The file %s a symbolic link\n\n", (S_ISLNK(fileStat.st_mode)) ? "is" : "is not");
  
    fprintf(stderr,"file=%d flags=%d\n",file,writable);
    if (file == -1) {
        printf("Cannot open file %s\n", argv[1]);
        return 1;
    }

    if (fstat(file,&fileStat) == -1) {
        printf("Cannot get size of file %s\n", argv[1]);
        return 1;
    }
    if (fileStat.st_size == 0) {
        printf("file %s is an empty file\n", argv[1]);
        return 1;
    }
    
    base = mmap(NULL, fileStat.st_size, PROT_READ | (PROT_WRITE * writable),
            MAP_SHARED, file, 0);
    fprintf(stderr,"base: \t\t%" PRIuPTR " \n", (uintptr_t) base);
    if (base == MAP_FAILED) {
        fprintf(stderr, "Cannot map file %s into memory\n", argv[1]);
        return 1;
    }

    return 0;
}

Running it on a test file >2GB produces this example output

1 2 3 4 5 6 7 8 9 10

Information for largefile.iso

File Size: 3019309056 bytes Number of Links: 1 File inode: 14948955 bytes File Permissions: -rw-rw-r-- The file is not a symbolic link file=3 flags=0 base: 4294967295 Cannot map file largefile.iso into memory

The line base == MAP_FAILED is comparing the 4294967295 to (void *) -1

Can someone clue me as to how to properly handle this large file with mmap? Am I running up against a physical memory limit or am I making some kind of integer size error?

Thanks in advance!

[Athar]

Am I running up against a physical memory limit

If it's a 32-bit program, yes. Your address space is 4 GB, of which you can use at most 3 GB.

[Eric99]

I think you are right. I did some memory allocation tests on my 32-bit machine and found the max size I could allocate was 2.8G.

How do I handle files greater than 2.8G? Do I need to load the file in chunks? Or is there another method that can handle paging large files?

[Athar]

Either map it in smaller chunks or just use regular file read/write operations instead of mapping it.

[Eric99]

Strange I thought that mmap used some kind of virtual memory map and didn't put it into RAM.

I think fseek methods will be slow, but I'll try rewriting the code and see.

[Athar]

mmap maps the file into your virtual address space, which has a size of 2^32 bytes=4 GB in a 32-bit application.
The contents of the file do not necessarily reside in physical RAM.