Optimal data structures and file formats
Projects that expect large amounts of data need careful thinking about the right type of storage. Databases and existing file formats can be a good solution in some cases, but in most cases a customized and optimized file format is the better choice.
This avoids overhead due to unused functionality and compromises and workarounds due to lack of functionality. The file format and the data structures in the main memory can be designed to be space-saving and fast accessible.
Contact
256.systems is available for consulting on Big Data projects and developing individual file formats including efficient read and write routines.