Fast lossless compression of time correlated 16 bit data
Modern instruments often deliver several GigaBytes per second of data. To save hard disk space, these data can be compressed. Conventional compression algorithms, however, are not fast enough and would slow down the reading of the data. A compression algorithm, which is faster than the read and write speed of hard disks, saves space and even allows data to be read and written at a higher speed than that of the hard disk.
Example: Data reaches the computer at 2 GB/s and is to be written to a hard disk, which has a write speed of only 1 GB/s. If a compression algorithm can compress at more than 2 GB/s and compresses the data down to half the size or less, it has the same effect as a hard disk that is twice as fast and twice as large.
fc16 (fast compression 16 bit)
Since digitized analog signals often have 12 or 16 bit resolution, the program fc16 was developed, which is specially tailored to such data. Typical measurement data, with medium noise and occasional spikes, are compressed at 2-3 GB/s per core (on a low-power Intel i5 processor) lossless to half the original size. This is possible by a sophisticated algorithm that is adapted to the hardware and uses AVX2 (Advanced Vector Extensions 2). Benchmarks show that fc16 is currently the best compression program for 16 bit data (see benchmark results below).
Contact
If there is interest in fc16 or other fast compression algorithms, 256.systems may be contacted.
Benchmark 1: Gerda
Left: compression speed.
Center: decompression speed.
Right: compression rate..
Benchmark 2: MPIK experiments
Left: mean compression rate compared to mean compression speed.
Right: mean compression rate compared to mean decompression speed.
Benchmark 3: Overall performance
