TPIE is a library for manipulating large datasets on a desktop computer. TPIE provides implementations of several external memory algorithms and data structures as well as a framework for efficient and portable data processing. Currently, TPIE provides implementations of the following algorithms and data structures:
These implementations are backed by the memory manager of TPIE that allows the user to specify how much internal memory to use at most. The progress reporting framework enables reporting accurate progress metrics.
TPIE is written in C++ and depends on the Boost C++ libraries. TPIE may be compiled with GCC, Clang and MSVC, and due to its portable nature, it should be easy to port TPIE to other systems if this is required.
The TPIE Pipelining framework makes it easier to implement large I/O-efficient applications. When combining multiple I/O-efficient algorithms in an application, a traditional application (without pipelining) would require intermediate results from one component to be written to disk before being passed to the next component. The pipelining framework enables automatic pipelining of such algorithms by passing output from one component to the next in-memory, avoiding these unnecessary additional I/Os.
The pipelining extension is part of TPIE 1.1 and is being actively developed along with the rest of TPIE.
For more information, see the pipelining paper (local copy), which appeared in a short version at IEEE BigData 2017.
To use TPIE, you need the library (libtpie.a on Linux,
tpie.lib on Windows) and the development headers.
These are provided as .deb-packages that may be installed on a
Debian-based system with dpkg -i or a similar tool;
as tarballs that may be installed into /usr/local or another installation prefix;
and as Git source checkouts, such that you may compile the library for yourself.
June 11, 2013.
TPIE 1.1 (Git tag v1.1):
deb
tarball
source
June 5, 2013.
TPIE 1.1 Release Candidate 1 (Git tag v1.1rc1):
deb
tarball
source
December 14, 2012.
TPIE 1.0 (Git tag v1.0):
deb
tarball
source
October 11, 2012.
TPIE 1.0 Release Candidate 2 (Git tag v1.0rc2):
deb
tarball
source
August 2, 2012.
TPIE 1.0 Release Candidate 1 (Git tag v1.0rc1):
deb
tarball
source
If you want to hack on TPIE, check it out using the Git command:
git clone git://github.com/thomasmoelhave/tpie.git
TPIE development is driven in part by two public mailing lists.