Skip to content

reproducibility.md

Latest commit

 

History

History
executable file
·
71 lines (61 loc) · 2.73 KB

reproducibility.md

File metadata and controls

executable file
·
71 lines (61 loc) · 2.73 KB
Raw

Manual Reproducibility (Optional)

Here we describe the inner-workings of the evaluate.sh script that compiles and evaluates Memento filter. The goal is to provide clarity into how the script works and allow for trouble-shooting in the event of an error.

Building

To compile the files needed to run the experiments, clone the repository including all submodules and compile the code:

git clone --recurse-submodules -j8 https://github.com/n3slami/Memento_Filter.git
cd Memento_Filter
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j8

Setting the Environment

Create a folder wherever desired (we assume it to be alongside the Memento_Filter path) to contain the artifacts of the experiments and move into it:

mkdir ../../paper_results && cd ../../paper_results

Then, download the books_200M_uint64, fb_200M_uint64 and osm_cellids_200M_uint64 datasets (they will be placed in the real_datasets subfolder) and generate the workloads:

bash ../Memento_Filter/bench/scripts/download_datasets.sh
bash ../Memento_Filter/bench/scripts/generate_datasets.sh ../Memento_Filter/build real_datasets

The generated workloads will be in the workloads subfolder.

Running the Experiments

Now you can execute the tests:

bash ../Memento_Filter/bench/scripts/execute_tests.sh ../Memento_Filter/build workloads

The results will be in the results subfolder.

Note that the experiments will take a long time to complete.

Finally, to generate the figures, run the Memento_Filter/bench/scripts/plot.py script. The figures will be saved in the figures subfolder as pdf files.

At the end of the process, the paper_results folder will have the following structure:

  • real_datasets: contains the original books_200M_uint64, fb_200M_uint64 and osm_cellids_200M_uint64 datasets
  • workloads: contains the generated datasets and workloads for each test
  • results: contains the results of the experiments in csv format
  • figures: contains the figures of the paper

Figures and Tables

The following results of the paper are generated by the plot.py script in the paper_results/figures folder:

  • Fig. 8. corr_test_(Fig_8).pdf
  • Fig. 9. fpr_test_all_(Fig_9).pdf and its accopanying table table_(Fig_9).tex
  • Fig. 10. true_queries_test_(Fig_10).pdf
  • Fig. 11. constr_time_test_(Fig_11).pdf
  • Fig. 12. vary_memento_test_(Fig_12).pdf

Expandability Results

To manually generate the figures pertaining to the expandable Memento filter, repeat the instructions above after switching to the expandable branch. The following figures will then be generated by plot.py and placed in the paper_resutls/figures directory:

  • Fig. 13. expansion_test_(Fig_13).pdf
  • Fig. 14. b_tree_test_(Fig_14).pdf