Call for Artifacts
This message invites you to submit your work for Artifact Evaluation. Artifact Evaluation is not only an important step of good science, but it will also gain you the badges that mark your research as conforming to the highest standards of good science: research that is not only reproducible, but also makes its artifacts available to the community to study, build on, and compare against.
Specifically, the output of the process will be:
- the paper will be awarded the corresponding badges:
- Artifacts Available – the artifact’s codebase is open-sourced and available online.
- Artifacts Functional – the artifact is functional.
- Results Reproduced – the artifact’s results match the ones in the corresponding paper.
All artifacts will be assessed by the ACM/IFIP MIDDLEWARE 2025 Artifact Evaluation Committee (AEC) to verify if the respective badges can be awarded. For that purpose, the authors need to submit a package consisting of a comprehensive reproducibility package (scripts, data, code and documentation), or at least only the artifacts if aiming for availability without reproducibility. All successful papers will be recognized during the MIDDLEWARE 2025.
Submission Instructions
Please submit your artifact at: Submission Instructions
Note that when submitting your code in the AEC call, you need to Specify in the Abstract all the badges that you need your artifact to be evaluated against, i.e.
- Artifacts Available badge
- Artifacts Functional badge
- Results Reproduced badge
The AEC will only test your artifact for badges that you mention in the abstract.
Note that the goal of AEC is to verify and reproduce all papers accepted at MIDDLEWARE 2025. To that end, we would like to ensure that (a) you submit the most comprehensive possible version of your artifacts and (b) you make yourselves available to the reviewers if needed for compiling, deploying, running the experiments, and verifying the results.
A few additional important notes about your artifact entry:
- Include detailed readme and scripts that can (i) run experiments, (ii) collect and parse data, and (iii) visualize the results by plotting your paper graphs. If some data collection needs to be manual (ideally, it should not), please add detailed step-by-step instructions on how to do so.
- Before submitting your package, please try to recreate all your guidelines to a fresh machine–that way, you will probably face any problems the reviewers might face, and you can minimize them prior to submission.
- If the submitted scripts do not reproduce some figures, add a concrete explanation as to why the core thesis of the paper can be verified without the missing experiments.
- Include a detailed description of the hardware used. If your experiments require large or specialized hardware, we may be unable to test them and it will be helpful if you can provide AEC members access to your hardware.
Details on how to package your material, how the process works, as well as tutorials and best practices for reproducibility, are available
here.