What is this?

Black-box Reliability Challenge 2019 is a challenge and framework to test the performance of structural reliability methods. It is the first challenge in structural reliability where the reliability problems are truly black boxes for participants, meaning that their performance functions are hidden from the participants by making them accessible via the internet in the form of submitted input arguments and received output. To our knowledge, it is also the first on-line challenge in structural reliability. It is inspired by black-box optimization competitions such as BBComp. After the challenge, all the details of the reliability problems will be made publicly available, moving them to the repository for the structural reliability benchmarking framework.

We intend the current challenge to be the first iteration which we wish to improve based on the gained experience and your feedback. We are indebted to those who have already contributed to the design of the challenge and collecting reliability problems. We hope that the challenge and repository will evolve into an open, community project!


Over the last decades, a plethora of reliability analysis methods has been developed with various capabilities, accuracy, and efficiency. Yet, an objective comparison and evaluation of these methods is missing. The currently available comparative studies are limited in scope and do not allow impartial analysis as only the final results are disclosed and they allow unrealistic exploration of the reliability problem by the analyst.

Here we propose a testing framework which is designed to:

  1. deliver the long-needed, systematic and structured comparison of the performance of reliability algorithms among simplified, academic problems and real-life engineering problems;
  2. provide invaluable guidance for practical applications of reliability analysis in the domains of geotechnical engineering, hydraulic engineering, structural engineering, and earthquake engineering; and
  3. aid setting future research directions.


Phase Start date End date
Workshop [Rozsas2018] 2018-Jan-24 NA
Announcement & test problems 2018-Dec-21 NA
Challenge 2019-March-28 2019-Dec-04
Announcement of results (Results) 2019-Dec-20 NA
Follow-up workshop 2020-Feb-24 2020-Feb-25

How to participate

Everyone is welcome to join the challenge (registration will open in early 2019). The participants use their computer and their selected method to solve reliability problems with performance functions located on the organizers servers. See the Quick start for a walk through how to do this.

Who we are

The idea of the Black-box Reliability Challenge, the benchmarking framework, and the reliability problem types are the outcome of a joint discussion and follow-up sessions between the participants of a workshop held in 2018 in Delft [Rozsas2018].

The organization and implementation of the challenge are spearheaded by the Department of Structural Reliability, TNO. The members of the team have selected the reliability problems which are known exclusively to them until the announcement of the results.