PhD Seminar • Formal Methods • Pierce: A Testing Tool for Neural Network Verification SolversExport this event to calendar

Wednesday, March 6, 2024 — 3:00 PM to 4:00 PM EST

Please note: This PhD seminar will take place online.

Joseph Scott, PhD candidate
David R. Cheriton School of Computer Science

Supervisors: Professors Jo Atlee, Vijay Ganesh

We introduce Pierce, a versatile and extensible testing tool aimed at solvers for the neural network verification (NNV) problem. At its core, Pierce implements a fuzzing engine over the Open Neural Network Exchange (ONNX) — a standardized model format for deep learning and classical machine learning, and VNN-LIB — a specification standard over the input-output behavior of machine learning systems. Pierce supports the entirety of the VNN-LIB and most of ONNX v18. The API of Pierce is designed to enable users to create a variety of software testing tools, such as performance and mutation fuzzers, as well as delta debuggers, with relative ease. For example, Pierce provides a rich generator for computation graphs and specifications that allows users to easily specify a wide variety of configurations, as well as mutators that ensure that mutated computation graphs are well-formed.

Using Pierce we build a reinforcement learning (RL) driven relative performance fuzzer. Using this fuzzer, we expose performance issues in four state-of-the-art solvers, such as Marabou, ERAN, MIPVerify, and nnenum, observing up to a 13.3x times slowdown in cumulative PAR-2 score in the target solvers relative to reference solvers. Further, we leverage Pierce to create a diverse benchmark suite with 10,000 competition-grade NNV instances for the community.


To attend this PhD seminar on Zoom, please go to https://us02web.zoom.us/j/88367466621.

Location 
Online PhD seminar
200 University Avenue West

Waterloo, ON N2L 3G1
Canada
Event tags 

S M T W T F S
31
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
1
2
3
4
  1. 2024 (118)
    1. May (4)
    2. April (37)
    3. March (27)
    4. February (25)
    5. January (25)
  2. 2023 (296)
    1. December (20)
    2. November (28)
    3. October (15)
    4. September (25)
    5. August (30)
    6. July (30)
    7. June (22)
    8. May (23)
    9. April (32)
    10. March (31)
    11. February (18)
    12. January (22)
  3. 2022 (245)
  4. 2021 (210)
  5. 2020 (217)
  6. 2019 (255)
  7. 2018 (217)
  8. 2017 (36)
  9. 2016 (21)
  10. 2015 (36)
  11. 2014 (33)
  12. 2013 (23)
  13. 2012 (4)
  14. 2011 (1)
  15. 2010 (1)
  16. 2009 (1)
  17. 2008 (1)