Since the beginning of the project we have been building the requirements directly from the Grant Agreement, the document the consortium signed with the European Commission. We analysed it sentence by sentence, classifying every commitment as functional or non-functional. Each requirement is traceable to its source text, and together they cover all ten of the project’s objectives. We made a point of completing the full register before the consortium meeting in Valencia, hosted by UPV.
The End User Advisory Board (EUAB) handled the validation, the group includes six organizations, such as SOCs, CSIRTs, and IT security teams from various countries. We collected their opinions through an anonymous questionnaire this summer; their answers has been linked to specific rows in our register allowing us to turn the results into clear decisions for every requirement. We aggregate the answers and interpret them qualitatively rather than numerically.
We used open AI models to assist with extraction and mapping though a specific pipeline that splits our documentation into traceable passages, extracting candidate requirements. These candidate requirements are further processed into a structured record in order to assemble the final register. Along each steps we have ran consistency checks. These models run on our own hardware to ensure no data leaves the company. We didn’t rely on automation alone, as a person reviewed every step before anything entered the register.
One of the most interesting outcomes from the Valencia meeting was not really technical, but about restraint on how AI is used. AI is shaping the future of automation, and with it come well-known problems and limitations, sycophancy and hallucination to name two. So it is better to have a platform that recommends and explains under human validation than an automatic tool that can go rogue. But it is not only about control, it is also about quality. What the stakeholders’ ideas converged on was how to make sure the actions taken are legitimate, and how we can be sure the AIs are not lying to us. Thanks to the work we have already started, we will adapt our assets to handle and mitigate exactly these scenarios.
If you run a SOC or a CSIRT and want to follow the platform, visit vantageproject.eu.
Funded by the European Union (Project 101249800 — VANTAGE — DIGITAL-ECCC-2024-DEPLOY-CYBER-07).