Anomaly Detection at 40 MHz

Introduction

The Compact Muon Solenoid (CMS) Experiment at CERN produces massive amounts of data every second ($ \sim \mathcal{O}(40)$ TB/s), most of which is not practically stored or even analyzedanalyzed entirely. To solve this issue, several filtration steps or triggers have been developed that operate hierarchically to filter the amount of incoming data and reduce the number of events (collision events) from 40 MHz to a manageable 1 kHz. Two triggering steps achieve this:

  • Level 1 Trigger (L1T): This is the first trigger that operates mainly on Field Programmable Gate Array (FPGAs) and is responsible for reducing the rate from 40 MHz to 100 kHz.
  • High Level Trigger (HLT): This is the second trigger that operates on CPU and GPU farms and uses a complete physics analysis (reconstruction) to make the trigger decision. HLT reduces the data rate from 100 kHz to 1 kHz.
CMS Trigger System
Overview of the CMS Trigger Hierarchy

Given the massive amount of data, a rough resolution (portion of the entire data) of the events is used to make a decision for the L1T, making physics reconstruction algorithms impossible. Hence, L1T is primarily a cut-based trigger, where thresholds for physics observables are set as criteria for trigger conditions. Even though L1T works exceptionally well and has been heavily developed and tuned for quite some time, it remains heavily biased by the physical intuitions and assumptions used to build it.

The Standard Model (SM) of physics, being one of the most successful theories in modern physics, still fails to account for several phenomena, such as the existence of gravity as a distinct entity from the other fundamental forces, CP Violation, and the existence of dark matter, among others. All these discrepancies have led to theories that try to incorporate these phenomena. For the verification of such theoretical Beyond Standard Model (BSM), the discovery of new BSM particles is necessary. Given that BSM is an uncharted territory that might or might not follow the physical assumptions of the existing L1 Trigger, calls for the development of a new, unbiased trigger, completely from scratch, that can efficiently filter data and remain based entirely on statistical properties rather than physics-based intuitions.

AXOL1TL

Anomaly eXtraction Online L1 Trigger Lightweight in L1 Global Trigger (AXOL1TL) is a trigger based on an anomaly detection algorithm that is deployed as a part of the L1T, which takes in a subset of the Level-1 (L1) objects (measurables) and tries to assign an anomaly score for every event. Events are selected based on their anomaly score, considering fixed-rate budgets.

Working of the existing baseline

AXOL1TL Baseline Architecture
Training and deployment of the baseline (v4) model

The diagram above illustrates the training and deployment of the baseline (v4) model. In this report, we discuss how we can improve on this architecture to make the trigger more accurate (in detection), robust (to changing detector conditions), and interpretable (for analysis).

The above method exhibits a few problems:

  1. The loss (used to train) the original Variational AutoEncoder (VAE) for anomaly detection (AD) does not align well with the final optimisation goal of the model, given that the headless (without Decoder) model no longer represents an autoencoder. Using latent vectors (in this case, their L2 norm) for anomaly detection can be challenging and unpredictable.
  2. VAEs do not guarantee any form of explicit clustering in the embedding space where similar samples strictly lie close to each other. Reconstruction error of a point might hint toward it being anomalous, but the same is not always valid for points that are close to each other (or have similar L2 norm).
  3. Taking the L2 norm of the latents explicitly assumes that the anomalous samples will always be strictly away from the center, but that might not always be true.

To address these problems, we modify the VAE. The modification will be discussed and developed over the course of the following few sections.

Self-Supervised Learning, metric and clustering.

Self-Supervised Learning (SSL) is a ML paradigm that focuses on the production of information-rich feature vectors from a dataset without explicitly using any labels. To achieve this, SSL models are trained on a surrogate task that utilizes pseudo labels from training and loss computation. For this study, we primarily focus on the VICReg, a well-celebrated SSL technique.

VICReg

VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning work by training on the surrogate task of generating embedding spaces that are invariant to specific augmentation spaces. This is achieved by using a Siamese-like setup, where two views of the same event are used, but both have been augmented to different degrees. The Euclidean distance in the embedding space between these two views is then minimised using gradient descent.

VICReg Training Procedure
VICReg Training Procedure Bardes et al.

Along with the MSE loss that acts across the two projection vectors Z, to ensure stability against collapse and the non-triviality of the embedding space, we have covariance and variance regularization terms. The covariance component prevents the embedding dimensions from capturing the same information. In contrast, the variance component prevents all embedding vectors in a batch from collapsing to a single point in the representation space.

Metric

When working with latent spaces, we are often interested in defining a metric on them. The definition of metric or distance in the embedding space provides insight into the exact nature of the embedding space. The metric on the latent space of an autoencoder is often ill-defined due to its erratic nature. Even though VAEs have partially solved this problem by making the space continuous through variational sampling, where two similar samples have similar embeddings due to the variational step, which facilitates the definition of local structures, the metric definition on a global level remains ill-defined. VICReg addresses this by introducing augmentations that often introduce inductive biases within the system, making the global structures clearer.

To get a stronger intuition of this idea, we will take a real-life example. We consider a cylindrical system ($r\cos{t}$,$r\sin{t}$,$h$) where the rotation across the axis (of the cylinder) is invariant to our use cases. We can think of it as a coordinate change. We fix the radius to 1 and the dimension of the embedding space to 2 for easier visualization. The input dataset looks like

Input Dataset Visualization

When we train a VAE and a VICReg encoder (with the same architecture) on this dataset, we observe the following latent space.

Latent Space Comparison
In the above diagram we chose $Z \in \{-0.5,\,-0.25,\,0,\,0.25,\,0.5\}$ hence the 5 different clusters.

As is already evident, VICReg does a fantastic job of removing the angular component from within the data. We can use the MSE distance between the points in this case for our theta invariant task. The same is not true for VAE; as it is an information-preserving system, it cannot eliminate the angular component, making the concept of distance in our latent space ambiguous. One thing to exercise caution in this entire argument is that the distance is only relevant with respect to proper augmentations. With only Gaussian blurring, we won't be able to define a metric even with our VICReg setup. Removing the rotation from the augmentation space renders our metric for VICReg's latent space ill-defined for our task at hand, where rotation along the axis does not alter the system. Below, we illustrate the nature of the latent space of VICReg without rotation augmentation.

VICReg without rotation

Using different augmentation strategies, we can easily encode various types of invariances within our system, making VICReg an ideal choice to build an embedding space. The metric in this space is the Euclidean distance between two points, indicating how the two separate events differ from each other. Moreover, within each cluster of points in the VICReg space, we can expect samples that are physically invariant to one another (i.e., samples that have different representations in $R^n$ but have similar physical meanings, such as Lorentz-boosted samples or rotated samples). Building systems that can cluster our anomalous samples together is our first step in developing our unified anomaly detection framework.

Anomaly Detection, Density and Clusters

VAEs are one of the best candidate models for anomaly detection (AD). VAEs work well in AD since they fail to reconstruct points that lie in low-density regions, which the model has difficulty learning. The figure below illustrates the operation of a VAE in the context of anomaly detection, when trained on a dataset of 5 overlapping Gaussians in a 2D space.

VAE Anomaly Detection

As we can see, when using the standard reconstruction loss, we can correctly identify the 5 cluster centers along with their distribution. In contrast, the surrounding region has a higher reconstruction loss, as there are fewer pointers available there. Although the $||z||_2$ (on the right) or $\mathbf{z}^\top \mathbf{z}$ setup without the Decoder (following the v4 architecture of AXOL1TL) works well in the vicinity of the cluster and in finding the global region of high density, but fails to capture the details of the distributions.

Out-of-the-box-distribution VAEs, as demonstrated, can have good anomaly detection capacities until they encounter the Out-of-the-box-distribution problem, in which case the VAE learns too well about the dataset and ends up reconstructing everything (including the anomalies) perfectly, making it a poor choice for anomaly detection in those cases. This problem can get further aggravated when the training dataset itself contains the anomalous datapoints (true in our case).

On the other hand, although we have established the paradigm of latent space creation through VICReg and formed tight clusters of points, we have yet to address the problem of identifying clusters where our anomalies or rare samples reside.

Mahalanobis Distance, Gaussian Mixtures and Anomaly Detection

Self-supervised methods, such as VICReg, due to their inherent nature of minimizing invariance loss over the embedding space, tend to create clusters in the latent embedding space. We can broadly classify the entire dataset D to be composed of three components:

  1. Clusters of Normals (Inliers): By normalies, we understand the physics processes/signals that occur frequently and can be readily found using the existing physics-based triggers. These inliers are defined based on their statistical significance, and such clusters are considered to have a large number of elements and strong distribution peaks. These clusters are importantimportant for Standard Model studies,, but given that we are here trying to look for exotic signatures that are statistically outnumbered,, we refer to these standard physics processes as our normals. We are not particularly interested in probing these clusters in this work, given the extensive amount of research that has already been conducted on such processes. Let's denote them by the notation $N_i$, we can have many such physics processes, all of which can be categorised into their own clusters as $N_0, N_1...N_n$
  2. Cluster of Anomalies: Given the pretext that the anomalies follow a different form of distribution pattern within the dataset, given that they originate from exotic physics processes, we can consider these anomalies to be contained within their own clusters, just like the nomalies, except these cluster centers are separated from $N_i$. Given that we assume these anomalies are indicative of exotic physics signatures, they must be present in clusters. The absence of clusters for anomalies means they can never be detected with certainty using our framework, which points towards a deeper problem, such as poor detector resolution or a poor choice of augmentation/architecture. Lets call these $A_0,..A_1,...A_a$
  3. Stray points: For the sake of completion, we can consider the remaining points to be spread across the embedding space. These points are generally statistical noise and can be attributed to various sources, including hadronic noise and detector issues. We can refer to all these points $S_0, S_1... S_s$

Given that there does not exist a rigid definition distinguishing $S$ and $A$, and that in the absence of a well-defined embedding space or identifiable anomalies, $S$ may consist of shallow distributions or even isolated points—which are inherently difficult to model—we absorb all $S$ into $A$.

We can now say $D = \left( \bigcup_{i \in n} N_i \right) \times \left( \bigcup_{j \in J} A_j \right)$

Now, finding $\bigcup_{i \in I} N_i$ is straightforward, as these correspond to some of the most well-studied processes in high-energy physics (HEP). However, we deliberately refrain from invoking physical intuition, as our goal is to construct an unbiased system. Instead, we adopt the statistical definition of $N_i$ — treating them as the most well-populated distributions.

We take a hyper-parameter $k \in \mathrm{N}$ assuming that we have $K$ clusters of normals. Now we fit a Gaussian mixture model (GMM) on the latent space with $k$ gaussians, making sure that all our standard physics signatures get absorbed within these $k$ clusters. Now, for any point $p$ on the embedding space, we can find how anomalous it is by finding how far it is from these cluster centers.

Had the value of $k$ been 1. We could have used the Mahalanobis distance for finding the distance of $p$ from the cluster center. For finding the distance of a point $p$ from $k$ gaussians, one can use methods like these. Given that such methods are extremely difficult to implement on an FPGA board (where we want to deploy our model), we skip it. Instead, we try to compute $D_i$ for $p$, where we define $D_i$ using the mahalanobis distance as

$$ D_i(\mathbf{p}) = \sqrt{(\mathbf{p} - \boldsymbol{\mu_i})^{T} \, \mathbf{\Sigma_i}^{-1} \, (\mathbf{p} - \boldsymbol{\mu_i})} $$

where:

  • $\mathbf{p}$ — point or event in the embedding space whose anomaly score is being evaluated.
  • $D_i(\mathbf{p})$ — Mahalanobis distance of point $\mathbf{p}$ from the $i^{\text{th}}$ cluster center.
  • $\boldsymbol{\mu_i}$ — mean of the $i^{\text{th}}$ cluster in the embedding space.
  • $\mathbf{\Sigma_i}$ — covariance matrix of the $i^{\text{th}}$ gaussian on the embedding space.

Now, given that we are considering our embedding space to be dominated by K clusters, we can use an aggregation method to combine $D_0, D_1,..., D_K$ to get a single composite anomaly metric. One can choose different metrics, such as the arithmetic mean, min-max, and geometric mean, among others. In this work, we found that the geometric mean outperforms other aggregation metrics for the task at hand.
Hence, we can write the anomaly score as:

$$ D(p) = \left( \prod_{i=0}^{k-1} D_i \right)^{\frac{1}{k}} $$
Model Implementation

The diagram above illustrates the simple implementation of the model. When evaluated and compared with the existing baseline (v4 of AXOL1TL) over a number of Monte Carlo signals that simulate possible exotic signatures, we see an overall improvement, hinting strongly towards the success of the model.

Process Ours V4
GluGluHToBB_M-12583.2731.76
GluGluHToGG_M-1258.764.01
GluGluHToGG_M-903.892.98
GluGluHToTauTau3.231.99
GluGluHTo2B2WtoLNu2Q41.2626.64
HHToBB80.0158.04
HHHTo4B2Tau74.9755.97
VBFHToTauTau12.2711.84
VBFHTo2B20.9613.81
WToTauTo3Mu0.120.11
ttHto2B81.4765.25
ttHto2C83.7067.83

The efficiency table above shows the efficiency of detecting the Monte Carlo signal when the trigger is operated at a working frequency of 1kHz.

VAEs with VICReg

We again go back to our discussion on VAEs, as the above method, though interpretable and better performing than the baseline (AXOL1TL v4), has a few shortcomings:

  1. Mahalanobis Distance for Gaussian Mixtures: The use of GMMs along with the Mahalanobis distance is not well defined when it comes to defining a single composite score/distance/metric. The said approach of using an aggregation function cannot be analytically justified and may lead to unexpected results.
  2. Computational Bottlenecks: The best-performing aggregation method for this use case is the geometric mean. We arrived at this result after extensive experimentation with various aggregation methods, including arithmetic mean, thresholding, and taking the minimum or maximum, among othersc. Implementing the geometric mean is extremely expensive on FPGA boards, making this model effective in theory but challenging to deploy in production.

The question now is whether one can design an estimator (of the MLP type) that can perform GMM clustering and subsequent distance estimation. Given that we are estimating the distance from the highly populated cluster, we can say that a VAE, when trained on these embeddings, can learn to reconstruct these strong cluster centres with more ease compared to the weaker anomalous ones. This is very different from using a VAE directly on top of the data, which can be better understood by revisiting the toy cylinder problem. The initial distribution of the data on the cylinder, though taken uniformly (for our ease), can be thought of as a combination of clusters. When density estimation using VAEs is performed on the input space alone, there is a strong likelihood that the model learns the entire distribution, as anomalies are not well separated in the input space. Moreover, due to the high dimensionality, the input space has a wider and more uniform distribution compared to the latent space, where the distribution collapses into clusters since the invariance conditions are enforced using the VICReg loss. The uniform/well-spread distribution on the input space makes it easier for the VAE to learn the entire input space, defeating our intention for it to only learn the statistically significant regions. When trained on the embedding space,, the VAE has a hard time learning to reconstructreconstruct small anomalous regions that are separated from the area of normals and are significantly outnumbered. The plot on the left shows our original input dataset, and the one on the right shows the nature of the embedding space.

Input Space Embedding Space

The distribution on the left (input space) is therefore easier for the VAE to generalize than the one on the right (embedding space). We can enforce this idea further by training a shallow VAE for the density estimation since embeddings derived from VICReg are disentangled, simplified and smaller in dimensionality making a shallow VAE sufficient to learn the normals Moreover using this setup we also successfully managed to incorporate some degree of physics information (that we can control through the augmentation) and detector information into the system by forcing the invariance. This lets us take the next step, where we add a VAE on top of the VICReg embedding. We follow the given recipe:

  1. Train VICReg with a suitable encoder on the raw data using physics-inspired and context-motivated augmentations.
  2. Strip away the encoder of the VICReg and freeze the weights.
  3. Extract the embeddings of the VICReg space.
  4. Train a simple VAE on the embedding space.
  5. Remove the variational part of the VAE along with the resampling step and convert it to an AE.
  6. Use the reconstruction loss between the input to the VAE (now AE) and output as the anomaly score.
VAE on VICReg

The above setup makes it very easy to deploy the above model onto the FPGA board, given that it is a stack of MLPs. Moreover, it outperforms the GMM+VICReg setup since the VAE backbox now manages everything. The comparison of efficiency in detecting exotic Monte-Carlo signals with respect to the last deployment (vanilla v4) is shown below.

The efficiency numbers for the V5 (ours) and V4 (old baseline) are as follows

Signal Name VICReg + GMM (ours) V4 (Baseline) VICReg + VAE(ours, deployed)
GluGluHToBB_M-12583.2731.7695.93771
GluGluHToGG_M-1258.764.0137.77188
GluGluHToGG_M-903.892.9815.854626
GluGluHToTauTau3.231.994.831854
GluGluHTo2B2WtoLNu2Q41.2626.6453.692781
HHToBB80.0158.0484.302641
HHHTo4B2Tau74.9755.9781.811932
VBFHToTauTau12.2711.8426.036048
VBFHTo2B20.9613.8127.53418
WToTauTo3Mu0.120.110.270025
ttHto2B81.4765.2586.183296
ttHto2C83.767.8388.221684

The efficiency vs Rate plot is given below. For the same budget (trigger rate independent of other triggers), both the v4 and v5 can be compared using the efficiency at the AXOL1TL medium rate.

Efficiency vs Rate Plot

For more details on the Model Architecture and performance, see

  1. https://cds.cern.ch/record/2904695/files/DP2024_059.pdf
  2. Model Report

MLoPs for ML Trigger at 40 MHz

Introduction

The CMS experiment at CERN records data at a rate of approximately 40 Terabytes per second. This level of information cannot be stored in storage systems. The amount of data is on such a scale that even online analysis of it on CPU and GPU farms is not possible. CMS records data at a frequency of 40 MHz (40 million events every second). To solve this problem, an elaborate trigger system exists that filters the data from the staggering rate of 40 MHz down to as small as 1kHz. This is mainly done in two stages:

  1. Level 1 Trigger (L1T): This is the first trigger that operates mainly on Field Programmable Gate Array (FPGAs) and is responsible for reducing the rate from 40 MHz to 100 kHz.
  2. High Level Trigger (HLT): This is the second trigger that operates on CPU and GPU farms and uses a complete physics analysis (reconstruction) to make the trigger decision. HLT reduces the data rate from 100 kHz to 1 kHz.
CMS Trigger System

The L1T comprises a combination of various trigger subsystems. One such system is the AXOL1TL, which stands for Anomaly eXtraction Online L1 Trigger Lightweight in the L1 Global Trigger. AXOL1TL is a completely ML-based trigger that is designed to trigger events that are unbiased from the physical intuitions used to build the other trigger systems, such that it can detect new exotic signatures. For more details regarding the ML side of the trigger and its associated development, please refer to this link.

The CMS detector goes through recalibration every few months. This necessitates the deployment and development of models each time the detector is recalibrated. Moreover, machine learning models can often start to diverge when the incoming input statistics change due to progressive changes in detector condition. All these factors reduce the lifespan of any ML-based trigger, and hence, such triggers require frequent redeployment. In this report, we will cover some MLoPs and infrastructures that we have developed over the last year to ease the task of model redeployment and eventually pave the way towards continual learning and fully autonomous trigger systems.

Recipe for an ML Trigger at 40 MHz

Before we delve deep into the infrastructure needed to develop such systems, we need to understand the overall high-level process of building and deploying such a trigger.

High Level Process
  1. Dataset Curation: For new training run curation, new data is necessary, given that the data distribution of the detector changes after every new calibration. The data curation consists of two steps:
    • Zerobias: The zerobias (ZB) refers to the dataset that is directly collected from the detector. This dataset undergoes no trigger or physics selection, making it completely independent of the trigger components and unbiased.
    • Monte Carlo: To test the performance of the trigger, various exotic physics processes are simulated using the Monte Carlo (MC) simulation. This set is used only during validation and not for training. MC is difficult to acquire due to the associated computational cost. Hence, we treat MC slightly differently when it comes to MLOps.
  2. Model Training: Model Training step consists of treating the acquired ZB and MC to make it ML-friendly. Following this, the model is trained following a fixed recipe that we developed through separate testing. The models are trained in PyTorch with quantisation-aware training. The hyperparameters are optimized using Optuna and ASHA, and the overall distributed setup is managed by SLURM and RAY.
  3. Deployment: The model deployment stage consists of taking the trained weights from PyTorch and building the model using HLS4ML, which converts into Verilog. This is done on a hardware machine. After this, the efficiencies are matched for the hardware model and the original one. Given a green signal from the team and the trigger committee, we deploy it. This step has been intentionally kept manual to maintain a degree of freedom in triggering.
  4. Divergence Tracker: The final step can be considered as an entirely separate system, too. The task of the divergence tracker, also known as the Scouting Analyser, is to continually stream data from the detector and compare the model output statistics over time. This system helps in knowing when the model starts to perform poorly and a new deployment is needed.

During my Technical Studentship, I primarily worked on developing steps 1 (refactoring), 2 (complete refactoring and redevelopment), and 4 (built from scratch). The deployment part is still mostly manual, but given that the most time-consuming steps, being steps 1 and 2, are now automated, the overall time to build a new trigger and have it deployed has reduced from a few weeks to a couple of days. This report will skip steps 1 and 3 of the pipeline, as it contains numerous internal components and code related to the experiment that have not yet been approved for public release.

Model Training

Model Training Schematic

The code associated with the above setup can be found here.

The model training schematic is shown above. The overall training process, along with hyperparameter tuning, can be broken down into a few successive steps.

  1. For all sorts of model training, we have a configuration.yml that we use to set up everything.
  2. The data_util subrepository handles the data processing and prepares it for training. The raw data is read from the CERN storage, then processed and stored as a temporary H5 file that will be used for training.
  3. Once the data preparation is complete and a suitable training method is found from the config, the associated recipe is launched. This recipe calls several RAY servers that start the training process.
  4. The RAY cluster begins the hyper-parameter optimisation sampling, which Optuna controls, and the early stopping criteria are managed by ASHA. This RAY cluster can be set up in two ways.
    • Using SLURM to schedule large-scale search and training runs. The entry point to these codes are through 1 and 2
    • Using standalone compute nodes. The entry point in this case is simply the recipe file itself.
  5. After completion of the training run and selection of the best model, the pipeline moves toward creating various plots and physics analysis. Once all the analysis is complete, the pipeline begins creating an HTML report that provides details of the entire run. This report is handy and helps with rapidly assessing the model's performance. The entire code for this step is here.
  6. Finally, the pipeline starts to put all these together into an H5 file. This includes all configurations, training data, Monte Carlo results, model history, and plots. The H5 is stored for future use and is also picked up by the deployment stage.

Divergence Tracker

Scouting Analyser System

The diagram shows the schematic of the Scouting Analyser System. This is the same system used to track divergences in online anomaly detection performance.

Processing Pipeline

Input FIFO: The data for our pipeline comes from the L1 Scouting System. The L1 Scouting system streams a subset of the complete data that passes through the L1 Trigger and stores it on tape for future analysis. Before passing it to tape, the system temporarily stores it on the CERN storage disk, which can be accessed readily. Given the massive rate of this data, ~1 TB every hour when the system is running. The system stores data for only 24 to 48 hours on the disk. Hence, this system acts like a FIFO channel.

Stage 0 (Data Ingestion)

The purpose of Stage 0 is to Ntuple the raw root files into a suitable format after reading them from the temporary storage disk. Moreover the first stage directly writes the Ntulpes to /dev/shm/StoreStage0/ to prevent IO bottlenecks down the line. This stage operates with 64 threads, all of which access a unique root file according to the FIFO schema. The benefit of using 64 threads in parallel for the ntuplising part is two-fold:

  1. The read write speed of a single thread on the CERN files system remains capped making it necessary to use multiple threads to prevent information bottleneck and make it possible to transfer ~1Tb data per hour.
  2. The Ntuplising step is extremely slow, although written in C++, the step handles a major chunk of data pre-processing. Moreover, this step filters much redundant event-level information, making it the slowest and the heaviest step in the entire pipeline.

Each '.root' file produced in this step contains ~70 million events and 23.31 seconds of data taking, or 1 Lumisection of data.

Stage 1 Data Pre-processing (64 Threads)

  • Input: ROOT files from /dev/shm/StoreStage0/
  • Output: Zarr format files in /dev/shm/StoreStage1/

This step is executed by 32 parallel threads, all of which access the same root file located in /dev/shm/StoreStage0/. This stage reads and extracts the hardware values of the L1 trigger objects (Muons, E/Gamma, Jets, Energy Sums) and pre-processes them. Additionally, it stores the associated metadata for every event it processes for future use. All 32 threads write in parallel to 32 different zarr files into /dev/shm/StoreStage1/.

Stage 2: Model Inference (16 Threads + 1 GPU)

  • Input: Zarr files from Stage 1
  • Output: Zarr files with AXOL1TL scores in /dev/shm/StoreStage2/

This step runs on 16 CPU threads, all of which share a single H100 GPU. Each of these threads loads the quantized Keras model. Following this, it reads from the zarr files created in stage 1 and runs inference on them. The anomaly scores calculated in this step are stored in Zarr format again in /dev/shm/StoreStage2/.

Stage 3: Histogram Generation (8 Threads)

  • Input: Score files from Stage 2
  • Output: Histogram shards in /dev/shm/StoreStage3/

This stage operates on a single thread, where it reads the Zarr files from the previous step that contain the anomaly scores and events, and creates bins with a range of [0, 20000] and a width of 1. Following this, it converts the histogram to sparse format (only non-zero bins are stored) and stores it in a text-encoded file with the suffix .axo_shards. The metadata for the range of events associated with the histogram is also stored in this shard.

Stage 4: Aggregation & Storage

  • Input: Histogram shards from Stage 3
  • Output: Final histograms in /eos/project/c/cms-axol1tl/Pipeline/Scouting/ScoreStream

This is again a process running on a single thread. This process scans the memory to find all the available shards and merges the 32 shards that belonged to the same lumisection. This also detects and removes duplicate shards that may have occurred due to racing conditions or failures in the inference step. The 32 shards of a single lumisection are merged into a histogram, which is stored in a sparse format with text encoding, using a '.axo' suffix, along with associated UTC timestamps.

Tracker: Resource Monitoring

The tracker monitors the memory and disk usage across the different stages

50 Hz Rate Monitoring

One important purpose of the above system is to monitor the raw rate at 50 Hz, which is the pure rate. The raw rate is the number of events that the trigger is allowed to pass through. The Pure rate can be though of the number of events that AXOL1TL uniquely pass and these events do not belong to any other trigger in the L1 Trigger System. Given that there is a lot of overlap between different triggers, it is not uncommon to find a very high raw rate (number of events passing through the trigger) for a nominal pure rate (number of unique events passing through the trigger). Even though the pure rate of 50 Hz has a sufficiently large associated raw rate, the thresholds to change significantly with undetected conditions, given that the L1 rate is close to kHz, which is orders of magnitude larger than this rate. The 50 Hz Rate and its thresholds are regularly monitored, and having these numbers as close as possible to real trigger conditions is important for the proper functioning of the trigger. This system gives a window where the 50 Hz rate numbers and their associated thresholds can be easily calculated by tapping the output of StoreStage2.

Retrain Trigger

Continual learning is something that we started our discussion with. For continual learning to be effective, it requires a trigger that activates the model retraining and redeployment procedures. This can be easily done within this system by monitoring the statistics it produces. Moreover, the retrain trigger has to curate the zero-bias for retraining the trigger. This is something that StoreStage1 handles, where the processed files are stored.

Large Scale Statistics

When working with anomaly detection, look into the tails of the anomaly distribution. To investigate this statistically rare region, one often needs many statistics, which can take a considerable amount of time to gather. Using the outputs of this system, statistics of ~1 to ~10 billion points are easily achievable, making studies on the tails of the anomaly distribution more effective.

Heatmap of Anomaly Score
The above heatmap illustrates the distribution of the anomaly score on the x-axis and its variation across different lumisections or time steps.
Raw Rate Variation
The above plot shows the variation of the raw rate with different lumisections.