Arcadia Collateral Risk Simulation: Progress Report 3
The third in a series of updates on the joint project by Llama Risk and Pragma Labs for the Arcadia v2 protocol.
Previous Reports
Report 1 - February 9
Report 2 - February 14
In our previous progress report, we detailed the technical framework to operationalize the methodology for the Arcadia simulation. This week, we will go over the simulation and validation processes to showcase how we ensure that the simulation functions as intended.
This progress report is structured into four parts:
Simulation Verification & Validation: An in-depth examination of how we ensure the simulation operates as designed and accurately reflects the real-world system it models.
Progress: A concise overview of the milestones reached and the enhancements made to the simulation during the current week.
Next Week's Goals: A forward-looking snapshot of our objectives aimed at further refining the simulation's accuracy and scope.
Next Week’s Post: A preview of the forthcoming update, focusing on a detailed exploration of the borrower data model and its implications for our simulation's realism and reliability.
Simulation Verification & Validation
Law and Kelton (1991) describe Simulation Verification as the process of ensuring that a simulation computer program operates according to its design, essentially focusing on debugging the program. Conversely, Simulation Validation is characterized as the evaluation of whether the conceptual model used in the simulation (rather than the computer program itself) accurately reflects the real-world system being studie
Synthetic Cases with Extreme Values
This approach involved creating artificial or synthetic test cases that incorporate extreme values or conditions. For the Arcadia Simulation we created multiple scenarios that simulate different price movements and created an entry in our testing file and an accompanying example.
These cases are not necessarily based on real-world scenarios but are designed to push the system to its limits. The purpose is to identify potential weaknesses or points of failure within the system that may not be evident under normal operating conditions. Below we included the code of a multi-asset scenario with a single rug testing the pipeline (e.g. Simulation Scenario M4):
# ------------------------
# User Account Setup
# ------------------------
# Example user account with 1 cbETH as collateral and a debt
account = MarginAccount(
address="0xTest",
debt=(4000 * 1e6), # in USDT terms
numeraire=usdt,
assets=[
AssetsInMarginAccount(
asset=cbETH,
metadata=AssetMetadata(
share=0,
amount=1e18, # holds 1 cbETH
current_amount=1e18,
risk_metadata=AssetValueAndRiskFactors(
collateral_factor=0.9, liquidation_factor=0.5, exposure=2 * 1e18
),
),
),
AssetsInMarginAccount(
asset=weth,
metadata=AssetMetadata(
share=0,
amount=1e18, # holds 1 cbETH
current_amount=1e18,
risk_metadata=AssetValueAndRiskFactors(
collateral_factor=0.9, liquidation_factor=0.5, exposure=2 * 1e18
),
),
),
],
)
....
The values shown in Price Feed Creation are extreme values unlikely to be a scenario that Arcadia would encounter in reality, but this method helps ensure that the simulation system is robust enough to handle outliers and extreme cases without failing or producing incorrect outputs.
Taking a Look at Logs
We can observe the result log of example M4.py and see whether the simulation behaves as anticipated:
[INFO][15:21:40][arcadiasim.pipeline.pipeline]-14124: Iteration :: 0 @ 1705473000
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: Account 0xTest is liquidatable
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: Account has been put for auction
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: Scan started
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: Active auctions 1
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: asset=<Asset: Coinbase Wrapped Staked ETH> price=2805.94266 bid_price=5999.994
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: asset=<Asset: Wrapped ETH> price=0.0 bid_price=0.0
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: No bids made by the liquidator
[INFO][15:21:40][arcadiasim.pipeline.pipeline]-14124: [STATE] {"simulation_time":1705473000,"all_liquidated_accounts":[],"auctions":{"0xTest":{"debt":4000000000,"assets":{"cbETH":1000000000000000000,"WETH":1000000000000000000}}}}
During the first iteration, account 0xTest was identified as liquidatable and consequently put up for auction. The liquidator initiated a scan, revealing one active auction and evaluating two assets: Coinbase Wrapped Staked ETH (cbETH) with a market price of $2805.94 and a bid price of $5999.99, and Wrapped ETH (WETH) with no price or bid price listed. In line with expectations, the liquidator decided not to place any bids. 0xTest is not liquidated.
[INFO][15:21:40][arcadiasim.pipeline.pipeline]-14124: Iteration :: 1 @ 1705476600
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: Account 0xTest is liquidatable
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: Account is already on auction
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: Scan started
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: Active auctions 1
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: asset=<Asset: Coinbase Wrapped Staked ETH> price=2834.277836 bid_price=4199.995800000354
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: asset=<Asset: Wrapped ETH> price=0.0 bid_price=0.0
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: No bids made by the liquidator
[INFO][15:21:40][arcadiasim.pipeline.pipeline]-14124: [STATE] {"simulation_time":1705476600,"all_liquidated_accounts":[],"auctions":{"0xTest":{"debt":4000000000,"assets":{"cbETH":1000000000000000000,"WETH":1000000000000000000}}}}
During the subsequent iterations, account 0xTest continued to be evaluated for liquidation with its assets remaining on auction. In iteration 1, the liquidator's scan identified the account as still liquidatable, noting one active auction but ultimately deciding against placing any bids. The price of Coinbase Wrapped Staked ETH (cbETH) had increased slightly to $2834.28, with its bid price adjusted to $4199.99, while Wrapped ETH (WETH) maintained its status with no listed price or bid.
[INFO][15:21:40][arcadiasim.pipeline.pipeline]-14124: Iteration :: 2 @ 1705480200
...
[INFO][15:21:40][arcadiasim.pipeline.pipeline]-14124: [STATE] {"simulation_time":1705480200,"all_liquidated_accounts":[],"auctions":{"0xTest":{"debt":4000000000,"assets":{"cbETH":1000000000000000000,"WETH":1000000000000000000}}}}
Moving into iteration 2, the situation appeared largely unchanged with account 0xTest still marked for liquidation and the auction ongoing. The cbETH saw a further price increase to $2850.13, yet its bid price decreased to $3299.99. Despite these fluctuations and continued evaluation, the liquidator opted not to place any bids in either iteration, leaving the state of the auction and the assets of account 0xTest unchanged.
[INFO][15:21:40][arcadiasim.pipeline.pipeline]-14124: Iteration :: 3 @ 1705483800
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: Account 0xTest is liquidatable
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: Account is already on auction
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: Scan started
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: Active auctions 1
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: asset=<Asset: Coinbase Wrapped Staked ETH> price=2901.136743 bid_price=2849.997150000266
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: asset=<Asset: Wrapped ETH> price=0.0 bid_price=0.0
[INFO][15:21:40][arcadiasim.arcadia.liquidator]-14124: [BID] {<Asset: Coinbase Wrapped Staked ETH>: 1000000000000000000, <Asset: Wrapped ETH>: 0} 51139592.999733925
[INFO][15:21:40][arcadiasim.pipeline.pipeline]-14124: [STATE] {"simulation_time":1705483800,"all_liquidated_accounts":[],"auctions":{"0xTest":{"debt":4000000000,"assets":{"cbETH":0,"WETH":1000000000000000000}}}}
In the third iteration, account 0xTest was once more identified as liquidatable and confirmed to be still listed in the ongoing auction. The liquidator's scan indicated one active auction. This time, the price of cbETH rose to $2901.14, with its bid price also increasing to $2849.99. Despite WETH still showing no value, the liquidator made a bid, presumably on the cbETH asset, as its entire amount was liquidated from account 0xTest, leaving only WETH in the auction. This iteration marked the first active bid by the liquidator, leading to the partial liquidation of account 0xTest's assets.
Conclusion
The verification test has passed, the simulation behaved as we expected, and the liquidator placed the bid at the expected iteration.
In addition to testing with synthetic extreme cases, we also conduct manual inspections of the outputs produced by the system at the intermediate stages of the simulation. For instance, we manually would calculate the bid price or start debt. This step is crucial for understanding how the system progresses through its computations or simulations and where it might be introducing errors or exhibiting unexpected behavior.
Validation
For the validation procedure, we employ straightforward observation techniques and basic statistical tests to ensure our system mirrors real-world operations. This process involves generating and comparing key metrics such as the number of liquidations and the volume of liquidations against historical data from established platforms.
The challenge in this procedure is amplified as Arcadia currently does not operate a live counterpart for direct comparison. Consequently, our validation efforts rely on drawing analogies with existing systems. This approach necessitates careful consideration and adaptation of validation criteria to suitably reflect the complexities and dynamics of Arcadia's model, ensuring a robust and accurate assessment despite the lack of direct real-world parallels.
More Validation will be implemented as we run more real-world scenarios during the simulation.
Goals for Week 7
For week 7, the team has set the following goals to advance our simulation model's development further:
Comprehensive Simulation Coverage: Expand simulations to encompass all single assets and LP tokens, broadening the model's applicability and insight range.
Enhanced Log Visualization: Implement log visualization tools for better data interpretation and performance tracking.
Refined Borrower Models: Develop detailed borrower models based on retrieved borrowing data to simulate diverse risk profiles.
Expanded Testing: Conduct thorough validation and verification testing against more real-world scenarios to ensure model accuracy and reliability.
Targeted Improvements: Execute smaller, targeted fixes to enhance model performance and address any identified issues.
These goals aim to refine the simulation's realism, usability, and analytical depth, reinforcing its utility as a decision-making tool.
Progress Report
The team has successfully achieved the objectives set for week 6. The progress can be summarized as follows:
Simulation Refinement: Integration of slippage and gas costs into the profit calculations and the implementation of new optimization algorithms have enhanced the simulation's accuracy and efficiency.
Data Enrichment: A substantial increase in borrower data has been retrieved including borrower data on major lending platforms on Base and Ethereum.
Testing Enhancements: Expanded verification and validation testing against real-world cases has solidified the model's robustness and real-world applicability.
Scenario Realism: The introduction of more realistic scenarios has improved our model's stress-testing capabilities, offering valuable insights under various conditions.
Logging and Monitoring: Improved logging has enhanced real-time performance monitoring, facilitating quicker issue identification and resolution.
Optimization and Fixes: Continuous optimization and debugging efforts have led to a streamlined, more accurate simulation environment.
Next Week’s Post:
In our upcoming post, we will take a closer look at the borrower data model, exploring its intricacies and the enhancements we're planning. Stay tuned for an update on the progress made over the week, where we'll highlight key achievements and outline our next steps in getting the simulation production ready.
Law, A.M., & Kelton, W.D. (1991). Simulation Modeling and Analysis (2nd ed., p. 299). McGraw-Hill, Inc.