Benchmarking

Purpose

Project members are generally scattered around a country, city, planet in todays connected world. The iModel benchmark lets you see underlying performance of the iModel you are accessing. Note there are many areas of performance enhancements, depending on how cold the iModel is

Sections

• Indicators - Show the relative performance to retrieve the various sections of an iModle
• Save - If you have Author entitlement, you can save the indicators to track performance trends.
• Export - You can export the indicators, to track performance locally.
• iModel Console - Each indicator has the ability to open the underlying query in imodelconsole.

Performance States

• Bad - The time to return the various query results are hashed and calculated based upon the iModel / User location and network performance.
• Medium - Generally the performance can be improved upon, consider the location of the iModel and your network performance. If you are in an office environment or shared network connection, consider how many users are utilising the same connection.
• Good - While the performance may still be slow, based upon the iModel lcoation and your location, the performance may still be good for the environment.

Tiles

• Elements - All records from engineering models in an iModel are elements. Even categories/levels/layers etc, so this represents the total in the iModel. Generally in the 100's of thousands, but will be millions in most iModels.
• Models - THe number of engineering models in the iModel. Depending upon the content, this will be 10's or 100's. While it can be 1000's that would generally be a bad indicator.
• Classes - The number of classes in the iModel. Should be in the 100's. 10's then the engineering content, may not have come from a structured source. 1000's then the engineering content has come from multiple disparate sources. Generally not a good indicator as it means a disparate project team.
• Aspects - Indicates the breakdown of the information. This is number of actual records rather than the actual Aspect classes. Generally should be a common multiple of the number of geometry elements. If the multiplier, alters that generally indicates either a new data source or missing data from a new data provider.
• Relationships - Similar to aspects, provides an insight into the connectedness of the data. This should be a regular multiple of the geometry elements.
• Geometry - The number of graphical elements in the iModel. Used as a trned to indicate if there is a lot of work ongoing in the project. A large spike or dips and troughs generally indicates that there is ongoing work. Depending upon the state of the models, this shoudl be expected, but large geometry changes during bud / construction phases should be a cause for concern.

iModel States

Cold iModel State

A Cold iModel refers to a state where the digital twin data resides strictly in cloud storage (like Azure Blob or AWS S3) and has not yet been opened, provisioned, or copy into active memory/local SSD storage by the backend execution nodes.

• ECSQL Performance: Very Slow (High Latency).
• What happens under the hood: When an ECSQL query is fired against a cold iModel, the infrastructure must first provision a container/backend worker, copy the required briefcase (.bim file) or pull the required change-sets, and mount the database.
• Impact: The initial query experiences a significant "cold start" delay, sometimes taking seconds or even minutes depending on the size of the infrastructure asset.

Cached (Warm) iModel State

A Cached (or Warm) iModel means that the physical database file has already been copied to the backend worker's local SSD storage, or the schema has been parsed, but it isn’t actively kept alive in the system's ultra-fast RAM.

• ECSQL Performance: Moderate to Fast.
• What happens under the hood: The database connection is closed or idling, but because the files are already locally cached on the host machine, it avoids the heavy cloud provisioning phase. When the ECSQL runs, SQLite opens the local to the worker file quickly.
• Impact: Performance is dramatically better than a cold state. However, the very first ECSQL query on a cached iModel will still experience minor disk I/O latency as SQLite populates its internal page cache from the local SSD into memory.

Hot iModel State

A Hot iModel represents the peak performance state. In this phase, the iModel is fully opened, active backend workers are actively running, and the necessary database pages, schemas, and frequently hit indices are fully loaded into the RAM of the execution node.

• ECSQL Performance: Optimal / Near Real-Time.
• What happens under the hood: The infrastructure keeps the iModel "warm and ready" in memory. Furthermore, if you are reusing Preprepared ECSQL Statements (ECSqlStatement), the costly process of parsing the ECSQL, mapping BIS classes to raw SQLite tables, and generating query execution plans is already finished.
• Impact: ECSQL queries return results in milliseconds. This state is critical for user-facing interactive dashboards, PowerBI integrations, and real-time model analysis.

iModel Cached Query

When a query for an iModel and Change state is initially ran then the results are stored in a performance cache. The second time the query is ran with the same finderprint, the results are returned from the cache rather than running the query against the hot iModel.

• ECSQL Performance: Not applicable, ecSQL is not executed.
• What happens under the hood: The query is hashed and the results of the hash are returned from an optimal cache. Large results are returned almost instantly and are accessible instantly.
• Impact: Large datasets are returned almost instantly, when the a query is re-ran. For dhasboards accessed by many users when an iModel is only being updated daily or weekly the performance hit is only taken by the first user.

Toolbox Cached Query

When a query is ran against an iModel and change set, the results still have to be downloaded to the users device. Based upon network speed and latency, this can still affect remote users on large datasets. Toolbox hashed and encrypts results into a local cache for user performance.

• ECSQL Performance : Not applicable, ecSQL is not executed.
• What happens under the hood: The query is hashed against the iModel and changeset and the results looked up in a local cache. All results are returned instantly. Only the first query on an iModel and changeset are actually ran against the iModel.
• Impact : For large datasets, the user requires storage on their local device, for multiple complex dashboards, this may have an impact on local storage. Users may have access to another users cache on insecure devices and run decryption techniques to access underlying data. All caches are secured against a specific user rather than in a common store, on secure devices this risk is mitigated.

Notes

• On the iModel performance dashboard, the results will show live information as the queries are not cached for this dashboard.