WebSafe 3.7github.com
|
|
🏠
Skip to content

Moving from circleci to gha WRT benchmark flows#3209

Open
filipecosta90 wants to merge 4 commits intomasterfrom
benchmarks.gha
Open

Moving from circleci to gha WRT benchmark flows#3209
filipecosta90 wants to merge 4 commits intomasterfrom
benchmarks.gha

Conversation

@filipecosta90
Copy link
Collaborator

@filipecosta90 filipecosta90 commented Oct 11, 2023 •
edited
Loading

This PR removes the usage of circleci for benchmark workflows. It introduces 3 github actions:

  • One for merges to version and master/main branches
  • One for triggering benchmarks during PR work
  • One for triggering nightly benchmarks once a week

It requires the following secrets to be present among the repo action secrets:

  • PERFORMANCE_RTS_AUTH
  • PERFORMANCE_RTS_HOST
  • PERFORMANCE_RTS_PORT
  • PERFORMANCE_WH_TOKEN
  • PERFORMANCE_GH_TOKEN
  • PERFORMANCE_EC2_ACCESS_KEY
  • PERFORMANCE_EC2_SECRET_KEY
  • PERFORMANCE_EC2_REGION
  • PERFORMANCE_EC2_PRIVATE_PEM

Checks:
@filipecosta90 filipecosta90 added the action:run-benchmark Trigger CI benchmarks label Oct 11, 2023
@filipecosta90 filipecosta90 requested a review from chayim October 11, 2023 05:43
@filipecosta90
Copy link
Collaborator Author

filipecosta90 commented Oct 12, 2023 •
edited
Loading

Automated performance analysis summary

This comment was automatically generated given there is performance data available.

In summary:

  • Detected a total of 16 stable tests between versions.
  • Detected a total of 3 highly unstable benchmarks.
  • Detected a total of 1 regressions bellow the regression water line 5.0.

You can check a comparison in detail via the grafana link

Comparison between master and benchmarks.gha.

Time Period from 30 days ago. (environment used: oss-standalone)

Test Case Baseline master (median obs. +- std.dev) Comparison benchmarks.gha (median obs. +- std.dev) % change (higher-better) Note
ENTITY_COUNT 49998 +- 0.0% (7 datapoints) 49998 +- nan% (1 datapoints) -0.0% -- no change --
EXPLICIT-EDGE-DELETION 50 +- 37.8% UNSTABLE (7 datapoints) 50 +- nan% (1 datapoints) -0.0% UNSTABLE (very high variance)
GRAPH500-SCALE_18-EF_16-1_HOP 33323 +- 18.9% UNSTABLE (7 datapoints) 33323 +- nan% (1 datapoints) -0.0% UNSTABLE (very high variance)
GRAPH500-SCALE_18-EF_16-1_HOP_MIXED_READ_65perc_WRITE_25perc_DEL_10perc 50 +- 2.6% (7 datapoints) 50 +- nan% (1 datapoints) -0.0% -- no change --
GRAPH500-SCALE_18-EF_16-1_HOP_MIXED_READ_75perc_WRITE_25perc 59 +- 2.4% (7 datapoints) 59 +- nan% (1 datapoints) 0.0% -- no change --
GRAPH500-SCALE_18-EF_16-2_HOP 11111 +- 0.0% (7 datapoints) 11111 +- nan% (1 datapoints) -0.0% -- no change --
GRAPH500-SCALE_18-EF_16-2_HOP_MIXED_READ_75perc_WRITE_25perc 59 +- 0.0% (7 datapoints) 59 +- nan% (1 datapoints) 0.0% -- no change --
GRAPH500-SCALE_18-EF_16-3_HOP 893 +- 0.5% (7 datapoints) 893 +- nan% (1 datapoints) -0.0% -- no change --
GRAPH500-SCALE_18-EF_16-3_HOP_MIXED_READ_75perc_WRITE_25perc 56 +- 0.0% (7 datapoints) 56 +- nan% (1 datapoints) 0.0% -- no change --
IMPLICIT-EDGE-DELETION 50 +- 0.0% (7 datapoints) 50 +- nan% (1 datapoints) -0.0% -- no change --
INSPECT-EDGES 2500 +- 0.0% (7 datapoints) 2500 +- nan% (1 datapoints) -0.0% -- no change --
MIX_READ_WRITE 10000 +- 0.0% (7 datapoints) 10000 +- nan% (1 datapoints) -0.0% -- no change --
NODE-BATCH-DELETE 50 +- 0.0% (7 datapoints) 50 +- nan% (1 datapoints) 0.0% -- no change --
NODE-INDEX-LOOKUP 33 +- 0.0% (7 datapoints) 33 +- nan% (1 datapoints) -0.0% -- no change --
SORT_ENTITIES 100 +- 0.0% (7 datapoints) 100 +- nan% (1 datapoints) 0.0% -- no change --
UPDATE-BASELINE 24999 +- 0.0% (7 datapoints) 24998 +- nan% (1 datapoints) -0.0% -- no change --
VARIABLE-LENGTH-FILTER 33323 +- 18.9% UNSTABLE (7 datapoints) 33324 +- nan% (1 datapoints) 0.0% UNSTABLE (very high variance)
VARIABLE_LENGTH_EXPAND_INTO 100 +- 0.0% (7 datapoints) 100 +- nan% (1 datapoints) 0.0% -- no change --
allShortestPaths-10hop-100Kpaths 30 +- 5.4% (7 datapoints) 30 +- nan% (1 datapoints) 0.0% waterline=5.4%. -- no change --
allShortestPaths-4hop-10Kpaths 357 +- 3.3% (7 datapoints) 333 +- nan% (1 datapoints) -6.7% REGRESSION
swilly22
swilly22 suggested changes Oct 16, 2023
View reviewed changes
@filipecosta90 filipecosta90 requested review from swilly22 and removed request for chayim October 20, 2023 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

action:run-benchmark Trigger CI benchmarks

2 participants

@filipecosta90 @swilly22

Comments