lupyuen/pr-labeler-test-cases.md

## pr-labeler-test-cases.md

      
    Raw
  

              pr-labeler-test-cases.md
            
          
    NuttX Apps: Test Cases for new implementation of NuttX PR Labeling

See the PR: apache/nuttx-apps#3408
Read the Article: https://lupyuen.org/articles/prtarget
Discussion: apache/nuttx#18359
Arch Labeling


Root CMakeLists.txt / Makefile: Should be labeled correctly as Area: Build system and trigger a Complete Build


Example CMakeLists.txt / Makefile: Should be labeled correctly as Area: Examples and trigger a Complete Build


CI Files: Should be labeled correctly as Area: CI and trigger a Complete Build


System Include include/system: Should be labeled correctly as Area: System and trigger a Complete Build


Size Labeling


Size XS, S, M, L, XL should be labeled correctly

Add 10 lines in 1 file: Correctly labeled as Size XS
Add 11 lines in 1 file: Correctly labeled as Size S
Add 100 lines in 1 file: Correctly labeled as Size S
Add 101 lines in 1 file: Correctly labeled as Size M
Add 500 lines in 1 file: Correctly labeled as Size M
Add 501 lines in 1 file: Correctly labeled as Size L
Add 1000 lines in 1 file: Correctly labeled as Size: L
Add 1001 lines in 1 file: Correctly labeled as Size: XL


Added / modified / removed lines in a Single File: Should be labeled correctly

Add 11 lines in 1 file: Correctly labeled as Size S
Remove 11 lines in 1 file: Correctly labeled as Size S
Remove 5 lines and add 6 lines in 1 file: Correctly labeled as Size: S


Added / modified / removed lines in Multiple Files: Should be labeled correctly

Add 10 lines across 2 files: Correctly labeled as Size: XS
Add 11 lines across 2 files: Correctly labeled as Size: S
Remove 11 lines across 2 files: Correctly labeled as Size: S
Remove 11 lines and add 11 lines across 2 files: Correctly labeled as Size: S


Deleted files should be ignored

Add file with 11 lines, delete file with 258 lines: Correctly labeled as Size: S


Response Time


PR Labeling must complete within 1.5 minutes
Why? The Build Workflow begins in the Fetch-Source stage, checking out the Entire Repo and uploading everything in 1.5 minutes, followed by the Select-Builds stage (arch.yml) reading the PR Labels. Before 1.5 minutes, rightfully our workflow_run trigger would have written the PR Labels to the PR.
(Then again: NuttX App PRs always trigger a Complete Build, never a Simple Arch-Specific Build)


New PR Labeling starts at 6:18:42, ends at 6:18:59. Total 17 elapsed seconds for New PR Labeling.


Old PR Labeling starts at 17:56:45, ends at 17:57:00. Total 15 elapsed seconds for Old PR Labeling.


Zizmor Security Scan


Zizmor Security Scan should not report any Security Issues. However Zizmor flags workflow_run as a Potential Security Issue, because it's unable to analyse the code inside the workflow. workflow_run is not forbidden in the ASF GitHub Actions Security Policy.

$ git clone https://github.com/lupyuen8/nuttx-apps

$ zizmor nuttx-apps/.github/workflows/labeler.yml
🌈 zizmor v1.22.0
 INFO audit: zizmor: 🌈 completed nuttx-apps/.github/workflows/labeler.yml
No findings to report. Good job! (4 suppressed)

$ zizmor nuttx-apps/.github/workflows/pr_labeler.yml
🌈 zizmor v1.22.0
 INFO audit: zizmor: 🌈 completed nuttx-apps/.github/workflows/pr_labeler.yml
error[dangerous-triggers]: use of fundamentally insecure workflow trigger
  --> nuttx-apps/.github/workflows/pr_labeler.yml:22:1
   |
22 | / on:
23 | |   workflow_run:
24 | |     workflows: ["Pull Request Labeler"]
25 | |     types:
26 | |       - completed
   | |_________________^ workflow_run is almost always used insecurely
   |
   = note: audit confidence → Medium

7 findings (6 suppressed): 0 informational, 0 low, 0 medium, 1 high
No results found