KofTwentyTwo/gclo
2
0 Fork 0
Code Issues 28 Pull Requests 1 Actions Packages Projects Releases 22 Wiki Activity

Tests that depend on the machine say so, and fail on CI when they cannot run #190

New Issue
Open
opened 2026-11-11 05:58:41 +00:00 by james.maes · 4 comments
james.maes commented 2026-11-11 05:58:41 +00:00
Owner
Copy link
Copy source

From the release reviews of 1.1.1 (2026-11-10). Not part of 1.1.1; planned for 1.2.0 unless said otherwise.

  • The tests that run the WSL clone script need a POSIX shell that can find git; without one they pass and say "Not run" only in their output. One test of links depends on the right to create symbolic links. On CI a test that did not run should fail, so that coverage that is counted on is really there.
  • The script tests never run under dash, which is what a WSL distribution's sh usually is.
  • Several "still waiting" tests use a 200 ms window: they cannot fail by accident, but under load they pass without proving the wait.
  • AccountsStoreSharedFileTests.WhileChangesAreBeingSaved_TheFileIsNeverAbsent failed once in a full run while the machine was under heavy load from other test runs, and passed in every run after.
  • The table-order tests do not cover a column sorted by the user, the column filters, or a status changed by a path recovery outside a run.
  • The UI suite is unreliable on a desktop that is in use.
  • LibGit2GitClient is excluded from the coverage gate as a whole. Measured with the exclusion removed: about 94% of its lines are covered, and about half of the gaps are ordinary decisions, not the transport and cancellation arms the exclusion is justified with. Bring it under the gate, or narrow the exclusion to those arms.
From the release reviews of 1.1.1 (2026-11-10). Not part of 1.1.1; planned for 1.2.0 unless said otherwise. - The tests that run the WSL clone script need a POSIX shell that can find git; without one they pass and say "Not run" only in their output. One test of links depends on the right to create symbolic links. On CI a test that did not run should fail, so that coverage that is counted on is really there. - The script tests never run under `dash`, which is what a WSL distribution's `sh` usually is. - Several "still waiting" tests use a 200 ms window: they cannot fail by accident, but under load they pass without proving the wait. - `AccountsStoreSharedFileTests.WhileChangesAreBeingSaved_TheFileIsNeverAbsent` failed once in a full run while the machine was under heavy load from other test runs, and passed in every run after. - The table-order tests do not cover a column sorted by the user, the column filters, or a status changed by a path recovery outside a run. - The UI suite is unreliable on a desktop that is in use. - `LibGit2GitClient` is excluded from the coverage gate as a whole. Measured with the exclusion removed: about 94% of its lines are covered, and about half of the gaps are ordinary decisions, not the transport and cancellation arms the exclusion is justified with. Bring it under the gate, or narrow the exclusion to those arms.
james.maes added the deferredquality labels 2026-11-11 05:58:41 +00:00
james.maes commented 2026-11-11 08:32:37 +00:00
Author
Owner
Copy link
Copy source

Seen again on 2026-11-11: VaultTests.CredentialManager_RoundTrips_AgainstTheRealStore failed once in a full run of the engine suite and passed when run alone a minute later. It is the one test that uses the real credential store.

Seen again on 2026-11-11: `VaultTests.CredentialManager_RoundTrips_AgainstTheRealStore` failed once in a full run of the engine suite and passed when run alone a minute later. It is the one test that uses the real credential store.
james.maes commented 2026-11-11 12:59:40 +00:00
Author
Owner
Copy link
Copy source

More detail, 2026-11-11. Two of the tests that use the real credential store failed in separate full runs of the engine suite while another full run of the suite was going on in a second checkout on the same machine (a reviewer's):

  • VaultTests.CredentialManager_RoundTrips_AgainstTheRealStore (twice);
  • VaultEdgeTests.AScopedVault_DoesNotSeeAnotherScopesEntry_AndCleansUpItsOwn: Assert.Null(one.TryRetrieve(id)) after one.Delete(id) got the stored value back.

Both use a scope and an id of their own per run, so two runs cannot name the same entry. What remains is the store itself under concurrent use: a delete that returned and an entry that was still read. Worth finding out whether CredentialManagerVault.Delete can return without the entry being gone (an error it does not report), because that would also be a token left behind in the app.

More detail, 2026-11-11. Two of the tests that use the real credential store failed in separate full runs of the engine suite while another full run of the suite was going on in a second checkout on the same machine (a reviewer's): - `VaultTests.CredentialManager_RoundTrips_AgainstTheRealStore` (twice); - `VaultEdgeTests.AScopedVault_DoesNotSeeAnotherScopesEntry_AndCleansUpItsOwn`: `Assert.Null(one.TryRetrieve(id))` after `one.Delete(id)` got the stored value back. Both use a scope and an id of their own per run, so two runs cannot name the same entry. What remains is the store itself under concurrent use: a delete that returned and an entry that was still read. Worth finding out whether `CredentialManagerVault.Delete` can return without the entry being gone (an error it does not report), because that would also be a token left behind in the app.
james.maes commented 2026-11-11 22:29:10 +00:00
Author
Owner
Copy link
Copy source

Seen twice more on 2026-11-11, each time as the only failure of a full gate run and green on the rerun: VaultTests.CredentialManager_RoundTrips_AgainstTheRealStore. Neither time was another suite run of this session going on, so an overlap of two runs is not the only cause; a gclo window was open on the machine both times.

Seen twice more on 2026-11-11, each time as the only failure of a full gate run and green on the rerun: VaultTests.CredentialManager_RoundTrips_AgainstTheRealStore. Neither time was another suite run of this session going on, so an overlap of two runs is not the only cause; a gclo window was open on the machine both times.
james.maes commented 2026-11-11 22:29:19 +00:00
Author
Owner
Copy link
Copy source

Correction to the comment above: I did not check whether a gclo window was open during those runs, so that part is a guess and should be disregarded. What is established: the test failed twice as the only failure, passed on the rerun, and no other suite run of this session overlapped.

Correction to the comment above: I did not check whether a gclo window was open during those runs, so that part is a guess and should be disregarded. What is established: the test failed twice as the only failure, passed on the rerun, and no other suite run of this session overlapped.
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
.NET
Pull requests that update .NET code
 bug
Something isn't working
 deferred
Agreed follow-up, not scheduled
 dependencies
Pull requests that update a dependency file
 documentation
Improvements or additions to documentation
 duplicate
This issue or pull request already exists
 enhancement
New feature or request
 github_actions
Pull requests that update GitHub Actions code
 good first issue
Good for newcomers
 help wanted
Extra attention is needed
 invalid
This doesn't seem right
 needs-owner
Blocked on an action only the owner can take
 priority
Owner-flagged: work this first
 quality
Standards, simplicity, enforcement
 question
Further information is requested
 refactor
Restructuring without behavior change
 release
Release pipeline / distribution
 wontfix
This will not be worked on
No labels deferred quality
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
forge-recovery (Forge Recovery) james.maes (James Maes)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: KofTwentyTwo/gclo#190