Adding Weighted Task Selection to a GitHub Agentic Workflow

In my previous post I introduced Repo Assist: Crunching the Technical Debt with GitHub Agentic Workflows.

In my first version of Repo Assist, tasks were selected by round-robin: the agent tracked which tasks it had run most recently and rotated through them. This worked, but it had a problem – the agent was spending equal time on labelling in a repo with zero unlabelled issues as it was on investigating bugs in a repo with 200 open issues. The priorities of the repository weren’t reflected in the task selection.

I’ve now updated Repo Assist to use a weighted random task selection that adapts to the live state of the repository. See PR #241. The idea is simple: before the agent starts reasoning, a deterministic pre-step runs a small Python script that fetches repo data and computes weights for each task. Two tasks are then drawn (without replacement) using those weights, and the agent prioritises those two tasks plus the mandatory monthly summary.

The key insight is that GitHub Agentic Workflows supports a steps: block that runs deterministic shell commands before the agentic step begins. This is where the weighting happens — no LLM involved in the selection, just arithmetic:

steps:
  - name: Fetch repo data for task weighting
    env:
      GH_TOKEN: ${{ github.token }}
    run: |
      mkdir -p /tmp/gh-aw
      # Fetch open issues with labels (up to 500)
      gh issue list --state open --limit 500 --json number,labels > /tmp/gh-aw/issues.json
      # Fetch open PRs with titles (up to 200)
      gh pr list --state open --limit 200 --json number,title > /tmp/gh-aw/prs.json
      # Compute task weights and select two tasks for this run
      python3 - << 'EOF'
      import json, random, os
      with open('/tmp/gh-aw/issues.json') as f:
          issues = json.load(f)
      with open('/tmp/gh-aw/prs.json') as f:
          prs = json.load(f)
      open_issues     = len(issues)
      unlabelled      = sum(1 for i in issues if not i.get('labels'))
      repo_assist_prs = sum(1 for p in prs if p['title'].startswith('[Repo Assist]'))
      other_prs       = sum(1 for p in prs if not p['title'].startswith('[Repo Assist]'))
      weights = {
          1:  1   + unlabelled,
          2:  3   + 0.3 * open_issues,
          3:  3   + 0.3 * open_issues,
          4:  5   + 0.2 * open_issues,
          5:  5   + 0.1 * open_issues,
          6:  float(repo_assist_prs),
          7:  0.1 * other_prs,
          8:  3   + 0.05 * open_issues,
          9:  3   + 0.05 * open_issues,
          10: 3   + 0.05 * open_issues,
      }
      # Seed with run ID for reproducibility within a run
      run_id = int(os.environ.get('GITHUB_RUN_ID', '0'))
      rng = random.Random(run_id)
      task_ids     = list(weights.keys())
      task_weights = [weights[t] for t in task_ids]
      # Weighted sample without replacement (pick 2 distinct tasks)
      chosen, seen = [], set()
      for t in rng.choices(task_ids, weights=task_weights, k=30):
          if t not in seen:
              seen.add(t)
              chosen.append(t)
          if len(chosen) == 2:
              break
      print('=== Repo Assist Task Selection ===')
      print(f'Open issues       : {open_issues}')
      print(f'Unlabelled issues : {unlabelled}')
      print(f'Repo Assist PRs   : {repo_assist_prs}')
      print(f'Other open PRs    : {other_prs}')
      print()
      print('Task weights:')
      for t, w in weights.items():
          tag = ' <-- SELECTED' if t in chosen else ''
          print(f'  Task {t:2d}: weight {w:6.1f}{tag}')
      print()
      print(f'Selected tasks for this run: Task {chosen[0]} and Task {chosen[1]}')
      result = {
          'open_issues': open_issues, 'unlabelled_issues': unlabelled,
          'repo_assist_prs': repo_assist_prs, 'other_prs': other_prs,
          'weights': {str(k): round(v, 2) for k, v in weights.items()},
          'selected_tasks': chosen,
      }
      with open('/tmp/gh-aw/task_selection.json', 'w') as f:
          json.dump(result, f, indent=2)
      EOF

The weights are intentionally straightforward. A repo with 100 open issues and 20 unlabelled will heavily favour labelling (weight 21) and investigation/fix (weight 33 each). A clean repo with 5 open issues and no unlabelled ones will spread weight more evenly across engineering, performance, and testing tasks. Task 6 (maintaining Repo Assist’s own PRs) is weighted purely by how many open Repo Assist PRs exist – if there are none, it gets zero weight and is never selected.

The prompt then tells the agent to read the selection file and execute exactly those tasks:

Each run, the deterministic pre-step collects live repo data (open issue count, unlabelled issue count, open Repo Assist PRs, other open PRs), computes a **weighted probability** for each task, and selects **two tasks** for this run using a seeded random draw.

The weighting scheme naturally adapts to repo state:

• When unlabelled issues pile up, Task 1 (labelling) dominates.
• When there are many open issues, Tasks 2 and 3 (commenting and fixing) get more weight.
• As the backlog clears, Tasks 4–10 (engineering, improvements, nudges, forward progress) draw more evenly.

This is a small change but it makes a meaningful difference. The round-robin approach treated every task as equally important regardless of repo state. The weighted approach means Repo Assist now does the right thing more often: when there’s a mountain of unlabelled issues it labels, when the backlog is clear it invests in engineering. The agent doesn’t have to decide priorities — the arithmetic does it before the agent even starts.

I also took the opportunity to reorganise the tasks themselves. The old Task 1 (Triage and Comment) was split into separate labelling (Task 1) and investigation (Task 2) tasks. Task 7 (Manage Labels) was promoted to Task 1 since labelling is foundational. New dedicated tasks for Performance Improvements (Task 8) and Testing Improvements (Task 9) were broken out from the old catch-all improvement task. Release Preparation and Welcome New Contributors were folded into general guidelines rather than occupying their own task slots.

As part of this change I also changed the default to run 4 times a day. This is because many people are reporting Repo Assist as genuinely useful for addressing serious technical debt, and this default cadence gives a good strong rate of forward progress.

The full change is in https://github.com/githubnext/agentics/pull/241.

You can update to the latest version of Repo Assist with

gh aw update 
<commit and push changes>

This will merge in my updates with any changes you’ve made locally. Simple!

If you have ideas for Repo Assist or feedback, or for GitHub Agentic Workflows more generally, please drop an issue at https://github.com/githubnext/agentics/issues or https://github.com/githubnext/gh-aw/issues