Docs and tests with Droid Exec

A nightly GitHub Actions workflow that updates docstrings and READMEs and writes missing unit tests for code changed in the last seven days, runs the tests without credentials, and opens a pull request.

This recipe runs every night in GitHub Actions. It finds source files changed in the last seven days, asks Droid to bring their docstrings and related READMEs up to date, and writes missing unit tests for the same files. A separate job runs the tests, and a third opens one pull request for review.

Tip

If you'd rather not maintain a workflow file, the Docs Refresh and Test Coverage starters in Custom Automations do similar work on each push and each pull request.

How it works

The workflow is split into three jobs so that no credential is present while repository tests run:

  • draft holds FACTORY_API_KEY and read-only repository access. Droid updates docs and writes tests at --auto low, which allows file edits but blocks installs and test runs.
  • verify holds no credentials. It applies Droid's changes, installs dependencies, and runs the tests.
  • open-pr holds PR_TOKEN. It applies the changes, then commits, pushes, and opens the pull request. It runs no repository code.

Every checkout that doesn't push uses persist-credentials: false, so the job's GitHub token isn't left in the Git configuration for later steps to read.

Prerequisites

  • Add FACTORY_API_KEY to the repository's Actions secrets. Generate a key from the Factory settings page.
  • Add a PR_TOKEN secret that holds a fine-grained personal access token or a GitHub App token with Contents and Pull requests write access to the repository. GitHub doesn't start other workflows for pull requests opened with the default GITHUB_TOKEN, so with a PR_TOKEN your normal CI runs on the pull request. Leave the token's Workflows permission off, so a tampered patch can't change your workflow files.
  • Add your usual runtime setup, such as actions/setup-node, to the verify job. The example uses npm.

The workflow

.github/workflows/droid-daily-docs-tests.yml
name: Droid daily docs and tests
 
on:
  schedule:
    - cron: '0 3 * * *' # Every day at 03:00 UTC
  workflow_dispatch:
 
permissions:
  contents: read
 
jobs:
  draft:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    outputs:
      changed: ${{ steps.patch.outputs.changed }}
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0 # full history, so git log can see the last 7 days
          persist-credentials: false
 
      - name: Install Droid CLI
        run: |
          curl -fsSL https://app.factory.ai/cli | sh
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
 
      - name: List recently changed source files
        id: changed
        run: |
          git log --since='7 days ago' --first-parent --diff-merges=first-parent \
            --name-only --pretty=format: -- \
            '*.ts' '*.tsx' '*.js' '*.py' \
            ':!*.test.*' ':!*.spec.*' \
            ':!*_test.py' ':!test_*.py' ':!*/test_*.py' \
            | sort -u | sed '/^$/d' \
            | xargs -r -d '\n' git ls-files -- > changed-files.txt
          if [ -s changed-files.txt ]; then
            echo "any=true" >> "$GITHUB_OUTPUT"
          fi
 
      - name: Update docs
        id: docs
        if: steps.changed.outputs.any == 'true'
        env:
          FACTORY_API_KEY: ${{ secrets.FACTORY_API_KEY }}
        run: |
          droid exec --auto low "For each file listed in changed-files.txt:
          update docstrings and JSDoc comments that no longer match the code,
          and add them to exported functions and classes that have none.
          If a change adds or alters a public feature, update the nearest
          README. Do not change code. Follow the documentation style already
          used in each file. Only read and edit files. Do not run shell
          commands. Append a short summary of what you changed to
          updates.md." \
            || { echo "::warning::droid exec exited $?. Keeping its edits."
                 echo "failed=true" >> "$GITHUB_OUTPUT"; }
 
      - name: Write missing tests
        id: tests
        if: steps.changed.outputs.any == 'true'
        env:
          FACTORY_API_KEY: ${{ secrets.FACTORY_API_KEY }}
        run: |
          droid exec --auto low "For each file listed in changed-files.txt,
          find exported behavior with no unit tests and add focused tests for
          it, including error cases. Follow the test framework, file layout,
          and patterns the repository already uses. Do not change source
          code. Only read and edit files. Do not run shell commands; a later
          job runs the tests. Append a short summary of the tests you added
          to updates.md." \
            || { echo "::warning::droid exec exited $?. Keeping its edits."
                 echo "failed=true" >> "$GITHUB_OUTPUT"; }
 
      - name: Save changes
        id: patch
        env:
          DOCS_FAILED: ${{ steps.docs.outputs.failed }}
          TESTS_FAILED: ${{ steps.tests.outputs.failed }}
        run: |
          git add -A -- . ':!changed-files.txt' ':!updates.md'
          git diff --cached --binary > changes.patch
          if [ -s changes.patch ]; then
            echo "changed=true" >> "$GITHUB_OUTPUT"
          elif [ "$DOCS_FAILED" = true ] && [ "$TESTS_FAILED" = true ]; then
            echo "::error::Both droid exec runs failed without edits." \
              "Check FACTORY_API_KEY and the step logs."
            exit 1
          fi
 
      - uses: actions/upload-artifact@v7
        if: steps.patch.outputs.changed == 'true'
        with:
          name: droid-changes
          path: |
            changes.patch
            updates.md
 
  verify:
    needs: draft
    if: needs.draft.outputs.changed == 'true'
    runs-on: ubuntu-latest
    outputs:
      tests: ${{ steps.test.outputs.result }}
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
 
      - uses: actions/download-artifact@v8
        with:
          name: droid-changes
 
      # Add your runtime setup here, such as actions/setup-node.
      - name: Run tests without credentials
        id: test
        run: |
          git apply changes.patch
          if npm ci && npm test; then result=passed; else result=failed; fi
          echo "result=$result" >> "$GITHUB_OUTPUT"
 
  open-pr:
    needs: verify
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          token: ${{ secrets.PR_TOKEN }}
 
      - uses: actions/download-artifact@v8
        with:
          name: droid-changes
 
      - name: Open pull request
        env:
          GH_TOKEN: ${{ secrets.PR_TOKEN }}
          TESTS: ${{ needs.verify.outputs.tests }}
        run: |
          touch updates.md
          git apply changes.patch
          branch="droid/daily-docs-tests-$(date +%Y%m%d)-$GITHUB_RUN_ID"
          git config user.name "github-actions[bot]"
          git config user.email \
            "41898282+github-actions[bot]@users.noreply.github.com"
          git checkout -b "$branch"
          git add -A -- . ':!changes.patch' ':!updates.md'
          git commit -m "chore: update docs and tests for recent changes"
          git push -u origin "$branch"
          draft=false
          if [ "$TESTS" != "passed" ]; then
            draft=true
            cat >> updates.md <<'EOF'
 
          Install or some new tests failed in the verify job.
          Check the workflow run before you mark this ready.
          EOF
          fi
          gh pr create --draft="$draft" \
            --title "Daily docs and test updates" \
            --body-file updates.md

At --auto low, droid exec exits with an error if Droid tries a command that needs more autonomy, even after it has finished editing. The two Droid steps keep the edits and log a warning instead of failing, so the verify job decides whether the changes are good. If both steps fail and leave no edits, as with an expired FACTORY_API_KEY, the Save changes step fails the run.

Droid doesn't run the tests it writes, so some may fail on the first pass. When that happens, the workflow opens the pull request as a draft with a note, and your CI shows which tests failed. The same happens if npm ci fails, for example because Droid added a dependency to package.json but can't update the lockfile at --auto low.

Example pull request

In a small TypeScript repository, one run added JSDoc to six files, updated the README's export list, and wrote six Vitest files that follow the existing test layout. A typical docstring change:

Diff
+/**
+ * Returns the project owned by `userId` and records a `project.read`
+ * audit event.
+ *
+ * @throws {NotFoundError} If no project has the given `projectId`.
+ * @throws {HttpError} With status 403 if `userId` does not own the project.
+ */
 export function getProject(projectId: string, userId: string) {

Adapt the workflow

  • Languages. Edit the path patterns in the List recently changed source files step, and the install and test commands in verify, to match your stack.
  • Window. Change 7 days ago to match how often the workflow runs. A nightly run with a seven-day window revisits the same files, which lets Droid fill gaps a previous run left, but costs more. A one-day window is cheaper.
  • Scope. For large repositories, split the file list into chunks and run one droid exec per chunk. See Batch and parallel patterns.
  • Review load. If daily pull requests are too frequent, run the workflow weekly or skip days when the previous pull request is still open.
  • No PR token. If you can't add PR_TOKEN, use github.token in the open-pr job, grant it contents: write and pull-requests: write, and allow Actions to create pull requests under Settings → Actions → General. Your CI won't run on those pull requests until someone closes and reopens them or pushes a commit.