Docs and tests with Droid Exec
A nightly GitHub Actions workflow that updates docstrings and READMEs and writes missing unit tests for code changed in the last seven days, runs the tests without credentials, and opens a pull request.
This recipe runs every night in GitHub Actions. It finds source files changed in the last seven days, asks Droid to bring their docstrings and related READMEs up to date, and writes missing unit tests for the same files. A separate job runs the tests, and a third opens one pull request for review.
If you'd rather not maintain a workflow file, the Docs Refresh and Test Coverage starters in Custom Automations do similar work on each push and each pull request.
How it works
The workflow is split into three jobs so that no credential is present while repository tests run:
draftholdsFACTORY_API_KEYand read-only repository access. Droid updates docs and writes tests at--auto low, which allows file edits but blocks installs and test runs.verifyholds no credentials. It applies Droid's changes, installs dependencies, and runs the tests.open-prholdsPR_TOKEN. It applies the changes, then commits, pushes, and opens the pull request. It runs no repository code.
Every checkout that doesn't push uses persist-credentials: false, so the job's GitHub token isn't left in the Git configuration for later steps to read.
Prerequisites
- Add
FACTORY_API_KEYto the repository's Actions secrets. Generate a key from the Factory settings page. - Add a
PR_TOKENsecret that holds a fine-grained personal access token or a GitHub App token with Contents and Pull requests write access to the repository. GitHub doesn't start other workflows for pull requests opened with the defaultGITHUB_TOKEN, so with aPR_TOKENyour normal CI runs on the pull request. Leave the token's Workflows permission off, so a tampered patch can't change your workflow files. - Add your usual runtime setup, such as
actions/setup-node, to theverifyjob. The example uses npm.
The workflow
name: Droid daily docs and tests
on:
schedule:
- cron: '0 3 * * *' # Every day at 03:00 UTC
workflow_dispatch:
permissions:
contents: read
jobs:
draft:
runs-on: ubuntu-latest
timeout-minutes: 60
outputs:
changed: ${{ steps.patch.outputs.changed }}
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0 # full history, so git log can see the last 7 days
persist-credentials: false
- name: Install Droid CLI
run: |
curl -fsSL https://app.factory.ai/cli | sh
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: List recently changed source files
id: changed
run: |
git log --since='7 days ago' --first-parent --diff-merges=first-parent \
--name-only --pretty=format: -- \
'*.ts' '*.tsx' '*.js' '*.py' \
':!*.test.*' ':!*.spec.*' \
':!*_test.py' ':!test_*.py' ':!*/test_*.py' \
| sort -u | sed '/^$/d' \
| xargs -r -d '\n' git ls-files -- > changed-files.txt
if [ -s changed-files.txt ]; then
echo "any=true" >> "$GITHUB_OUTPUT"
fi
- name: Update docs
id: docs
if: steps.changed.outputs.any == 'true'
env:
FACTORY_API_KEY: ${{ secrets.FACTORY_API_KEY }}
run: |
droid exec --auto low "For each file listed in changed-files.txt:
update docstrings and JSDoc comments that no longer match the code,
and add them to exported functions and classes that have none.
If a change adds or alters a public feature, update the nearest
README. Do not change code. Follow the documentation style already
used in each file. Only read and edit files. Do not run shell
commands. Append a short summary of what you changed to
updates.md." \
|| { echo "::warning::droid exec exited $?. Keeping its edits."
echo "failed=true" >> "$GITHUB_OUTPUT"; }
- name: Write missing tests
id: tests
if: steps.changed.outputs.any == 'true'
env:
FACTORY_API_KEY: ${{ secrets.FACTORY_API_KEY }}
run: |
droid exec --auto low "For each file listed in changed-files.txt,
find exported behavior with no unit tests and add focused tests for
it, including error cases. Follow the test framework, file layout,
and patterns the repository already uses. Do not change source
code. Only read and edit files. Do not run shell commands; a later
job runs the tests. Append a short summary of the tests you added
to updates.md." \
|| { echo "::warning::droid exec exited $?. Keeping its edits."
echo "failed=true" >> "$GITHUB_OUTPUT"; }
- name: Save changes
id: patch
env:
DOCS_FAILED: ${{ steps.docs.outputs.failed }}
TESTS_FAILED: ${{ steps.tests.outputs.failed }}
run: |
git add -A -- . ':!changed-files.txt' ':!updates.md'
git diff --cached --binary > changes.patch
if [ -s changes.patch ]; then
echo "changed=true" >> "$GITHUB_OUTPUT"
elif [ "$DOCS_FAILED" = true ] && [ "$TESTS_FAILED" = true ]; then
echo "::error::Both droid exec runs failed without edits." \
"Check FACTORY_API_KEY and the step logs."
exit 1
fi
- uses: actions/upload-artifact@v7
if: steps.patch.outputs.changed == 'true'
with:
name: droid-changes
path: |
changes.patch
updates.md
verify:
needs: draft
if: needs.draft.outputs.changed == 'true'
runs-on: ubuntu-latest
outputs:
tests: ${{ steps.test.outputs.result }}
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: actions/download-artifact@v8
with:
name: droid-changes
# Add your runtime setup here, such as actions/setup-node.
- name: Run tests without credentials
id: test
run: |
git apply changes.patch
if npm ci && npm test; then result=passed; else result=failed; fi
echo "result=$result" >> "$GITHUB_OUTPUT"
open-pr:
needs: verify
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
token: ${{ secrets.PR_TOKEN }}
- uses: actions/download-artifact@v8
with:
name: droid-changes
- name: Open pull request
env:
GH_TOKEN: ${{ secrets.PR_TOKEN }}
TESTS: ${{ needs.verify.outputs.tests }}
run: |
touch updates.md
git apply changes.patch
branch="droid/daily-docs-tests-$(date +%Y%m%d)-$GITHUB_RUN_ID"
git config user.name "github-actions[bot]"
git config user.email \
"41898282+github-actions[bot]@users.noreply.github.com"
git checkout -b "$branch"
git add -A -- . ':!changes.patch' ':!updates.md'
git commit -m "chore: update docs and tests for recent changes"
git push -u origin "$branch"
draft=false
if [ "$TESTS" != "passed" ]; then
draft=true
cat >> updates.md <<'EOF'
Install or some new tests failed in the verify job.
Check the workflow run before you mark this ready.
EOF
fi
gh pr create --draft="$draft" \
--title "Daily docs and test updates" \
--body-file updates.mdAt --auto low, droid exec exits with an error if Droid tries a command that needs more autonomy, even after it has finished editing. The two Droid steps keep the edits and log a warning instead of failing, so the verify job decides whether the changes are good. If both steps fail and leave no edits, as with an expired FACTORY_API_KEY, the Save changes step fails the run.
Droid doesn't run the tests it writes, so some may fail on the first pass. When that happens, the workflow opens the pull request as a draft with a note, and your CI shows which tests failed. The same happens if npm ci fails, for example because Droid added a dependency to package.json but can't update the lockfile at --auto low.
Example pull request
In a small TypeScript repository, one run added JSDoc to six files, updated the README's export list, and wrote six Vitest files that follow the existing test layout. A typical docstring change:
+/**
+ * Returns the project owned by `userId` and records a `project.read`
+ * audit event.
+ *
+ * @throws {NotFoundError} If no project has the given `projectId`.
+ * @throws {HttpError} With status 403 if `userId` does not own the project.
+ */
export function getProject(projectId: string, userId: string) {Adapt the workflow
- Languages. Edit the path patterns in the List recently changed source files step, and the install and test commands in
verify, to match your stack. - Window. Change
7 days agoto match how often the workflow runs. A nightly run with a seven-day window revisits the same files, which lets Droid fill gaps a previous run left, but costs more. A one-day window is cheaper. - Scope. For large repositories, split the file list into chunks and run one
droid execper chunk. See Batch and parallel patterns. - Review load. If daily pull requests are too frequent, run the workflow weekly or skip days when the previous pull request is still open.
- No PR token. If you can't add
PR_TOKEN, usegithub.tokenin theopen-prjob, grant itcontents: writeandpull-requests: write, and allow Actions to create pull requests under Settings → Actions → General. Your CI won't run on those pull requests until someone closes and reopens them or pushes a commit.