The other day I was looking for solutions to run my Ghost Inspector end-to-end tests on PR builds before they were merged and deployed to production in Netlify.
Completed code: https://github.com/jacobarriola/personal-site/tree/master/plugins/netlify-plugin-ghost-inspector
From the Netlify Build Plugins docs:
Run JavaScript code in response to different events happening during the build-deploy lifecycle.
Perfect! This was exactly what I wanted: at a specific point in the Netlify build, run some JS to trigger a Ghost Inspector test and talk to the GitHub API with the results of that test so that my PR can render the status of the tests like any other task.
API key stuff
To make all of this work, we will need to get and save API keys as Netlify environmental variables.
GitHub Personal Access Token (this is necessary to interact with the GitHub REST API since we'll be making write requests)
Ghost Inspector API Key
Add the keys to the Netlify Dashboard via: Settings > Deploys > Environments
Ghost Inspector - Start URL
To ensure that our dynamic URLs work as expected, we need to remove any hard coded URLs from our Ghost Inspector tests.
To do so, in your Ghost Inspector test, visit Settings > Test Defaults and set the Start URL
to your production URL.
Next, ensure that all of your tests do not have a hard-coded Start URL
and the top. Ghost Inspector will default to the previous url we set or will use a startUrl
that is passed along during an API request.
Netlify: kick things off by setting a pending status in the PR
This isn't mandatory, but I wanted to let PR authors know that Ghost Inspector tests were part of the checks that the PR would complete. Otherwise, the check wouldn't show up until the onSuccess
method is called.
We'll make this GitHub API call by hooking into the onPreBuild
handler.
const { Octokit } = require('@octokit/rest')
module.exports = {
onPreBuild: async () => {
// Only run this in PR deploys
const context = process.env.CONTEXT
if (!context) {
// eslint-disable-next-line no-console
console.log(`No context. Skipping Ghost Inspector tests.`)
return
}
if (context !== 'deploy-preview') {
// eslint-disable-next-line no-console
console.log(`Not in deploy-preview. Skipping Ghost Inspector tests.`)
return
}
const octokit = new Octokit({ auth: process.env.GITHUB_API_TOKEN })
await octokit.request('POST /repos/{owner}/{repo}/statuses/{sha}', {
owner: 'jacobarriola',
repo: 'personal-site',
sha: process.env.COMMIT_REF, // <-- this comes from Netlify
state: 'pending',
description: 'Waiting for Netlify build to complete',
context: 'Ghost Inspector E2E Tests',
})
},
}
Once that code is in, our PR check will now reflect our pending status when a Netlify build starts:
Run Ghost Inspector after the build is complete
We need to run Ghost Inspector once our preview URL is publicly available. The best place to do this is the onSuccess
handler.
Note: I tried to do this on the
onPostBuild
handler but couldn't because the preview URL wasn't publicly available. Therefore, Ghost Inspector could not run its tests.
const fetch = require('cross-fetch')
const { Octokit } = require('@octokit/rest')
module.exports = {
...
onSuccess: async ({ utils }) => {
// Note: for simplicity, I'm skipping a lot of checks. See the final
// version for a less contrived example.
const context = process.env.CONTEXT
const deployUrl = process.env.DEPLOY_PRIME_URL
const ghostInspectorApiKey = process.env.GHOST_INSPECTOR_API_KEY
const suiteId = process.env.GHOST_INSPECTOR_SUITE
const githubApiToken = process.env.GITHUB_API_TOKEN
const octokit = new Octokit({ auth: process.env.GITHUB_API_TOKEN })
try {
console.log(`👻 Starting Ghost Inspector E2E tests on ${deployUrl} ...`)
// Make API request to the Ghost Inspector API
const res = await fetch(
`https://api.ghostinspector.com/v1/suites/${suiteId}/execute/?apiKey=${ghostInspectorApiKey}&startUrl=${deployUrl}`
)
if (res.status >= 400) {
throw new Error(`Bad response from Ghost Inspector server.`)
}
const result = await res.json()
const didTestsPass = result.data.every(test => test.passing === true)
if (false === didTestsPass) {
const testResult = result.data.map(({ name, passing }) => {
return { name, passing }
})
// Send a failure status to the GitHub commit
await octokit.request('POST /repos/{owner}/{repo}/statuses/{sha}', {
owner: 'jacobarriola',
repo: 'personal-site',
sha: process.env.COMMIT_REF, // <-- this comes from Netlify
state: 'failure',
description: 'At least one test failed',
context: 'Ghost Inspector E2E Tests',
target_url: `https://app.ghostinspector.com/suites/${suiteId}`,
})
// Bail
return utils.build.failPlugin(
`🚫 At least one Ghost Inspector test failed. Visit https://app.ghostinspector.com/suites/${suiteId} for details. Failed tests:
${testResult}`
)
}
console.log(`✅ All Ghost Inspector tests passed!`)
// Send a success status to the Github commit
await octokit.request('POST /repos/{owner}/{repo}/statuses/{sha}', {
owner: 'jacobarriola',
repo: 'personal-site',
sha: process.env.COMMIT_REF, // <-- this comes from Netlify
state: 'success',
description: 'All tests passed!',
context: 'Ghost Inspector E2E Tests',
target_url: `https://app.ghostinspector.com/suites/${suiteId}`,
})
return utils.status.show({
title: `Ghost Inspector E2E tests`,
summary: `✅ All tests passed`,
text: `Visit https://app.ghostinspector.com/suites/${suiteId} for test results`,
})
} catch (error) {
return utils.build.failPlugin(error.message)
}
},
}
If all of our tests pass, we should see the status updated in the PR checks along with a link to the test. Great success!
Takeaways
The GitHub API is crazy powerful, once you get through the sheer amount of different endpoints. The key is understanding what you're trying to do and then finding the endpoint that helps you accomplish that.
The Netlify Build API is as straightforward as a developer would hope for - no SDK, no library, etc. It's just a few handlers that are available in the build lifecycle for you to hook into in order to accomplish something. Good stuff.