Verify links in awesome projects
awesome_bot checks for valid URLs in a file, it can be used to verify pull requests updating a README.
$ gem install awesome_bot
Usage: awesome_bot [file or files]
awesome_bot [options]
-f, --files [files] Comma separated files to check
-a, --allow [errors] Status code errors to allow
--allow-dupe Duplicate URLs are allowed
--allow-ssl SSL errors are allowed
--allow-redirect Redirected URLs are allowed
--allow-timeout URLs that time out are allowed
--base-url [base url] Base URL to use for relative links
-d, --request-delay [seconds] Set request delay
-t, --set-timeout [seconds] Set connection timeout (default: 30)
--skip-save-results Skip saving results
-w, --white-list [urls] Comma separated URLs to white list-
You can check multiple files (comma separated or
*pattern, look below for details). -
By default, duplicate URLs or any status code other than
200are flagged as failures.- Use option
--allow-dupeto allow duplicates. - Use option
--allow-redirectto allow redirects. - Use option
--allowto allow specific status code errors. - Use option
--white-list(-wfor short) to prevent links from being flagged:-w domain1.com/post/article,domain2.comwhite listsdomain1.com/post/articleand all links matchingdomain2.com.
- Use option
$ awesome_bot README.md
> Checking links in README.md
Links found: 56, 37 unique
01. https://github.com/sindresorhus/awesome
02. http://i.giphy.com/urvsFBDfR6N32.gif
03. https://travis-ci.org/dkhamsing/awesome_bot.svg
# ...
37. https://github.com/dkhamsing
Checking URLs: ✓✓✓→?✓→✓→→✓✓→✓✓✓→✓✓✓✓✓✓✓✓✓✓✓→✓✓✓✓✓→✓✓
Issues :-(
> Links
1. [L007] 301 https://travis-ci.org/dkhamsing/awesome_bot.svg → https://api.travis-ci.org/dkhamsing/awesome_bot.svg
2. [L008] 302 https://badge.fury.io/rb/awesome_bot → http://rubygems.org/gems/awesome_bot
# ...
> Dupes
1. [L03] https://github.com/sindresorhus/awesome
2. [L05] http://i.giphy.com/urvsFBDfR6N32.gif
# ...$ awesome_bot README.md --allow-dupe --allow-redirect -w rubydoc,giphy
# allow redirects, dupes and white list all links matching rubydoc and giphy
$ awesome_bot README.md,README-zh.md
# check links in 2 files
$ awesome_bot docs/*.md
# check all Markdown files in the docs/ directory
$ awesome_bot README.md --allow-timeout -t 5
# speed up validation by setting a timeout of 5 seconds per link request and allowing timeouts
$ awesome_bot README.md --allow 403,429
# allow status code errors 403 and 429
# --allow 301 would be similar to --allow-redirect
$ awesome_bot README.md --base-url https://github.com/IDR/idr-notebooks/blob/master/
# check relative links using the base URL provided(master) $ git branch
* master
(master) $ git checkout -b new-branch
Switched to a new branch 'new-branch'
(new-branch) $ touch new-readme.md && echo 'https://github.com/dkhamsing' >> new-readme.md
(new-branch) $ git add new-readme.md
(new-branch) $ git commit -m 'Testing'
[new-branch ef47336] Testing
1 file changed, 1 insertion(+)
create mode 100644 new-readme.md
(new-branch) $ git diff master.. --name-only | grep '.md' | xargs awesome_bot
> Checking links in new-readme.md
Links to check: 1
1. https://github.com/dkhamsing
Checking URLs: ✓
No issues :-)
Wrote results to ab-results-new-readme.md.jsonIf you do not want to install Ruby or its dependencies you can simply use Docker and Docker image.
Here is an example for checking the links in the Markdown files in your current directory/subdirectories:
docker run -ti --rm -v $PWD:/mnt:ro dkhamsing/awesome_bot --white-list "test.com" --allow-dupe --allow-redirect --skip-save-results `find . -name "*.md"`or just check the links in a single file located at ./templates/ubuntu.md:
docker run -ti --rm -v $PWD:/mnt:ro dkhamsing/awesome_bot --allow-dupe --allow-redirect --skip-save-results ./templates/ubuntu.mdYou always need to specify the path to the file so you cannot simply use *.md; instead use ls *.md":
docker run -ti --rm -v $PWD:/mnt:ro dkhamsing/awesome_bot --white-list "test.com" --allow-dupe --allow-redirect --skip-save-results `ls *.md`irb(main):001:0> require 'awesome_bot'
=> true
irb(main):002:0> content = File.read 'README.md'
=> "..."
irb(main):003:0> result = AwesomeBot.check content
=> #<AwesomeBot::Result:0x007fdde39f4408 @links=...>
# AwesomeBot Result with success, statuses_issues, dupes and more
irb(main):004:0> puts result.success ? 'No errors' : ':-('
:-(More information at rubydoc.
Does your GitHub README contain a lot of links? awesome_bot can help you validate them when a pull request is created (or a commit is pushed). It is used by:
- https://github.com/tiimgreen/github-cheat-sheet
- https://github.com/enaqx/awesome-react
- https://github.com/ziadoz/awesome-php
- https://github.com/vsouza/awesome-ios
- https://github.com/alebcay/awesome-shell
- https://github.com/matteocrippa/awesome-swift
and more.
Tips
To use awesome_bot with GitHub Actions (workflows), here is an example:
name: Ruby
on:
push:
branches: [ '*' ]
pull_request:
branches: [ '*' ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Ruby 2.6
uses: ruby/setup-ruby@v1
with:
ruby-version: '2.6'
- name: Checks
run: |
gem install awesome_bot
awesome_bot check-unique.txt --allow-ssl -a 302,429 -w xbmc/xbmcTo use awesome_bot with Travis CI, connect your repo and create a .travis.yml file.
language: ruby
rvm: 2.4.1
before_script: gem install awesome_bot
script: awesome_bot README.mdTo turn off email notifications, add the lines below
notifications:
email: falseIn case you want to use the docker image inside Travis CI follow this example which will check broken links in all *.md files in your repository:
sudo: required
services:
- docker
script:
# Link Checks
- docker run -ti --rm -v $PWD:/mnt:ro dkhamsing/awesome_bot --allow-dupe --allow-redirect --skip-save-results `find . -name "*.md"`CircleCI, Codeship, and Semaphore CI support running tests without adding a file to the repo (a public configuration file can however help others contribute).
# Codeship
Setup
rvm use 2.4.1 --install
gem install awesome_bot
Test
awesome_bot README.md
# Semaphore CI
Language: Ruby
Ruby version: 2.4.1
Databases for: don't generate
Setup:
gem install awesome_bot
awesome_bot README.md
To add the Travis CI build status badge above to your project, use the following code
[](https://travis-ci.org/<username>/<project>)
i.e.
[](https://travis-ci.org/dkhamsing/awesome_bot)
As it happens, the default code snippet provided contains a redirect so adding a badge could fail your status 😭.. one way to fix this is to white list travis-ci, i.e.
- awesome_bot README.md --white-list travis-ci
You can also add a badge for other CI tools, check out shields.io.
Integrate awesome_bot with Danger and have results reported back to the pull request.
Here's the step in your Dangerfile:
# Check links
require 'json'
results = File.read 'ab-results-README.md-markdown-table.json'
j = JSON.parse results
if j['error']==true
fail j['title']
markdown j['message']
endThis project is available under the MIT license. See the LICENSE file for more info.