CONTRIBUTING.md

Contribution Guidelines

Busy Beaver welcomes any, and all, contributions. Every little bit helps!

Table of Contents

• Architecture Overview
  • Application Stack
  • Deployment
  • Services
• Development Environment
  • Slack Integration
  • Setting up Development Environment
  • Verify Installation
  • Running Tests
• Developer Documentation
  • Adding New Requirements
  • Modifying Third-Party Integrations
  • Adding New Third-Party Integrations
  • Creating new CLI Command
• Slack Slash Commands
• Task Queues
  • Creating a New Task
  • Notes
• Miscellaneous
  • Pre-commit
  • PDB++ Configuration
  • Windows Development
  • Pushing Commits to Pull Request made from Fork

Architecture Overview

Busy Beaver is a Python application with a Slack frontend. The application consists of a set of REST endpoints with integrations to various services (GitHub, Slack, YouTube, Trello, etc) using public APIs / 3rd party libraries.

While the application is currently a monolith, it is built with Service-Oriented Architecture (SOA) in mind. API endpoints are implemented using Flask blueprints and services are integrated using the Adapter / Facade pattern.

Busy Beaver tasks are kicked off via CRON jobs that schedule tasks to be run on async workers.

Application Stack

• PostgreSQL
• SQLAlchemy
• Flask
• flask-sqlalchemy
• flask-migrate
• requests

Tests

• pytest
• vcr.py
• pytest-vcr
• pytest-freezegun

vcr.py records cassettes of requests and responses for new tests, and replays them for previously written tests. Make sure to filter credentials

Deployment

• Helm
• Kubernetes
• Docker
• DigitalOcean

Services

We are grateful to the following organizations for providing free services to open source projects:

• GitHub
• DockerHub

Development Environment

It is recommended that users create a personal Slack Workspace to use for bot development. This will allow for independent development without having to wait for project maintainers to grant access to the Busy Beaver development Slack.

Slack Integration

An in-depth guide can be followed at Create a Busy Beaver Slack Dev-Bot

1. Create a Slack workspace
2. Create a Slack App and set the Development Slack Workspace to the workspace from the previous step - Create a Slack Dev-Bot - Init a Slack App
3. Configure the Slack App settings - Create a Slack Dev-Bot - Slack App Settings
4. Install Slack App to Slack Workspace - Create a Slack Dev-Bot - Install App to Workspace

Setting up Development Environment

Prerequisites for developing on Windows

1. pip install pre-commit

2. Install Git-LFS

3. Clone repo

4. cd <directory of the git repo>

5. cp .env.template .env

6. Define the following .env config values:

   Key	Source Value	Details
   SLACK_BOTUSER_OAUTH_TOKEN	From the Dev-Bot Slack App API page under Install App > OAuth Tokens for Your Team > Bot User OAuth Access Token	Value obtained after - Create a Slack Dev-Bot - Define App OAuth and Permissions
   SLACK_SIGNING_SECRET	From the Dev-Bot Slack App API page under Basic Information > App Credentials > Signing Secret	Value obtained after - Create a Slack Dev-Bot - Define App OAuth and Permissions
   SLACK_DEV_WORKSPACE_ID	From Browser Console; follow instructions from StackOverflow
   SLACK_DEV_WORKSPACE_CHANNEL_ID	Channel to install integration into (Need to join a channel). How to get this?

7. make up to refresh environment variables inside of Busy Beaver

Verify Installation

1. make flaskshell

2. Try sending a message to a channel for the development Slack Workspace with the command:

   # Make sure the the channel for example "test" exists!
   slack.post_message("This is a test message", channel="test")`

3. Check #test channel in your Slack Workspace to see if the message was posted

Running Tests

To run the test suite, first, bring up the service. Busy Beaver tests depend on a running database and other pieces so the following Makefile target will start Docker Compose and get the required software going.

$ make up

Once Docker Compose is running, run pytest with:

$ make test

Developer Documentation

Adding New Requirements

Busy Beaver uses pip-tools to manage pip dependencies.

If you need to update any mainline dependencies in requirements.txt, please add the package to requirements.in and run the following command: pip-compile --output-file=requirements.txt requirements.in.

You will need to have pip-tools installed on your local machine to perform this actin.

Modifying Third-Party Integrations

As each integration requires API credentials, it is recommended that contributors create apps for integration connect to their personal accounts.

Adding New Third-Party Integrations

Provide detailed instructions on how to set up the integration so we can roll the feature out to the production instance of Busy Beaver with correct credentials.

Creating new CLI Command

Create a cli.py module. Place this in your application folder next to blueprint.py.

# cli.py

import logging

import click

from .blueprint import bp

logger = logging.getLogger(__name__)


@click.option("--workspace", required=True)
@bp.cli.command("function_name", help="Help prompt")
def function_name(*args, **kwargs)):
    pass

Slack Slash Commands

Users are able to interact with Busy Beaver using the /busybeaver [command] interface provided through the Slack UI. All slash commands are routed to a Busy Beaver endpoint that was enabled earlier via the Slack Slash Command webhook.

Busy Beaver uses the dictionary dispatch pattern to run command-specific logic. We leverage a EventEmitter class, inspired by the node.js EventEmitter, to allow for the creation of new slash commands with a simple decorator interface.

You can create a custom slash command, i.e. /busybeaver news, as follows:

@slash_command_dispatcher.on("news")
def fetch_news(**data):
    # business logic to handle command goes here

• Slack Docs: Slash Commands

Task Queues

Busy Beaver uses RQ to queue jobs and process them in the background with workers. Redis is used as the message broker in this asynchronous architecture.

The Docker Compose development environment spins up a single worker along with a Redis instance. For testing purposes, we set is_async=False to force code to be executed synchronously. Need to find a way to simulate production environment with workers in Travis, or it might make sense to migrate to Jenkins.

Creating a New Task

1. Create a SQLAlchemy model to store task-specific information to the database
2. Run a database migration, $ make migration m="migration message"
3. Create two functions: background job function (decorated with @rq.job) and trigger function to start background job
4. In the trigger function, start the background job and save job specific information to the database
5. Write tests unti you have confidence that your code works and can be reviewed by others

Notes

• The Flask-RQ library provides convenient, Flask-specific helpers. Background tasks are identified using the @rq.job decorator; jobs can be created using the [background_task_function_name].queue(params) method.
• Both the trigger function and background task are unit tested to ensure things occur as expected. A high-level integration test can help codify the requirements of the workflow.
• Currently the workers listen on the default and failed queues. No particular reason, but haven't had the needed to use other queues.

Miscellaneous

Pre-commit

Pre-commit is a tool used to enforce linting with flake8 and code formatting with black. To get started using pre-commit, pip install pre-commit==1.14.4 (this is in the requirements.txt file). Then run pre-commit install to install the flake8 and black environments locally.

Pre-commit will run on files staged for change automatically. You can also check pre-commit hook compliance on staged files by running pre-commit run at any time. Note that pre-commit ignores files that are not staged for change.

PDB++ Configuration

PDB++ improves the debugging experience inside the shell. Create a .pdbrc.py file inside of the root project folder.

# ./.pdbrc.py

import pdb


class Config(pdb.DefaultConfig):
    sticky_by_default = True  # start in sticky mode
    current_line_color = 40  # black

Windows Development

Windows Prerequisites

1. Configure Git on Windows to properly handle line endings

git config --global core.autocrlf input
git config --global core.eol lf

1. Install Windows Sub-System for Linux (WSL)
2. Install Docker on WSL
3. Configure Docker for Windows:
   • Expose daemon on tcp://localhost without TLS
   • Add Shared Drive
     
     
   
4. Proceed with Setting up Development Environment

Windows Troubleshooting

If run into issues with make up, it's most likely Windows and Docker. Try the following:

1. Restart Computer
2. Set File Permissions & Fix Line Endings
3. Launch WSL
4. Navigate to busy-beaver root directory and run following:

   sudo apt-get install dos2unix
   sudo dos2unix scripts/entrypoint.sh
   sudo chmod +x scripts/entrypoint.sh
   docker-compose build --no-cache
   make up

5. Docker >>> Settings >> Shared Drives:
   • Uncheck any Shared drive, Click Apply
   • Check Shared Drives, Click Apply
6. Docker >>> Settings >> Reset
   • Reset to factory defaults...
7. Uninstall Windows, Install Linux

Pushing Commits to Pull Request made from Fork

GitHub allows upstream maintainers, with permission, to push to downstream forks.

We are using Git-LFS to keep binary files out of our Git tree. Unfortunately LFS makes it a bit tricky to push to downstream forks out of the box, see issue.

To push to downstream forks, you will need to rm .git/hooks/pre-push. It is recommended that you move the file somewhere else before making the change. Replace the file once you are done.

Additional Resources

• Pushing to someone else's pull request on GitHub
• Git-LFS Issue -- Authentication required : You must have push access to verify locks error