I know there are plenty of tools that solve this. But I wanted something dead simple and fast to run in the terminal so I can see pending work across all my projects at once. So I built it: ghpending.

Runs on Linux and macOS. I tested on Linux (Arch + Omarchy) but haven’t run it on macOS yet. If you try it there and hit a bug, file an issue at github.com/akitaonrails/ghpending/issues.

Install

Three paths, pick whichever fits your setup.

Via Homebrew (macOS or Linux):

brew tap akitaonrails/tap && brew install ghpending

Via Cargo:

cargo install ghpending

Via mise:

mise use -g github:akitaonrails/ghpending

Upgrading

# Homebrew
brew upgrade ghpending

# Cargo
cargo install ghpending --force

# mise
mise upgrade ghpending

# From source
cd ghpending && git pull && cargo install --path .

Usage

Four commands. The normal flow is to run ghpending add once to set up which repos you want to track, then just ghpending whenever you want to see the digest.

ghpending add    # pick repos from a GitHub user or org to track
ghpending        # print the digest of open issues and PRs
ghpending list   # show tracked repos
ghpending rm     # remove repos from tracking via interactive menu

add asks for a user or organization, lists their public repos, and lets you check off the ones you want to follow. The username gets saved, so re-running add skips that prompt.

The main command (ghpending with no args) fetches every tracked repo concurrently and prints the table with open issues and PRs.

GitHub token (optional)

It works without auth, but GitHub caps anonymous traffic at 60 requests per hour. To raise that ceiling to 5,000 req/h, export a token:

GITHUB_TOKEN=$(gh auth token) ghpending

It’s read silently at startup, no config needed.

Where the config lives

• Linux: ~/.config/ghpending/config.toml
• macOS: ~/Library/Application Support/ghpending/config.toml

You can edit it directly if you want to reorder repos or change the user. Shape:

user = "akitaonrails"
repos = ["akitaonrails/ai-memory", "akitaonrails/ai-jail"]

Why this CLI exists

The honest answer: tools I build for myself, I use. Tools written by other people always come with some friction (web login, a different architectural opinion, a feature I wanted missing or one I didn’t want included). Since the goal was to reduce the friction of checking 30 repos a week, 200 lines of Rust on a Saturday afternoon beat learning a new stack.

Now I’m hoping I can respond to contributors faster. If you find it useful, it’s MIT licensed, PRs are welcome.

My impressions stem from very specific circumstances, so this is not to be read as a general recommendation for every situation. For example, I didn’t review GitLab’s hosted options, which may be more compatible with circumstances different than mine.

In my case, the move to GitLab was part of an internal company strategy to have more control over our own data. In that strategy we also moved from using Slack to Mattermost, and from using Pivotal Tracker to my own open source alternative called Central, among other things.

Before the migration, I had dozens of active projects over GitHub and lots of archived ones at Bitbucket. I’m the kind that hates to lose data, so I keep redundant backups and try to never erase anything.

I moved almost everything over to my own GitLab server and this is almost 200 repositories, spread in 4 groups, with almost 80 active users. This accounts for more than 13,500 notes and over 5,200 Merge Requests already, and over 2,500 builds in the CI, and the internal Sidekiq controlling everything has over 51,000 jobs processed.

In the last 3 months since the migration, I am paying a Digital Ocean bill of around $140 every month for the exact infrastructure I described in the previous article, with a separated box for the GitLab Core, the CI Runner, the CI Docker Registry Mirror, the CI Cache server and disposable machines for parallel builds, controlled by the Runner.

Although the reduction in cost is welcome, the whole endeavor was not done because of it. I could easily pay double or triple just for the convenience, but the underlying strategy is more important in this case.

My company has over 60 developers, working from our own offices in 6 different cities in Brazil (no freelancers and no home-office), communicating daily over Mattermost, Code Reviewing through GitLab’s Merge Requests, having immediate automated tests feedback from GitLab CI, and project management through our own Central application.

The only piece that is currently missing is a good static code analysis tool. We heavily rely on Code Climate, which I strongly believe is one of the best out there from our own experience.

I was about to start an endeavor to build a simplified OSS version to integrate into GitLab CI’s, but Bryan Helmkamp recently announced that they would release their own “Community Edition” version of Code Climate in the next few months.

This is exactly the missing piece in my stack, and it will make my strategy move faster for next year.

Hybrid-SaaS Era

In the past 5 to 10 years there was a fast move towards 3rd party “micro-services” structure. And this is really great as it allowed many small companies or even independent developers to tap into technology that made them not only move faster, but with increasingly more quality in their delivery.

It’s now super easy to have top-notch upload and storage service through tools like Cloudinary.

Also very easy to have top-notch relational database, with replication and scalability, such as Heroku Postgres or AWS RDS.

Project management as a service with Pivotal Tracker, Trello.

Communication as a service with Slack, Hangout, etc.

Knowledge Management as a service with GitHub, Bitbucket, etc.

So now many tech companies are a mashup of several different 3rd party services.

My small beef is that our entire knowledge, experience, portfolio is spread across a dozen or more opaque services, completely out of our control and depending on each company’s (or their investors) whim.

Don’t get me wrong, I am not against using those services, on the contrary. I use several of them and I will continue to use many of them for the foreseeable future. Some endeavors would not even be possible without the efficiency of this sample of a technology-based free market.

But every now and then you reach a tipping point where it becomes important to have more control over your own data, your own identity. Not only for pure ownership but also to be able to use this data with better intent.

And I believe that at that point we should have an option that doesn’t cost a couple of limbs to replicate.

Before an option such as GitLab, we were either limited to choose between walled-gardens such as GitHub or Bitbucket or you could invest a ton of resources trying to use small open sourced components to try to build your own. With GitLab we now have the option to make a smooth transition to its hosted option, so we don’t have to deal with infrastructure maintenance or we can choose to have full control. And the users will not suffer in the process.

Which is why I believe that the best SaaS to endure the next decade will start to go into Hybrid-mode: having a commercial, usually “cheaper”, hosted option, and a DIY (do-it-yourself) OSS version.

GitLab is like that. Mattermost is like that. Now Code Climate is getting like that.

In that option, companies like GitLab have a win-win situation. Many more people and companies can contribute towards making a mature and robust platform that everybody enjoys using, and at the same time, each participant can extend it to their own particular plans. GitLab can have a sustainable business model serving the long tail of companies and developers that want a reasonably affordable hosted service while the top 20% can pursue more specific endeavors using the same technology.

That way we remove the discussion of confidentiality, the way companies treat our data, and we move forward into more productive actions such as building a tool that benefits both the external public and each other’s internal plans.

Walled gardens are here to stay. Releasing integration — while opaque — public APIs is not enough. It’s very exciting to see those new options arising to fill this gap. The OSS environment along with a sustainable business model makes sense. And I hope to continue to see more and more competitors to closed services following the Hybrid model in the near future.

Why go through the trouble to install, maintain and tweak open sourced alternatives to well-established hosted (albeit opaque) services? I believe it’s more than a discussion over convenience, maintenance and cutting costs. We should start small, less risky, but as we grow we should be able to take control back. But we usually can’t get out of walled gardens without a significant - and sometimes impossible - investment in reinventing the wheel. We always had the small open sourced components that those services are built on, but to go from those small components to a full-featured system, it’s an unrealistic path.

With this possible trend of post-SaaS, or Hybrid-SaaS as I call it, we may have just gained the missing link to go from just convenience to full control without the inherent costs and risks chasm.

Outside of the kernel development almost no one wanted to learn it and we had very worthy competitors, from Subversion, to Mercurial, to Bazaar, to Darcs, to Perforce, and so on. But those of us who dug deeper knew that Git had the edge and it was a matter of time.

Then GitHub showed up in 2008 and the rest is history. For many years it was just “cool” to be in GitHub. The Ruby community drove GitHub up into the sky. Finally it became the status quo and the one real monopoly in information repositories - not just software source code, but everything.

I always knew that we should have a “local” option, which is why I tried to contribute to Gitorious way back in 2009. Other options arose, but eventually GitLab appeared around 2011 and picked up steam in the last couple of years.

GitHub itself raised USD 350 million in funding and one of its required goals is to nail the Enterprise Edition for big corporations that don’t want their data outside their closed gardens. Although GitHub hosts every single open source project out there, they are themselves closed-source.

GitLab Inc. started differently with an open source-first approach with their Community Edition (CE) and having both a GitHub-like hosted option as well as a supported Enterprise Edition for risk-averse corporations. They already raised USD 5.62 million in funding, and they are the most promising alternative to GitHub so far.

Of course, there are other platforms such as Atlassian’s Bitbucket. But I believe Atlassian’s strategy is slower and they have a larger suite of enterprise products to sell first, such as Confluence and Jira. I don’t think they ever posed much of a competition against GitHub.

GitLab really started accelerating in 2015 as this commit graph shows:

It’s been steadily growing since 2011, but they seem to have crossed the first tipping point around late 2014, from early adopters to the early majority. This became more important as GitHub announced their pricing changes in May.

They said they haven’t committed to a deadline to enforce the change, so organizations can opt out of the new format for the time being. They are changing from “limited repositories and unlimited users” to “unlimited repositories and limited users”.

The Cost-Benefit Conundrum

For example, if you have up to 8 developers on the USD 50/month plan (20 private repositories), the change won’t affect you, as you will pay USD 25/month for 5 users and USD 9 for additional users (total of USD 52/month).

Now, if you have a big team of 100 developers currently in the Diamond Plan of USD 450/month (300 private repositories), you would have to pay USD 25/month + 95 times USD 9, which totals a staggering USD 880/month! Double the amount!

This is an extra USD 10,560 per year!

And what does GitLab afford you instead?

You can have way more users and more repositories in a USD 40/month virtual box (4GB of RAM, 60GB SSD, 4TB transfer).

And it doesn’t stop there. GitLab also has very functional GitLab Multi Runner which you can install in a separate box (actually, at least 3 boxes - more on that below).

You can easily connect this runner to the build system over GitLab so every new git push triggers the runner to run the automated test suite in a Docker image of your choosing. So it’s a fully functional, full featured Continuous Integration system nicely integrated in your GitLab project interface:

Remind you of anything? Yep, it’s a fully functional alternative to Travis-CI, Semaphore, CircleCI or any other CI you’re using with a very easy to install procedure. Let’s say you’re paying Travis-CI USD 489/month to have 10 concurrent jobs.

You can install GitLab Runner in 3 boxes of USD 10/month (1GB RAM, 1 Cores, 30GB SSD) and have way more concurrent jobs (20? 50? Auto-Scale!?) that run faster (in a simple test, one build that took 15 minutes over Travis took less than 8 minutes at Digital Ocean).

So let’s do the math for a year’s worth of service. First considering no GitHub plan change:

USD 5,400 (GitHub) + USD 5,868 (Travis) = USD 11,268 a year.

Now, the GitLab + GitLab Runner + Digital Ocean for the same features and unlimited users, unlimited repositories, unlimited concurrent builds:

USD 480 (GitLab) + USD 840 (Runner box) = USD 1,320 a year.

This is already almost 8.5x cheaper with almost no change in quality.

For the worst case scenario, compare it when GitHub decides to enforce the new plans:

USD 10,560 (GitHub new plans) + USD 5,868 (Travis) = USD 16,428

Now the GitLab option is 11x cheaper! You’re saving almost USD 15,000 a year! This is not something you can ignore in your cost sheet.

As I said, the calculations above are only significant in a scenario with 100 developers. You must do your own math taking into account your team size and number of active projects (you can always archive unused projects).

Even if you don’t have 100 developers. Let’s consider the scenario for 30 developers in the new GitHub per user plans and a smaller Travis configuration for 5 concurrent jobs:

USD 3,000 (GitHub new plan) + USD 3,000 (Travis) = USD 6,000

It’s 4.5x cheaper in the Digital Ocean + GitLab suite option.

Heck, let’s consider the Current GitHub plan (the Platinum one, for up to 125 repositories):

USD 2,400 (GitHub current plan) + USD 3,000 (Travis) = USD 5,400

Still at least 4x more expensive than a GitLab-based solution!

And how long will it take for a single developer to figure out the set up and migrate everything from GitHub over to the new GitLab installation? I will say that you can reserve 1 week of work for the average programmer to do it following the official documentation and my tips and tricks below.

Installing GitLab CE

I will not bore you with what you can readily find over the Web. I highly recommend you start with the easiest solution first: Digital Ocean’s One-Click Automatic Install. Install it in at least a 4GB RAM machine (you will want to keep it if you like it).

Of course, there is a number of different installation options, from AWS AMI images to Ubuntu packages you can install manually. Study the documentation.

It will cost you USD 40 for a month of trial. If you want to save as much as tens of thousands of dollars, this is a bargain.

GitLab has many customization options. You can lock down your private GitLab to allow only users with an official e-mail from your domain, for example. You can configure OAuth2 providers so your users can quickly sign in using their GitHub, Facebook, Google or other accounts.

A Few Gotchas

I’ve stumbled upon a few caveats in the configuration. Which is why I recommend that you plan ahead - study this entire article ahead of time! - do a quick install that you can blow away, so you can “feel” the environment before trying to migrate all your repos over to your brand new GitLab. As a reference, this is a part of my /etc/gitlab/gitlab.rb:

— ruby

register a domain for your server and place it here:

external_url “http://my-gitlab-server.com/"

you will want to enable LFS

gitlab_rails[’lfs_enabled’] = true

register your emails

gitlab_rails[‘gitlab_email_from’] = “no-reply@my-gitlab-server.com” gitlab_rails[‘gitlab_support_email’] = “contact@my-gitlab-server.com”

add your email configuration (template for gmail)

gitlab_rails[‘smtp_enable’] = true gitlab_rails[‘smtp_address’] = “smtp.gmail.com” gitlab_rails[‘smtp_port’] = 587 gitlab_rails[‘smtp_user_name’] = “– some no-reply email —” gitlab_rails[‘smtp_password’] = “– the password —” gitlab_rails[‘smtp_domain’] = “my-gitlab-server.com” gitlab_rails[‘smtp_authentication’] = “login” gitlab_rails[‘smtp_enable_starttls_auto’] = true gitlab_rails[‘smtp_openssl_verify_mode’] = ‘peer’

this is where you enable oauth2 integration

gitlab_rails[‘omniauth_enabled’] = true

CAUTION!

This allows users to login without having a user account first. Define the allowed providers

using an array, e.g. [“saml”, “twitter”], or as true/false to allow all providers or none.

User accounts will be created automatically when authentication was successful.

gitlab_rails[‘omniauth_allow_single_sign_on’] = [‘github’, ‘google_oauth2’, ‘bitbucket’] gitlab_rails[‘omniauth_block_auto_created_users’] = true

gitlab_rails[‘omniauth_providers’] = [ { “name” => “github”, “app_id” => “– github app id –”, “app_secret” => “– github secret –”, “url” => “https://github.com/”, “args” => { “scope” => “user:email” } }, { “name” => “google_oauth2”, “app_id” => “– google app id –”, “app_secret” => “– google secret –”, “args” => { “access_type” => “offline”, “approval_prompt” => ‘’, hd => ‘codeminer42.com’ } }, { “name” => “bitbucket”, “app_id” => “– bitbucket app id –”, “app_secret” => “– bitbucket secret id –”, “url” => “https://bitbucket.org/” } ]

if you’re importing repos from GitHub, Sidekiq workers can grow as high as 2.5GB of RAM and the default Sidekiq Killer config will cap it down to 1GB, so you want to either disable it by adding ‘0’ or adding a higher limit

gitlab_rails[’env’] = { ‘SIDEKIQ_MEMORY_KILLER_MAX_RSS’ => ‘3000000’ }

There are dozens of default variables you can override, just be careful on your testings.

Every time you change a configuration, you can just run the following commands:

sudo gitlab-ctl reconfigure sudo gitlab-ctl restart

You can open a Rails console to inspect production objects like this:

gitlab-rails console

I had a lot of trouble importing big repos from GitHub, but after a few days debugging the problem with GitLab Core Team developers Douglas Alexandre, Gabriel Mazetto, a few Merge Requests and some local patching, I was finally able to import relatively big projects (more than 5,000 commits, more than 1,000 issues, more than 1,200 pull requests with several comments worth of discussion threads). A project of this size can take a couple of hours to complete, mainly because it’s damn slow to use GitHub’s public APIs (they are slow and they have rate limits and abuse detection, so you can’t fetch everything as fast as your bandwidth would allow).

(By the way, don’t miss that GitLab will be over at Rubyconf Brazil 2016, on Sep 23-24)

Migrating all my GitHub projects took a couple of days, but they all went through smoothly and my team didn’t have any trouble, just adjusting their git remote URLs and they’re done.

The import procedure from GitHub is quite complete, it brings not only the git repo per se, but also all the metadata, from labels to comments and pull request history - which is the one that usually takes more time.

But I’d recommend waiting for at least version 8.11 (it’s currently 8.10.3) before trying to import large GitHub projects.

If you’re on Bitbucket, unfortunately there are fewer features in the importer. It will mostly just bring the source code. So be aware of that if you extensively depend on their pull request system and you want to preserve this history. More features will come and you can even help them out, they are very resourceful and willing to make GitLab better.

Side-track: Customizations for every Digital Ocean box

Assume that you should run what’s in this section for all new machines you create over Digital Ocean.

First of all, they come without a swap file. No matter how much RAM you have, the Linux OS is meant to work better by combining a swap file. You can read more about it later, for now just run the following as root:

fallocate -l 4G /swapfile chmod 600 /swapfile mkswap /swapfile swapon /swapfile

sysctl vm.swappiness=10 sysctl vm.vfs_cache_pressure=50

Edit the /etc/fstab file and add this line:

/swapfile none swap sw 0 0

Finally, edit the /etc/sysctl.conf file and add these lines:

vm.swappiness=10 vm.vfs_cache_pressure = 50

Don’t forget to set the default locale of your machine. Start by editing the /etc/environment file and adding:

LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8

Then run:

sudo locale-gen en_US en_US.UTF-8 sudo dpkg-reconfigure locales

Finally, you should have Ubuntu automatically install stable security patches for you. You don’t want to forget machines online without the most current security fixes, so just run this:

sudo dpkg-reconfigure –priority=low unattended-upgrades

Choose “yes” and you’re done. And of course, for every fresh install, it’s always good to run the good old:

sudo apt-get update && sudo apt-get upgrade

This is the very basics, I believe it’s easier to have an image with all this ready, but if you use the standard Digital Ocean images, these settings should do the trick for now.

Installing the CI Runner

Once you finish your GitLab installation, it’s super easy to deploy the GitLab Runner. You can use the same machine but I recommend you install it in a separate machine.

If you don’t know what a runner is, just imagine it like this: It’s basically a server connected to the GitLab install. When it’s available and online, whenever someone pushes a new commit, merge request, to a repository that has a gitlab-ci-yml file present, GitLab will push a command to the runner.

Depending on how you configured the runner, it will receive this command and spawn a new Docker container. Inside the container it will execute whatever you have defined in the gitlab-ci.yml file in the project. Usually it’s fetching cached files (dependencies, for example), and run your test suite.

In the most basic setup, you will only have one Runner and any subsequent builds from other users will wait in line until they finish. If you’ve used external CI services such as Travis-CI or CircleCI, you know that they charge for some number of concurrent builds. And it’s very expensive.

The less concurrent builds available, the more your users will have to wait for feedback on their changes, and less productive you will become. People may even start to avoid adding new tests, or completely ignore the tests, which will really hurt the quality of your project over time. If there is one thing you must not do is not having good automated test suites.

Gabriel Mazetto pointed me to a very important GitLab CI Runner feature: auto-scaling. This is what they use in their hosted offering over at GitLab.com.

You can easily set up a runner that can use “docker-machine” and your IaaS provider APIs to spin up machines on the fly to run as many concurrent builds as you want, and it will be super cheap!

For example, on Digital Ocean you can be charged USD 0.06 (6 cents) per hour of usage of a 4GB machine. Over at AWS EC2 you can be charged USD 0.041 per hour for an m3.medium machine.

There is extensive documentation but I will try to summarize what you have to do. For more details I highly recommend you to study their official documentation.

Start by creating 3 new machines at Digital Ocean, all in the same Region with private networking enabled! I will list a fake private IP address just for the sake of advancing in the configuration examples:

• a 1GB machine called “docker-registry-mirror”, (ex 10.0.0.1)
• a 1GB machine called “ci-cache”, (ex 10.0.0.2)
• a 1GB machine called “ci-runner”, (ex 10.0.0.3)

Yeah, they can be small as very little will run on them. You can be conservative and choose the 2GB RAM options just to be on the safe side (and pricing will still be super cheap).

Don’t forget to execute the basic configuration I mentioned above to enable a swapfile, auto security update and locale regeneration.

SSH in to “docker-registry-mirror” and just run:

docker run -d -p 6000:5000
-e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io
–restart always
–name registry registry:2

Now you will have a local Docker images registry proxy and cache at 10.0.0.1:6000 (take note of the real private IP).

SSH in to “ci-cache” and run:

mkdir -p /export/runner

docker run -it –restart always -p 9005:9000
-v /.minio:/root/.minio -v /export:/export
–name minio
minio/minio:latest /export

Now you will have an AWS S3 clone called Minio running. I didn’t know this project even existed, but it is a nifty little service written in Go to clone the AWS S3 behavior and APIs. So now you can have your very own S3 inside your infrastructure!

After Docker spins up, it will print out the Access Key and Secret keys, make notes. And this service will be running at 10.0.0.2:9005.

You can even open a browser and see their web interface at http://10.0.0.2:9005 and use the access and secret keys to login. Make sure you have a bucket named “runner”. The files will be stored at the /export/runner directory.

Make sure the bucket name is valid (it must be a valid DNS naming, for example, DO NOT use underlines).

Open this URL from your freshly installed GitLab-CE: http://yourgitlab.com/admin/runners and take note of the Registration Token. Let’s say it’s 1aaaa_Z1AbB2CdefGhij

Finally, SSH in to “ci-runner” and run:

curl -L https://github.com/docker/machine/releases/download/v0.7.0/docker-machine-`uname -s-uname -m` > /usr/local/bin/docker-machine

chmod +x /usr/local/bin/docker-machine

curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.deb.sh | sudo bash

sudo apt-get install gitlab-ci-multi-runner

rm -Rf ~/.docker # just to make sure

Now you can register this new runner with your GitLab install, you will need the Registration Token mentioned above.

sudo gitlab-ci-multi-runner register

You will be asked a few questions, and this is what you can answer:

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/ci ) https://yourgitlab.com/ci Please enter the gitlab-ci token for this runner 1aaaa_Z1AbB2CdefGhij # as in the example above Please enter the gitlab-ci description for this runner my-autoscale-runner INFO[0034] fcf5c619 Registering runner… succeeded Please enter the executor: shell, docker, docker-ssh, docker+machine, docker-ssh+machine, ssh? docker+machine Please enter the Docker image (eg. ruby:2.1): codeminer42/ci-ruby:2.3 INFO[0037] Runner registered successfully. Feel free to start it, but if it’s running already the config should be automatically reloaded!

Let’s make a copy of the original configuration, just to be safe:

cp /etc/gitlab-runner/config.toml /etc/gitlab-runner/config.bak

Copy the first few lines of this file (you want the token), it will look like this:

concurrent = 1 check_interval = 0

[[runners]] name = “my-autoscale-runner” url = “http://yourgitlab.com/ci" token = “— generated runner token —” executor = “docker+machine”

The important part here is the “token”. You will want to take note of it. And now you also will want to create a new API Token over at Digital Ocean. Just Generate a New Token and take note.

You can now replace the entire config.toml file for this:

concurrent = 20 check_interval = 0

[[runners]] name = “my-autoscale-runner” url = “http://yourgitlab.com/ci" token = “— generated runner token —” executor = “docker+machine” limit = 15 [runners.docker] tls_verify = false image = “codeminer42/ci-ruby:2.3” privileged = false [runners.machine] IdleCount = 2 # There must be 2 machines in Idle state IdleTime = 1800 # Each machine can be in Idle state up to 30 minutes (after this it will be removed) MaxBuilds = 100 # Each machine can handle up to 100 builds in a row (after this it will be removed) MachineName = “ci-auto-scale-%s” # Each machine will have a unique name (’%s’ is required) MachineDriver = “digitalocean” # Docker Machine is using the ‘digitalocean’ driver MachineOptions = [ “digitalocean-image=coreos-beta”, “digitalocean-ssh-user=core”, “digitalocean-access-token=– your new Digital Ocean API Token –”, “digitalocean-region=nyc1”, “digitalocean-size=4gb”, “digitalocean-private-networking”, “engine-registry-mirror=http://10.0.0.1:6000” ] [runners.cache] Type = “s3” # The Runner is using a distributed cache with Amazon S3 service ServerAddress = “10.0.0.2:9005” # minio AccessKey = “– your minio access key –” SecretKey = “– your minio secret key” BucketName = “runner” Insecure = true # Use Insecure only when using with Minio, without the TLS certificate enabled

And you can restart the runner to pick up the new configuration like this:

gitlab-ci-multi-runner restart

As I said before, you will want to read the extensive official documentation (and every link within).

If you did everything right, changing the correct private IPs for the docker registry and cache, the correct tokens, and so forth, you can log in to your Digital Ocean dashboard and you will see something like this:

And from the ci-runner machine, you can list them like this:

docker-machine ls

NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS runner-xxxx-ci-auto-scale-xxxx-xxxx - digitalocean Running tcp://191.168.0.1:2376 v1.10.3 runner-xxxx-ci-auto-scale-xxxx-xxxx - digitalocean Running tcp://192.169.0.2:2376 v1.10.3

They should not list any errors, meaning that they are up and running, waiting for new builds to start.

There will be 2 new machines listed in your Digital Ocean dashboard, named “runner-xxxxx-ci-auto-scale-xxxxx”. This is what IdleCount = 2 does. If they stay idle for more than 30 minutes (IdleTime = 1800 ) they will be shut down so you don’t get charged.

You can have several “runner” definitions, each with a limit of builds/machines that can be spawned in Digital Ocean. You can have other runner definitions for other providers, for example. But in this example we are limited to at most 15 machines, so 15 concurrent builds.

The concurrent limit is a global setting. So if I had 3 runner definitions, each with a limit of 15, they would still be globally limited to 20 as defined in the concurrent global variable.

You can use different providers for specific needs, for example, to run OS X builds or Raspberry Pi builds or other exotic kinds of builds. In the example I am keeping it simple and just setting many builds in the same provider (Digital Ocean).

And don’t worry about the monthly fee for each machine. When used in this manner, you will be paying per hour.

Also, make sure you spun up all your machines (docker-registry, minio cache, CI runner) all with private networking enabled (so they talk through the internal VLAN instead of having to go all the way through the public internet) and that they are all in the same region data center (NYC1 is New York 1 - New York has 3 sub-regions, for example). Don’t start machines in different regions.

Because we have Docker proxy/cache and Minio/S3 cache, your builds will take longer the first time (let’s say, 5 minutes), and then subsequent builds will fetch everything from the cache (taking, let’s say, 1:30 minute). It’s fast and it’s convenient.

Setting up each Project for the Runner

The Runner is one of the newest pieces of the GitLab ecosystem so you might have some trouble at first to figure out a decent configuration. But once you have the whole infrastructure figured out as described in the previous section, now it’s as easy as adding a .gitlab-ci.yml file to your root directory. Something like this:

— yaml

This file is a template, and might need editing before it works on your project.

image: codeminer42/ci-ruby:2.3

Pick zero or more services to be used on all builds.

Only needed when using a docker container to run your tests in.

Check out: http://docs.gitlab.com/ce/ci/docker/using_docker_images.html#what-is-service

services:

• postgres:latest
• redis:latest

cache: key: your-project-name untracked: true paths: - .ci_cache/

variables: RAILS_ENV: ’test’ DATABASE_URL: postgresql://postgres:@postgres CODECLIMATE_REPO_TOKEN: – your codeclimate project token –

before_script:

• bundle install –without development production -j $(nproc) –path .ci_cache
• cp .env.sample .env
• cp config/database.yml.example config/database.yml
• bundle exec rake db:create db:migrate

test: script: - xvfb-run bundle exec rspec

My team at Codeminer 42 prepared a simple Docker image with useful stuff pre-installed (such as the newest phantomjs, xvfb, etc), so it’s now super easy to enable automated builds within GitLab by just adding this file to the repositories. (Thanks to Carlos Lopes, Danilo Resende and Paulo Diovanni - who will be talking about Docker at Rubyconf Brasil 2016, by the way)

GitLab-CI even supports building a pending Merge Request, and you can enforce the request so it can only be merged if builds pass, just like in GitHub + Travis. And as Code Climate is agnostic to Repository host or CI runner, you can easily integrate it as well.

Conclusion

The math is hard to argue against: the GitLab + GitLab-CI + Digital Ocean combo is a big win. GitLab’s interface is very familiar so users from GitHub or Bitbucket will feel quite at home in no time.

We can use all the Git flows we’re used to.

GitLab-CE is still a work in progress though, the team is increasing their pace but there are currently more than 4,200 open issues. But as this is all Ruby on Rails and Ruby tooling, you can easily jump in and contribute. No contribution is too small. Just reporting how to reproduce a bug is help enough to assist the developers to figure out how to improve faster.

But don’t shy away because of the open issues, it’s fully functional as of right now and I have not found any bugs that could be considered show stoppers.

They have many things right. First of all, it’s a “simple” Ruby on Rails project. It’s a no-frills front-end with plain jQuery. The choice of HAML for the views is questionable but it doesn’t hurt. They use good old Sidekiq+Redis for asynchronous jobs. No black magic here. A pure monolith that’s not difficult to understand and to contribute.

The APIs are all written using Grape. They have the GitLab CE project separated from other components, such as the GitLab-Shell and GitLab-CI-Multi-Runner.

They also forked Omnibus in order to be able to package the CE Rails project as a “.deb”. Everything is orchestrated with Docker. And when a new version is available, you only need to apt-get update && apt-get upgrade and it will do all the work of backing up and migrating Postgresql, updating the code, bundling in new dependencies, restarting the services and so forth. It’s super convenient and you should take a look at this project if you have complicated Rails deployments into your own infrastructure (out of Heroku, for example).

I am almost done moving hundreds of repositories from both BitBucket and GitHub to GitLab right now and the developers from my company are already using it on a daily basis without any problems. We are almost at the point where we can disengage from BitBucket, GitHub and external CIs.

You will be surprised how easily your company can do it too and save a couple thousand dollars in the process, while having fun doing it!