This job is stuck, because the project doesn't have any runners online assigned to it. go to runners page

Shared vs specific Runners

After installing the Runner, you can either register it as shared or
specific. You can only register a shared Runner if you have admin access to
the GitLab instance. The main differences between a shared and a specific Runner
are:

Shared Runners are useful for jobs that have similar requirements,
between multiple projects. Rather than having multiple Runners idling for
many projects, you can have a single or a small number of Runners that handle
multiple projects. This makes it easier to maintain and update them.
Shared Runners process jobs using a .
In contrast to specific Runners that use a FIFO queue, this prevents
cases where projects create hundreds of jobs which can lead to eating all
available shared Runners resources.
Specific Runners are useful for jobs that have special requirements or for
projects with a specific demand. If a job has certain requirements, you can set
up the specific Runner with this in mind, while not having to do this for all
Runners. For example, if you want to deploy a certain project, you can setup
a specific Runner to have the right credentials for this. The
may be useful in this case. Specific Runners process jobs using a FIFO queue.

A Runner that is specific only runs for the specified project(s). A shared Runner
can run jobs for every project that has enabled the option Allow shared Runners
under Settings ➔ CI/CD.

Projects with high demand of CI activity can also benefit from using specific
Runners. By having dedicated Runners you are guaranteed that the Runner is not
being held up by another project’s jobs.

You can set up a specific Runner to be used by multiple projects. The difference
with a shared Runner is that you have to enable each project explicitly for
the Runner to be able to run its jobs.

Specific Runners do not get shared with forked projects automatically.
A fork does copy the CI settings (jobs, allow shared, etc) of the cloned
repository.

Determine the IP address of a runner

It may be useful to know the IP address of a runner so you can troubleshoot
issues with that runner. GitLab stores and displays the IP address by viewing
the source of the HTTP requests it makes to GitLab when polling for jobs. The
IP address is always kept up to date so if the runner IP changes it will be
automatically updated in GitLab.

The IP address for shared runners and specific runners can be found in
different places.

Determine the IP address of a shared runner

To view the IP address of a shared runner you must have admin access to
the GitLab instance. To determine this:

Visit Admin Area > Overview > Runners.
Look for the runner in the table and you should see a column for IP Address.

Determine the IP address of a specific runner

To can find the IP address of a runner for a specific project,
you must have Owner for the project.

Go to the project’s Settings > CI/CD and expand the Runners section.
On the details page you should see a row for IP Address.

Using shared Runners effectively

If you are planning to use shared Runners, there are several things you
should keep in mind.

Using tags

You must setup a Runner to be able to run all the different types of jobs
that it may encounter on the projects it’s shared over. This would be
problematic for large amounts of projects, if it wasn’t for tags.

By tagging a Runner for the types of jobs it can handle, you can make sure
shared Runners will only run the jobs they are equipped to run.

For instance, at GitLab we have Runners tagged with «rails» if they contain
the appropriate dependencies to run Rails test suites.

Preventing Runners with tags from picking jobs without tags

You can configure a Runner to prevent it from picking jobs with tags when
the Runner does not have tags assigned. This setting can be enabled the first
time you register a Runner and can be changed afterwards under
each Runner’s settings.

To make a Runner pick tagged/untagged jobs:

Visit your project’s Settings ➔ Pipelines
Find the Runner you wish and make sure it’s enabled
Click the pencil button
Check the Run untagged jobs option
Click Save changes for the changes to take effect

Be careful with sensitive information

If you can run a job on a Runner, you can get access to any code it runs
and get the token of the Runner. With shared Runners, this means that anyone
that runs jobs on the Runner, can access anyone else’s code that runs on the
Runner.

In addition, because you can get access to the Runner token, it is possible
to create a clone of a Runner and submit false jobs, for example.

The above is easily avoided by restricting the usage of shared Runners
on large public GitLab instances and controlling access to your GitLab instance.

Forks

Whenever a project is forked, it copies the settings of the jobs that relate
to it. This means that if you have shared Runners setup for a project and
someone forks that project, the shared Runners will also serve jobs of this
project.

Use predefined CI/CD variables to run jobs only in specific pipeline types

You can use predefined CI/CD variables to choose
which pipeline types jobs run in, with:

The following table lists some of the variables that you can use, and the pipeline
types the variables can control for:

Branch pipelines that run for Git events to a branch, like new commits or tags.
Tag pipelines that run only when a new Git tag is pushed to a branch.
Merge request pipelines that run for changes
to a merge request, like new commits or selecting the Run pipeline button
in a merge request’s pipelines tab.
Scheduled pipelines.

Branch	Tag	Merge request	Scheduled
Yes			Yes
	Yes		Yes, if the scheduled pipeline is configured to run on a tag.
Yes	Yes
			Yes
		Yes
		Yes

For example, to configure a job to run for merge request pipelines and scheduled pipelines,
but not branch or tag pipelines:

Specify when jobs run with only and except

You can use and
to control when to add jobs to pipelines.

Use to define when a job runs.
Use to define when a job does not run.

/ examples

or used without is the same as

In the following example, runs only for:

Git tags
Scheduled pipelines

To execute jobs only for the parent repository and not forks:

This example runs for all branches on ,
except and branches that start with .

/ examples

You can use to exclude jobs based on a commit message:

You can use with and
to build more complicated variable expressions:

When multiple entries are specified in , the job runs when at least one of them evaluates to .
You can use in a single entry when multiple conditions must be satisfied at the same time.

/ examples

You can skip a job if a change is detected in any file with a
extension in the root directory of the repository:

If you change multiple files, but only one file ends in ,
the job is still skipped. The job does not run for any of the files.

Use with pipelines for merge requests

With pipelines for merge requests,
it’s possible to define a job to be created based on files modified
in a merge request.

Use this keyword with so GitLab can find the correct base
SHA of the source branch. File differences are correctly calculated from any further
commits, and all changes in the merge requests are properly tested in pipelines.

For example:

In this scenario, if a merge request changes
files in the directory or the , GitLab creates
the job.

For example:

In this example, the pipeline might fail because of changes to a file in .

A later commit that doesn’t have changes in
but does have changes to the can pass. The job
only tests the changes to the .

GitLab checks the most recent pipeline that passed. If the merge request is mergeable,
it doesn’t matter that an earlier pipeline failed because of a change that has not been corrected.

When you use this configuration, ensure that the most recent pipeline
properly corrects any failures from previous pipelines.

Use without pipelines for merge requests

Without pipelines for merge requests, pipelines
run on branches or tags that don’t have an explicit association with a merge request.
In this case, a previous SHA is used to calculate the diff, which is equivalent to .
This can result in some unexpected behavior, including:

When pushing a new branch or a new tag to GitLab, the policy always evaluates to true.
When pushing a new commit, the changed files are calculated by using the previous commit
as the base SHA.

Use with scheduled pipelines

always evaluates as true in Scheduled pipelines.
All files are considered to have changed when a scheduled pipeline runs.

Combine multiple keywords with or

If you use multiple keywords with or , the keywords are evaluated
as a single conjoined expression. That is:

includes the job if all of the keys have at least one condition that matches.
excludes the job if any of the keys have at least one condition that matches.

With , individual keys are logically joined by an . A job is added to
the pipeline if the following is true:

(any listed refs are true) AND (any listed variables are true) AND (any listed changes are true) AND (any chosen Kubernetes status matches)

In the following example, the job is only created when all of the following are true:

The pipeline is scheduled or runs for .
The keyword matches.
The service is active on the project.

With , individual keys are logically joined by an . A job is not
added if the following is true:

(any listed refs are true) OR (any listed variables are true) OR (any listed changes are true) OR (a chosen Kubernetes status matches)

In the following example, the job is not created when any of the following are true:

The pipeline runs for the branch.
There are changes to the file in the root directory of the repository.

Be careful with sensitive information

With some runner executors,
if you can run a job on the runner, you can get full access to the file system,
and thus any code it runs as well as the token of the runner. With shared runners, this means that anyone
that runs jobs on the runner, can access anyone else’s code that runs on the
runner.

In addition, because you can get access to the runner token, it is possible
to create a clone of a runner and submit false jobs, for example.

The above is easily avoided by restricting the usage of shared runners
on large public GitLab instances, controlling access to your GitLab instance,
and using more secure runner executors.

Prevent runners from revealing sensitive information

You can protect runners so they don’t reveal sensitive information.
When a runner is protected, the runner picks jobs created on
protected branches or protected tags only,
and ignores other jobs.

To protect or unprotect a runner:

Go to the project’s Settings > CI/CD and expand the Runners section.
Find the runner you want to protect or unprotect. Make sure it’s enabled.
Click the pencil button.
Check the Protected option.
Click Save changes.

Forks

Whenever a project is forked, it copies the settings of the jobs that relate
to it. This means that if you have shared runners set up for a project and
someone forks that project, the shared runners serve jobs of this project.

Attack vectors in runners

Mentioned briefly earlier, but the following things of runners can be exploited.
We’re always looking for contributions that can mitigate these
Security Considerations.

Reset the runner registration token for a project

If you think that a registration token for a project was revealed, you should
reset it. A token can be used to register another runner for the project. That new runner
may then be used to obtain the values of secret variables or to clone project code.

To reset the token:

Go to the project’s Settings > CI/CD.
Expand the General pipelines settings section.
Find the Runner token form field and click the Reveal value button.
Delete the value and save the form.
After the page is refreshed, expand the Runners settings section
and check the registration token — it should be changed.

From now on the old token is no longer valid and does not register
any new runners to the project. If you are using any tools to provision and
register new runners, the tokens used in those tools should be updated to reflect the
value of the new token.

Protect production secrets

Production secrets are needed to deploy successfully. For example, when deploying to the cloud,
cloud providers require these secrets to connect to their services. In the project settings, you can
define and protect CI/CD variables for these secrets.
are only passed to pipelines running on protected branches
or protected tags.
The other pipelines don’t get the protected variable. You can also
.
We recommend that you use protected variables on protected environments to make sure that the
secrets aren’t exposed unintentionally. You can also define production secrets on the
.
This prevents other maintainers from reading the secrets and makes sure that the runner only runs on
protected branches.

For more information, see .

How shared Runners pick jobs

Shared Runners abide to a process queue we call fair usage. The fair usage
algorithm tries to assign jobs to shared Runners from projects that have the
lowest number of jobs currently running on shared Runners.

Example 1

We have following jobs in queue:

Job 1 for Project 1
Job 2 for Project 1
Job 3 for Project 1
Job 4 for Project 2
Job 5 for Project 2
Job 6 for Project 3

With the fair usage algorithm jobs are assigned in following order:

Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (i.e. all projects)
Job 4 is next, because 4 is now the lowest job number from projects with no running jobs (Project 1 has a job running)
Job 6 is next, because 6 is now the lowest job number from projects with no running jobs (Projects 1 and 2 have jobs running)
Job 2 is next, because, of projects with the lowest number of jobs running (each has 1), it is the lowest job number
Job 5 is next, because Project 1 now has 2 jobs running, and between Projects 2 and 3, Job 5 is the lowest remaining job number
Lastly we choose Job 3… because it’s the only job left

Example 2

We have following jobs in queue:

Job 1 for project 1
Job 2 for project 1
Job 3 for project 1
Job 4 for project 2
Job 5 for project 2
Job 6 for project 3

With the fair usage algorithm jobs are assigned in following order:

Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (i.e. all projects)
We finish job 1
Job 2 is next, because, having finished Job 1, all projects have 0 jobs running again, and 2 is the lowest available job number
Job 4 is next, because with Project 1 running a job, 4 is the lowest number from projects running no jobs (Projects 2 and 3)
We finish job 4
Job 5 is next, because having finished Job 4, Project 2 has no jobs running again
Job 6 is next, because Project 3 is the only project left with no running jobs
Lastly we choose Job 3… because, again, it’s the only job left (who says 1 is the loneliest number?)

Problem

The user Job to be Done (JTBD), or problem to solve, is that their CI/CD pipelines execute reliably and in as short a time as possible. Today, a significant amount of pipeline execution time is due to the uploading and downloading of intermediate build elements between jobs in a pipeline. This is especially true for pipelines with multiple jobs that create intermediate build elements GB’s in size.

The outcome is to have a solution that allows for source code, dependencies, and compiled binaries to be persisted and shared between pipeline stages without the overhead of archiving, uploads, and downloads of files.

Use tags to limit the number of jobs using the runner

You must set up a runner to be able to run all the different types of jobs
that it may encounter on the projects it’s shared over. This would be
problematic for large amounts of projects, if it weren’t for tags.

By tagging a runner for the types of jobs it can handle, you can make sure
shared runners will .

For instance, at GitLab we have runners tagged with if they contain
the appropriate dependencies to run Rails test suites.

When you register a runner, its default behavior is to only pick
.
To change this, you must have Owner for the project.

To make a runner pick untagged jobs:

Go to the project’s Settings > CI/CD and expand the Runners section.
Find the runner you want to pick untagged jobs and make sure it’s enabled.
Click the pencil button.
Check the Run untagged jobs option.
Click the Save changes button for the changes to take effect.

NOTE: Note:
The runner tags list can not be empty when it’s not allowed to pick untagged jobs.

Below are some example scenarios of different variations.

runner runs only tagged jobs

The following examples illustrate the potential impact of the runner being set
to run only tagged jobs.

Example 1:

The runner is configured to run only tagged jobs and has the tag.
A job that has a tag is executed and stuck.

Example 2:

The runner is configured to run only tagged jobs and has the tag.
A job that has a tag is executed and run.

Example 3:

The runner is configured to run only tagged jobs and has the tag.
A job that has no tags defined is executed and stuck.

runner is allowed to run untagged jobs

The following examples illustrate the potential impact of the runner being set
to run tagged and untagged jobs.

Example 1:

The runner is configured to run untagged jobs and has the tag.
A job that has no tags defined is executed and run.
A second job that has a tag defined is executed and run.

Example 2:

The runner is configured to run untagged jobs and has no tags defined.
A job that has no tags defined is executed and run.
A second job that has a tag defined is stuck.

Using shared Runners effectively

If you are planning to use shared Runners, there are several things you
should keep in mind.

Using tags

By tagging a Runner for the types of jobs it can handle, you can make sure
shared Runners will only run the jobs they are equipped to run.

For instance, at GitLab we have Runners tagged with «rails» if they contain
the appropriate dependencies to run Rails test suites.

Preventing Runners with tags from picking jobs without tags

To make a Runner pick tagged/untagged jobs:

Visit your project’s Settings ➔ CI/CD
Find the Runner you wish and make sure it’s enabled
Click the pencil button
Check the Run untagged jobs option
Click Save changes for the changes to take effect

Be careful with sensitive information

With some Runner Executors,
if you can run a job on the Runner, you can get access to any code it runs
and get the token of the Runner. With shared Runners, this means that anyone
that runs jobs on the Runner, can access anyone else’s code that runs on the
Runner.

In addition, because you can get access to the Runner token, it is possible
to create a clone of a Runner and submit false jobs, for example.

The above is easily avoided by restricting the usage of shared Runners
on large public GitLab instances, controlling access to your GitLab instance,
and using more secure Runner Executors.

Forks

Regular expressions

The symbol denotes the beginning of a ref’s repository path.
To match a ref name that contains the character in a regular expression,
you must use the hex character code match .

Only the tag or branch name can be matched by a regular expression.
The repository path, if given, is always matched literally.

To match the tag or branch name,
the entire ref name part of the pattern must be a regular expression surrounded by .
For example, you can’t use to match all tag names or branch names
that begin with , but you can use .

Regular expression flags must be appended after the closing . Pattern matching
is case-sensitive by default. Use the flag modifier, like , to make
a pattern case-insensitive:

Use anchors and to avoid the regular expression
matching only a substring of the tag name or branch name.
For example, is equivalent to ,
while just would also match a branch called .

/ regex syntax

In GitLab 11.9.4, GitLab began internally converting the regexp used
in and keywords to RE2.

RE2 limits the set of available features
due to computational complexity, and some features, like negative lookaheads, became unavailable.
Only a subset of features provided by Ruby Regexp
are now supported.

From GitLab 11.9.7 to GitLab 12.0, GitLab provided a feature flag to
let you use unsafe regexp syntax. After migrating to safe syntax, you should disable
this feature flag again:

Shared vs specific Runners

Shared Runners are useful for jobs that have similar requirements,
between multiple projects. Rather than having multiple Runners idling for
many projects, you can have a single or a small number of Runners that handle
multiple projects. This makes it easier to maintain and update them.
Shared Runners process jobs using a .
In contrast to specific Runners that use a FIFO queue, this prevents
cases where projects create hundreds of jobs which can lead to eating all
available shared Runners resources.
Specific Runners are useful for jobs that have special requirements or for
projects with a specific demand. If a job has certain requirements, you can set
up the specific Runner with this in mind, while not having to do this for all
Runners. For example, if you want to deploy a certain project, you can setup
a specific Runner to have the right credentials for this. The
may be useful in this case. Specific Runners process jobs using a FIFO queue.

A Runner that is specific only runs for the specified project(s). A shared Runner
can run jobs for every project that has enabled the option Allow shared Runners
under Settings ➔ Pipelines.

Projects with high demand of CI activity can also benefit from using specific
Runners. By having dedicated Runners you are guaranteed that the Runner is not
being held up by another project’s jobs.

You can set up a specific Runner to be used by multiple projects. The difference
with a shared Runner is that you have to enable each project explicitly for
the Runner to be able to run its jobs.

Specific Runners do not get shared with forked projects automatically.
A fork does copy the CI settings (jobs, allow shared, etc) of the cloned
repository.

Registering a specific Runner

Registering a specific can be done in two ways:

Creating a Runner with the project registration token
Converting a shared Runner into a specific Runner (one-way, admin only)

Registering a specific Runner with a project registration token

To create a specific Runner without having admin rights to the GitLab instance,
visit the project you want to make the Runner work for in GitLab:

Go to Settings ➔ CI/CD to obtain the token
Register the Runner

Making an existing shared Runner specific

If you are an admin on your GitLab instance, you can turn any shared Runner into
a specific one, but not the other way around. Keep in mind that this is a one
way transition.

Go to the Runners in the admin area Overview ➔ Runners ()
and find your Runner
Enable any projects under Restrict projects for this Runner to be used
with the Runner

From now on, the shared Runner will be specific to those projects.

Registering a specific Runner

Registering a specific can be done in two ways:

Creating a Runner with the project registration token
Converting a shared Runner into a specific Runner (one-way, admin only)

Registering a specific Runner with a project registration token

To create a specific Runner without having admin rights to the GitLab instance,
visit the project you want to make the Runner work for in GitLab:

Go to Settings ➔ Pipelines to obtain the token
Register the Runner

Making an existing shared Runner specific

If you are an admin on your GitLab instance, you can turn any shared Runner into
a specific one, but not the other way around. Keep in mind that this is a one
way transition.

Go to the Runners in the admin area Overview ➔ Runners ()
and find your Runner
Enable any projects under Restrict projects for this Runner to be used
with the Runner

From now on, the shared Runner will be specific to those projects.

CI/CD variable expressions

Version history

Introduced in GitLab 10.7 for
Expanded in GitLab 12.3 with

Use variable expressions to control which jobs are created in a pipeline after changes
are pushed to GitLab. You can use variable expressions with:

For example, with :

Compare a variable to a string

You can use the equality operators and to compare a variable with a
string. Both single quotes and double quotes are valid. The order doesn’t matter,
so the variable can be first, or the string can be first. For example:

Check if a variable exists

You can check for the existence of a variable by using just the variable name in
the expression. The variable must not be empty. For example:

if: $VARIABLE

Compare a variable to a regex pattern

You can do regex pattern matching on variable values with the and operators.
Variable pattern matching with regular expressions uses the
RE2 regular expression syntax.

Expressions evaluate as if:

Matches are found when using .
Matches are not found when using .

For example:

Pattern matching is case-sensitive by default. Use the flag modifier to make a
pattern case-insensitive. For example: .

Join variable expressions together with or

Introduced in GitLab 12.0

You can join multiple expressions using (and) or (or), for example:

The precedence of operators follows the Ruby 2.5 standard,
so is evaluated before .

Group variable expressions together with parentheses

Version history

Introduced in GitLab 13.3.
Feature flag removed in GitLab 13.5.

You can use parentheses to group expressions together. Parentheses take precedence over
and , so expressions enclosed in parentheses are evaluated first, and the
result is used for the rest of the expression.

You can nest parentheses to create complex conditions, and the inner-most expressions
in parentheses are evaluated first.

For example:

User Requirements (summarized)

It is essential to have runners ‘sticking’ to all job stages and persistent workspace on a particular runner machine.
I want folders like vendor/ updated, installed, built once and then re-used in following pipeline stages without having a lot of these folders stored as artifacts.
For my Android project I need the .class files that are created in the first pipeline stage to be passed to subsequent stages of the same pipeline in a guaranteed manner and without needing to upload and download the files from GitLab.
I need an efficient way to use the same Runner for subsequent jobs of the same pipeline execution.
I need a feature that allows me to keep temporary build data between stages. This needs to be reliable and not interfere with current GitLab Runner features.
I need a solution that is able to solve for anything with a compilation step, (example Java, Javascript, Android) where having some sort of temporary build data is vital.
I want separate stages and separate containers per stage, but the ability to say that these are all tightly integrated steps of a single build path and should run on a single Runner and share files directly.
I want a feature that I can use for my NuGet build pipeline so I am not constantly required to do a .net build for each job in the pipeline.
I need an option that allow me to force execution of the entire pipeline on the same Runner the first job in the pipeline started on.
I need a way of making files available between jobs/stages without having to store them as artifacts in the GitLab instance.
I need to be able to have a simple pipeline with several stages run on a single runner without wasting time and bandwidth copying files between stages/jobs.

Reference: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/336

Set maximum job timeout for a runner

For each runner, you can specify a maximum job timeout. This timeout,
if smaller than the , takes precedence.

This feature can be used to prevent your shared runner from being overwhelmed
by a project that has jobs with a long timeout (for example, one week).

When not configured, runners do not override the project timeout.

How this feature works:

Example 1 — Runner timeout bigger than project timeout

You set the maximum job timeout for a runner to 24 hours
You set the CI/CD Timeout for a project to 2 hours
You start a job
The job, if running longer, will be timed out after 2 hours

Example 2 — Runner timeout not configured

You remove the maximum job timeout configuration from a runner
You set the CI/CD Timeout for a project to 2 hours
You start a job
The job, if running longer, will be timed out after 2 hours

Example 3 — Runner timeout smaller than project timeout

You set the maximum job timeout for a runner to 30 minutes
You set the CI/CD Timeout for a project to 2 hours
You start a job
The job, if running longer, will be timed out after 30 minutes

Ensure only one deployment job runs at a time

Pipeline jobs in GitLab CI/CD run in parallel, so it’s possible that two deployment
jobs in two different pipelines attempt to deploy to the same environment at the same
time. This is not desired behavior as deployments should happen sequentially.

You can ensure only one deployment job runs at a time with the in your .

For example:

Example of a problematic pipeline flow before using the resource group:

job in Pipeline-A starts running.
job in Pipeline-B starts running. This is a concurrent deployment that could cause an unexpected result.
job in Pipeline-A finished.
job in Pipeline-B finished.

The improved pipeline flow after using the resource group:

job in Pipeline-A starts running.
job in Pipeline-B attempts to start, but waits for the first job to finish.
job in Pipeline-A finishes.
job in Pipeline-B starts running.

For more information, see .

Skip outdated deployment jobs

The execution order of pipeline jobs can vary from run to run, which could cause
undesired behavior. For example, a deployment job in a newer pipeline could
finish before a deployment job in an older pipeline.
This creates a race condition where the older deployment finished later,
overwriting the “newer” deployment.

You can ensure that older deployment jobs are cancelled automatically when a newer deployment
runs by enabling the feature.

Example of a problematic pipeline flow before enabling Skip outdated deployment jobs:

Pipeline-A is created on the default branch.
Later, Pipeline-B is created on the default branch (with a newer commit SHA).
The job in Pipeline-B finishes first, and deploys the newer code.
The job in Pipeline-A finished later, and deploys the older code, overwriting the newer (latest) deployment.

The improved pipeline flow after enabling Skip outdated deployment jobs:

Pipeline-A is created on the default branch.
Later, Pipeline-B is created on the default branch (with a newer SHA).
The job in Pipeline-B finishes first, and deploys the newer code.
The job in Pipeline-A is automatically cancelled, so that it doesn’t overwrite the deployment from the newer pipeline.