path: root/README.md

# GitLab License Management

[![pipeline status](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/badges/master/pipeline.svg)](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/commits/master)
[![coverage report](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/badges/master/coverage.svg)](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/commits/master)

GitLab tool for detecting licenses of the dependencies used by the provided source.
It is currently based on [License Finder][license_finder]
only, but this may change in the future.

## How to use

1. `cd` into the directory of the source code you want to scan
1. Run the Docker image:

    ```sh
    docker run \
      --volume "$PWD":/code \
      --env=LM_REPORT_VERSION="2.1" \
      --env=CI_PROJECT_DIR=/code \
      registry.gitlab.com/gitlab-org/security-products/analyzers/license-finder:latest
    ```

1. The results will be stored in the `gl-license-scanning-report.json` file in the application directory.

## Development

### Getting Started

Install [Docker](https://docs.docker.com/engine/installation/).

```sh
$ git clone git@gitlab.com:gitlab-org/security-products/analyzers/license-finder.git
$ cd license-finder
```

Download the latest version of the Docker image.

```sh
$ ./bin/docker-pull
```

Launch a shell in the Docker container.

```sh
$ ./bin/docker-shell
```

### Running the application

License Management is a Docker image. You can build it like this from the project root:

```sh
$ ./bin/docker-build
```

You can then run License Management on some target directory:

```sh
$ docker run --rm --volume "/path/to/my/project":/code --env CI_PROJECT_DIR=/code
```

### Running the tests

You can run the tests from inside a docker container:

```sh
$ ./bin/docker-shell
$ ./bin/setup
$ ./bin/test [path to file]
```

If you need to debug any specific issues you can do this from within the docker container by
following these steps:

```sh
$ ./bin/docker-shell
$ enable_dev_mode
$ bundle open license_finder
```

The `docker-shell` script will mount the current project as a volume into `/builds/gitlab-org/security-products/analyzers/license-finder`.
This allows you to edit code from your host machine using your preferred editor and
see the affect of those changes from within the running docker container.

### Building Debian packages

Debian packages for each tool is built using [omnibus](https://github.com/chef/omnibus).
Read the [documentation](https://github.com/chef/omnibus#-omnibus) for more information.
The list of projects that can be built can be found in `config/projects/`.

E.g.

To build the `golang` package follow these steps:

```sh
$ ./bin/docker-omnibus
$ GOLANG_VERSION=1.15.3 ./bin/omnibus build golang
```

When a new version of a tool needs to be built, add
the new version to the tool file located in `./config/software/asdf_<tool>.rb`
and include the appropriate checksum.

### Updating the SPDX index

We will need to periodically update the SPDX index. This can be achieved with
the following command.

```bash
$ ./bin/update-spdx
```

## Supported languages and package managers

The following table shows which languages and package managers are supported.

| Language   | Package managers                                                  |
|------------|-------------------------------------------------------------------|
| .NET       | [.NET Core CLI][dotnet_core], [Nuget][nuget]                      |
| C/C++      | [Conan][conan]                                                    |
| Go         | [Go modules][gomod], [Godep][godep], go get                       |
| Java       | [Gradle][gradle], [Maven v3.2.5+][maven]                          |
| JavaScript | [npm][npm], [yarn][yarn], [Bower][bower]                          |
| PHP        | [composer][composer]                                              |
| Python     | [pip][pip], [pipenv][pipenv]                                      |
| Ruby       | [Bundler][bundler]                                                |

Inject `SETUP_CMD` to the docker command to override the given package managers
and run your custom command to setup your environment with a custom package manager.

```sh
docker run \
  --volume "$PWD":/code \
  --env "SETUP_CMD=./my-custom-install-script.sh" \
  --rm \
  registry.gitlab.com/gitlab-org/security-products/analyzers/license-finder:latest analyze /code
```

## Settings

The License Management tool can be customized with environments variables for some project types.

| Environment variable | Project type | Function |
|----------------------|--------------|----------|
| ADDITIONAL_CA_CERT_BUNDLE | * | Additional certificate chain to install in the trusted store. |
| MAVEN_CLI_OPTS       | Java (Maven) | Additional arguments for the mvn executable. If not supplied, defaults to `-DskipTests`. |
| LICENSE_FINDER_CLI_OPTS | * | Additional arguments for the `license_finder` executable. |
| LM_JAVA_VERSION      | Java (Maven) | Version of Java. If set to `11`, Maven and Gradle use Java 11 instead of Java 8. |
| LM_PYTHON_VERSION    | Python       | Version of Python. If set to `3`, dependencies are installed using Python 3 instead of Python 2.7. |
| LOG_LEVEL    | * | Control the verbosity of the logs. (`debug`, `info`, `warn` (default), `error`, `fatal`)  |
| LM_REPORT_FILE    | * | Name of the generated report. If not supplied, defaults to `gl-license-scanning-report.json`  |


Inject the required environment variables to the docker command using the [`--env` option flag](https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file)
or its shorthand form (`--env MY_SETTING_VAR`) if the configuration comes from an external environment.

## Versioning and release process

Please check the [Release Process documentation](https://gitlab.com/gitlab-org/security-products/release/blob/master/docs/release_process.md).

## Upgrading to the latest version of LicenseFinder

1. Check for the latest version of `LicenseFinder` at [https://rubygems.org/gems/license_finder][license_finder]
1. Check the version of the `license_finder` gem that is currently being used in the [`Gemfile.lock`][gemfile_lock]
1. If an update is available, create a new branch
1. Bump the license management version in [CHANGELOG.md][changelog] and in [version.rb][version_rb]
1. Update the `license_finder` version constraint in the [gemspec][gemspec]
1. Run `bundle update license_finder`
1. Test the changes by following the instructions for [running the tests](#running-the-tests)
1. Submit a merge request.

## Step by Step: Detection

1. Run the Docker image:

    ```sh
    docker run \
      --volume "$PWD":/code \
      --env=LM_REPORT_VERSION="2.1" \
      --env=CI_PROJECT_DIR=/code \
      registry.gitlab.com/gitlab-org/security-products/analyzers/license-finder:latest
    ```

1. The `ENTRYPOINT` for the container will execute [run.sh](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/191185c4303768c6d9a1431c35143501c06ee4d7/run.sh):

    ```Dockerfile
    ENTRYPOINT ["/run.sh"]
    ```

1. This shell script sets up the runtime environment then invokes the `license_management` executable:

    ```sh
    #!/bin/bash -l
    export LM_JAVA_VERSION=${LM_JAVA_VERSION:-"8"}
    export LM_PYTHON_VERSION=${LM_PYTHON_VERSION:-"3"}
    export LM_REPORT_FILE=${LM_REPORT_FILE:-'gl-license-scanning-report.json'}
    ...
    license_management report $@
    ```

1. The `license_management` executable loads monkey patches for [license_finder][license_finder] then invokes the CLI:

    ```ruby
    require 'license/management'

    LicenseFinder::CLI::Main.start(ARGV)
    ```

1. [license_finder][license_finder] searches for lockfiles in the project.

    ```ruby
    def active?
      project_path.join('pom.xml').exist?
    end
    ```

1. When a [license_finder][license_finder] determines that a package manager is active, it then invokes the `prepare` step for that package manager.

    ```ruby
    def prepare
      within_project_path do
        tool_box.install(tool: :java, version: java_version, env: default_env)
      end
    end
    ```

1. The `tool_box` determines the required version of tools (i.e Java, Ruby, Python etc) for the package manager and then installs it by looking in `/opt/toolcache/` for a matching `*.deb` file or falls back to `asdf` to install the tool from source.

    ```ruby
    def install(tool:, version: , env: {})
      Dir.chdir project_path do
        deb = deb_for(tool, version)
        shell.execute([:dpkg, '-i', deb]) if deb&.exist?
        shell.execute([:asdf, :install, tool.to_s, version], env: env)
      end
    end

    def deb_for(tool, version)
      Pathname.glob("/opt/toolcache/#{tool}-#{version}*.deb")[0]
    end
    ```

1. After the tool(s) are installed the package manager class builds a list of dependencies identified in the project. If an `install_path` is provided then the files in this directory are scanned for software licenses.

    ```ruby
    def current_packages
      within_project_path do
        return [] unless shell.execute(detect_licenses_command, env: default_env)[-1].success?

        resource_files.flat_map { |file| map_from(file.read) }.uniq
      end
    end
    ```

1. Once all the dependencies and their licenses are identified a JSON report is generated for the desired version of the report. The `Report` class is backwards compatible and able to generate any previous version of the report.

    ```ruby
    def to_s
      JSON.pretty_generate(version_for(report_version).to_h)
    end

    def version_for(version)
      VERSIONS.fetch(version.to_s).new(dependencies)
    end
    ```

1. The final JSON report is written to [gl-license-scanning-report.json](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/191185c4303768c6d9a1431c35143501c06ee4d7/spec/fixtures/schema/v2.1.json) in the root of the project.

    ```json
    {
      "version": "2.1",
      "licenses": [
        {
          "id": "MPL-2.0",
          "name": "Mozilla Public License 2.0",
          "url": "https://opensource.org/licenses/MPL-2.0"
        }
      ],
      "dependencies": [
        {
          "name": "rhino",
          "version": "1.7.10",
          "package_manager": "maven",
          "path": "pom.xml",
          "licenses": [
            "MPL-2.0"
          ]
        }
      ]
    }
    ```

# Contributing

If you want to help, read the [contribution guidelines](CONTRIBUTING.md).

If an unknown license is detected, please consider updating the mapping defined
in [normalized-licenses.yml](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/blob/master/normalized-licenses.yml). A mapping can be for a detected name or url and must correspond to an SPDX identifier found in [spdx-licenses.json](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/blob/master/spdx-licenses.json).

[bower]: https://bower.io/
[bundler]: https://bundler.io/
[changelog]: https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/master/CHANGELOG.md
[composer]: https://getcomposer.org
[conan]: https://conan.io/
[dotnet_core]: https://docs.microsoft.com/en-us/dotnet/core/tools/
[gemfile_lock]: https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/master/Gemfile.lock
[gemspec]: https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/master/license-management.gemspec
[godep]: https://github.com/tools/godep
[gomod]: https://github.com/golang/go/wiki/Modules
[gradle]: https://gradle.org/
[license_finder]: https://rubygems.org/gems/license_finder
[maven]: https://maven.apache.org/
[npm]: https://www.npmjs.com/
[nuget]: https://www.nuget.org/
[pip]: https://pip.pypa.io/en/stable/
[pipenv]: https://github.com/pypa/pipenv
[version_rb]: https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/master/lib/license/management/version.rb
[yarn]: https://yarnpkg.com/