Custom Datasource¶

Table of values¶

Name	Value	Notes
Identifier	`custom`
Default versioning	No default versioning
Custom registry support	Yes
Release timestamp support	No
Source URL support	No

Description¶

This custom datasource allows requesting version data from generic HTTP(S) endpoints.

Usage¶

The customDatasources option takes a record of customDatasource configs. This example shows how to update the k3s.version file with a custom datasource and a regex custom manager:

Options:

option	default	description
defaultRegistryUrlTemplate	`""`	URL used if no `registryUrl` is provided when looking up new releases
format	`"json"`	format used by the API. Available values are: `html`, `json`, `plain`, `toml`, `yaml`
transformTemplates	`[]`	JSONata rules to transform the API output. Each rule will be evaluated after another and the result will be used as input to the next

Tip

Use JSONata Exerciser to test your JSONata rules.

Available template variables:

packageName
currentValue

{
  "customManagers": [
    {
      "customType": "regex",
      "managerFilePatterns": ["/k3s.version/"],
      "matchStrings": ["(?<currentValue>\\S+)"],
      "depNameTemplate": "k3s",
      "versioningTemplate": "semver-coerced",
      "datasourceTemplate": "custom.k3s"
    }
  ],
  "customDatasources": {
    "k3s": {
      "defaultRegistryUrlTemplate": "https://update.k3s.io/v1-release/channels",
      "transformTemplates": [
        "{\"releases\":[{\"version\": $$.(data[id = 'stable'].latest),\"sourceUrl\":\"https://github.com/k3s-io/k3s\",\"changelogUrl\":$join([\"https://github.com/k3s-io/k3s/releases/tag/\",data[id = 'stable'].latest])}],\"sourceUrl\": \"https://github.com/k3s-io/k3s\",\"homepage\": \"https://k3s.io/\"}"
      ]
    }
  }
}

After all transformations, the resulting JSON must match one of these formats:

Minimal-supported object:

{
  "releases": [
    {
      "version": "v1.1.0"
    },
    {
      "version": "v1.2.0"
    }
  ]
}

All available options:

{
  "releases": [
    {
      "version": "v1.0.0",
      "isDeprecated": true,
      "releaseTimestamp": "2022-12-24T18:21Z",
      "changelogUrl": "https://github.com/demo-org/demo/blob/main/CHANGELOG.md#v0710",
      "sourceUrl": "https://github.com/demo-org/demo",
      "sourceDirectory": "monorepo/folder",
      "digest": "c667f758f9e46e1d8111698e8d3a181c0b10f430",
      "isStable": true
    }
  ],
  "sourceUrl": "https://github.com/demo-org/demo",
  "sourceDirectory": "monorepo/folder",
  "changelogUrl": "https://github.com/demo-org/demo/blob/main/CHANGELOG.md",
  "homepage": "https://demo.org"
}

Debugging¶

Renovate writes tracing logs entries before transformation. If Renovate finds an unexpected dataformat, it also writes a tracing log after transformation.

Getting trace level logs on hosted app¶

If you use the Mend Renovate app, use the logLevelRemap config option to get the trace log.

Getting trace logs from the Mend Renovate app

{
  "logLevelRemap": [
    {
      "matchMessage": "/^Custom datasource/",
      "newLogLevel": "info"
    }
  ]
}

Getting trace level logs when self-hosting¶

If you self-host Renovate, follow these steps to get the trace logs:

Set the LOG_FILE_LEVEL env var to trace
Run Renovate in dryRun mode

Formats¶

JSON¶

If json is used processing works as described above. The returned body will be directly interpreted as JSON and forwarded to the transformation rules.

Plain¶

If the format is set to plain, Renovate will call the HTTP endpoint with the Accept header value text/plain. The body of the response will be treated as plain text and will be converted into JSON.

Suppose the body of the HTTP response is as follows:

1.0.0
2.0.0
3.0.0

When Renovate receives this response with the plain format, it will convert it into the following:

{
  "releases": [
    {
      "version": "1.0.0"
    },
    {
      "version": "2.0.0"
    },
    {
      "version": "3.0.0"
    }
  ]
}

After the conversion, any jsonata rules defined in the transformTemplates section will be applied as usual to further process the JSON data.

Yaml¶

If yaml is used, response is parsed and converted into JSON for further processing.

Suppose the body of the HTTP response is as follows:

releases:
  - version: 1.0.0
  - version: 2.0.0
  - version: 3.0.0

When Renovate receives this response with the yaml format, it will convert it into the following:

{
  "releases": [
    {
      "version": "1.0.0"
    },
    {
      "version": "2.0.0"
    },
    {
      "version": "3.0.0"
    }
  ]
}

After the conversion, any jsonata rules defined in the transformTemplates section will be applied as usual to further process the JSON data.

TOML¶

If toml is used, response is parsed and converted into TOML for further processing.

The below TOML document

[[releases]]
version = "1.0.0"
[[releases]]
version = "2.0.0"
[[releases]]
version = "3.0.0"

Will convert applying any jsonata rules defined in the transformTemplates section will be applied.

{
  "releases": [
    {
      "version": "1.0.0"
    },
    {
      "version": "2.0.0"
    },
    {
      "version": "3.0.0"
    }
  ]
}

HTML¶

If the format is set to html, Renovate will call the HTTP endpoint with the Accept header value text/html. The body of the response will be treated as a HTML document, and all hyperlinks will be converted to versions.

For the following HTML document:

<html>
  <body>
    <a href="package-1.0.tar.gz">package-1.0.tar.gz</a>
    <a href="package-2.0.tar.gz">package-2.0.tar.gz</a>
  </body>
</html>

The following JSON will be generated:

{
  "releases": [
    {
      "version": "package-1.0.tar.gz"
    },
    {
      "version": "package-1.0.tar.gz"
    }
  ]
}

After the conversion, any jsonata rules defined in the transformTemplates section will be applied to process the JSON data.

To extract the version number, you may use extractVersion or JSONata rules.

Examples¶

K3s¶

You can use this configuration to request the newest version available to K3s

{
  "customDatasources": {
    "k3s": {
      "defaultRegistryUrlTemplate": "https://update.k3s.io/v1-release/channels",
      "transformTemplates": [
        "{\"releases\":[{\"version\": $$.(data[id = 'stable'].latest),\"sourceUrl\":\"https://github.com/k3s-io/k3s\",\"changelogUrl\":$join([\"https://github.com/k3s-io/k3s/releases/tag/\",data[id = 'stable'].latest])}],\"sourceUrl\": \"https://github.com/k3s-io/k3s\",\"homepage\": \"https://k3s.io/\"}"
      ]
    }
  }
}

Hashicorp¶

You can use this configuration to request the newest versions of the Hashicorp products:

{
  "customManagers": [
    {
      "customType": "regex",
      "managerFilePatterns": ["/\\.yml$/"],
      "datasourceTemplate": "custom.hashicorp",
      "matchStrings": [
        "#\\s*renovate:\\s*(datasource=(?<datasource>.*?) )?depName=(?<depName>.*?)( versioning=(?<versioning>.*?))?\\s*\\w*:\\s*(?<currentValue>.*)\\s"
      ],
      "versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
    }
  ],
  "customDatasources": {
    "hashicorp": {
      "defaultRegistryUrlTemplate": "https://api.releases.hashicorp.com/v1/releases/{{packageName}}?license_class=oss",
      "transformTemplates": [
        "{ \"releases\": $map($, function($v) { { \"version\": $v.version, \"releaseTimestamp\": $v.timestamp_created, \"changelogUrl\": $v.url_changelog, \"sourceUrl\": $v.url_source_repository } }), \"homepage\": $[0].url_project_website, \"sourceUrl\": $[0].url_source_repository }"
      ]
    }
  }
}

To have the latest Nomad version in your Ansible variables, use this snippet after adding the above configuration:

# renovate: depName=nomad
nomad_version: 1.6.0

Grafana Dashboard¶

You can use the following configuration to upgrade the Grafana Dashboards versions in your Grafana Helm chart:

{
  "customManagers": [
    {
      "customType": "regex",
      "managerFilePatterns": ["/\\.yml$/"],
      "matchStrings": [
        "#\\s+renovate:\\s+depName=\"(?<depName>.*)\"\\n\\s+gnetId:\\s+(?<packageName>.*?)\\n\\s+revision:\\s+(?<currentValue>.*)"
      ],
      "versioningTemplate": "regex:^(?<major>\\d+)$",
      "datasourceTemplate": "custom.grafana-dashboards"
    }
  ],
  "customDatasources": {
    "grafana-dashboards": {
      "defaultRegistryUrlTemplate": "https://grafana.com/api/dashboards/{{packageName}}",
      "format": "json",
      "transformTemplates": [
        "{\"releases\":[{\"version\": $string(revision)}]}"
      ]
    }
  }
}

Grafana Helm chart values.yaml snippet:

dashboards:
  default:
    1860-node-exporter-full:
      # renovate: depName="Node Exporter Full"
      gnetId: 1860
      revision: 31
      datasource: Prometheus
    15760-kubernetes-views-pods:
      # renovate: depName="Kubernetes / Views / Pods"
      gnetId: 15760
      revision: 20
      datasource: Prometheus

Custom offline dependencies¶

Sometimes the "dependency version source" is not available via an API. To work around a missing API, you can create dependency "files". These files are served via HTTP(S), so that Renovate can access them. For example, imagine the following file versiontracker.json for the software something:

[
  {
    "version": "77"
  },
  {
    "version": "76"
  }
]

By writing a custom datasource, Renovate can process the versiontracker.json file, see below. This example uses Nexus as the webserver.

{
  "customDatasources": {
    "nexus_generic": {
      "defaultRegistryUrlTemplate": "https://nexus.example.com/repository/versiontrackers/{{packageName}}/versiontracker.json",
      "transformTemplates": [
        "{ \"releases\": $map($, function($v) { { \"version\": $v.version, \"sourceUrl\": $v.filelink } }) }"
      ]
    }
  }
}

This could be used to update Ansible YAML files with the latest version through a custom manager. For example, with the following Ansible content:

# renovate: datasource=custom.nexus_generic depName=something versioning=loose
something_version: '77'

And the following custom manager:

{
  "customManagers": [
    {
      "customType": "regex",
      "managerFilePatterns": ["/\\.yml$/"],
      "datasourceTemplate": "custom.nexus_generic",
      "matchStrings": [
        "#\\s*renovate:\\s*(datasource=(?<datasource>.*?)\\s*)?depName=(?<depName>.*?)(\\s*versioning=(?<versioning>.*?))?\\s*\\w*:\\s*[\"']?(?<currentValue>.+?)[\"']?\\s"
      ],
      "versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
    }
  ]
}

Or if you have the datasource locally, you can also define your local registry by prefixing it with file://:

{
  "customDatasources": {
    "local_generic": {
      "defaultRegistryUrlTemplate": "file://dependencies/{{packageName}}/versiontracker.json",
      "transformTemplates": [
        "{ \"releases\": $map($, function($v) { { \"version\": $v.version, \"sourceUrl\": $v.filelink } }) }"
      ]
    }
  }
}

Renovate will then parse your file from your current folder to access it.

nginx directory listing¶

Sometimes all you have is a directory with files, and a HTTP server that can generate directory listings.

Let's use nginx itself as an example:

{
  "customDatasources": {
    "nginx": {
      "defaultRegistryUrlTemplate": "https://nginx.org/download",
      "format": "html"
    }
  },
  "packageRules": [
    {
      "matchDatasources": ["custom.nginx"],
      "extractVersion": "^nginx-(?<version>.+)\\.tar\\.gz$"
    }
  ]
}

HTML page¶

You can use the html format to extract versions from a typical "Downloads" page:

{
  "customDatasources": {
    "curl": {
      "defaultRegistryUrlTemplate": "https://curl.se/download.html",
      "format": "html"
    }
  },
  "packageRules": [
    {
      "matchDatasources": ["custom.curl"],
      "extractVersion": "/curl-(?<version>.+)\\.tar\\.gz$"
    }
  ]
}