Artifactory Tech Talk Notes

Introduction to Artifactory

Where is it?

Trial instance

What is it?

Essentially GitHub for binary files.

From wikipedia:

A binary repository manager is a software tool designed to optimize the download and storage of binary files used and produced in software development. It centralizes the management of all the binary artifacts generated and used by the organization to overcome the complexity arising from the diversity of binary artifact types, their position in the overall workflow and the dependencies between them.

A binary repository is a software repository for packages, artifacts and their corresponding metadata. It can be used to store binary files produced by an organization itself, such as product releases and nightly product builds, or for third party binaries which must be treated differently for both technical and legal reasons.

Artifactory

How does it work?

Artifactory Pro provides a number of predefined repository templates that all follow the same pattern with a set of three repositories:

  1. A remote repository which provides indirect access to the public repository. For example, there’s a docker-remote repository that provides access to Docker Hub through Artifactory. When you download an image, such as the community golang image, for the first time Artifactory allows you to access it via the remote repository and stores it in docker-remote-cache for future requests.

  2. A local repository which stores artifacts that you upload to artifactory either directly or as the output of a build. In the case of Docker, if you build a new image based on the golang image and push it to Artifactory, it’s saved in the docker-local repository.

  3. A virtual repository which combines the remote and local repository so that clients can request artifacts from the repository without needing to know if they are available locally or remotely.

Repository types - Artifactory Pro

Docker

Pulling an existing image from the Artifactory Docker repository

  1. Verify that you can contact the Artifactory server

curl -I -k -v http://b4v7eywig.activestate.com:8081/artifactory/api/system/ping

  1. Log in

docker login -u admin -p thisisatest123 b4v7eywig.activestate.com:8081

  1. Pull an image

docker pull b4v7eywig.activestate.com:8081/docker/active-python-3-5-4:latest

Pulling an image that isn’t in the local Artifactory Docker repository

docker pull b4v7eywig.activestate.com:8081/docker/golang

In this case the first time the image is pulled it comes from library\golang on Docker Hub and is stored in library\golang in Artifactory’s docker-remote-cache repository. Future pulls of the same image will come from the cache.

Tagging an updated image and pushing to the repository

  1. Make your changes and build the new image

docker build --rm -t b4v7eywig.activestate.com:8081/docker/active-python-3-5-4:3.0 -t b4v7eywig.activestate.com:8081/docker/active-python-3-5-4:latest .

  1. Push the updates to Artifactory

docker push b4v7eywig.activestate.com:8081/docker/active-python-3-5-4:3.0

docker push b4v7eywig.activestate.com:8081/docker/active-python-3-5-4:latest

Note: You need to configure the Artifactory registry as an insecure registry since the trial instance isn’t using ssl. When configured for production the commands would be more like typical docker hub commands, for example:

docker push hub.activestate.com/docker/active-python-3-5-4:latest

Docker insecure registry configuration

Python Package Index (PyPI)

~/.pip/pip.conf

~/.pypirc

Each Artifactory repository has a “Set me up” link that provides details about the configuration changes you need to make.

pip install pythonPT --trusted-host b4v7eywig.activestate.com
pip install pythonPT==0.1.0 --trusted-host b4v7eywig.activestate.com
pip install pythonPT==0.1.1 --trusted-host b4v7eywig.activestate.com
pip install bleach --trusted-host b4v7eywig.activestate.com

pythonPT is a custom package so it is stored in the local repository. bleach comes from PyPi via the remote repository and is stored in pypy-remote-cache for future requests. To the end user it looks the same. They just need to do a standard pip install.

Install packages

Note: Once it’s configured Pip works just like it would if it was pointing to the PyPi repository. The --trusted-host flag is only required because we don’t have ssl configured on the trial instance.

Go and vgo

Vgo is the proposed official standard dependency management tool and registry specification for Go. While not yet included in the official Go tools set, it is poised to be the standard dependency manager distributed with each Go release. Artifactory supports vgo though its jfrog cli tool. It fills in a few gaps that currently exist in the vgo CLI. JFrog CLI wraps vgo and adds additional capabilities with simple commands.

  • Local repositories in Artifactory let you set up secure, private Go registries with fine-grained access control to packages according to projects or development teams.
  • A remote repository in Artifactory is a caching proxy for remote Go resources such as a GitHub project. Accessing a go proxy through Artifactory removes your dependency on the network, or on GitHub since all dependencies needed for your Go builds are cached in Artifactory and are therefore locally available. This also removes the risk of someone mutating or removing a dependency from version control, or worse, force-pushing changes to a remote Git tag thus changing what should be an immutable version which can create a lot of confusion and instability for dependant projects.
  • A virtual repository aggregates local and remote registries
  1. Install go and vgo
  2. Install the jfrog cli: curl -fL https://getcli.jfrog.io | sh
  3. Configure the connection information for the Artifactory server
  4. Use the jfog cli to build and deploy your release
jfrog rt config  ## Use jfrog rt config show to display an existing configuration
jfrog rt go build --no-registry 
jfrog rt gp go-local --self=false --deps=ALL
jfrog rt go build go-virtual
jfrog rt gp go-local v1.0.0 --build-name=my-build --build-number=1
jfrog rt bce my-build 1
jfrog rt bp my-build 1

Perl and CPAN:

Artifactory does not have support for CPAN. It’s been requested but the best they can do right now is cache modules that have been requested from CPAN. Possibly the same can be done for Tcl.

How to proxy a CPAN repo with Artifactory

Circleci

NPM

[CircleCI project]() Github repo

You can adding jFrog cli commands to the CircleCI configuration file (.circleci/config.yaml) publish builds to Artifactory.

  1. Download the jFrog cli
  2. Configure the Artifactory connection. Uses environment variables configured in
  3. Run NPM install (npmi).
  4. Collect environment variables.
  5. Pack and deploy the npm package to the designated npm repository.
  6. Publish build info
- run: curl -fL https://getcli.jfrog.io | sh
- run: ./jfrog rt config --url $ARTIFACTORY_URL --user $ARTIFACTORY_USER --password $ARTIFACTORY_PASSWORD --interactive=false
- run: rm -rf node_modules/
- run: ./jfrog rt npmi npm --build-name=$CIRCLE_JOB --build-number=$CIRCLE_BUILD_NUM
- run: ./jfrog rt bce circleci-npm-artifactory $CIRCLE_BUILD_NUM
- run: ./jfrog rt npmp npm --build-name=circleci-npm-artifactory --build-number=$CIRCLE_BUILD_NUM
- run: ./jfrog rt bp circleci-npm-artifactory $CIRCLE_BUILD_NUM

Full config file

Python

- run: curl -fL https://getcli.jfrog.io | sh
- run: ./jfrog rt config --url $ARTIFACTORY_URL --user $ARTIFACTORY_USER --password $ARTIFACTORY_PASSWORD --interactive=false
- run: ./jfrog rt bce circleci-python-artifactory $CIRCLE_BUILD_NUM
- run: ./jfrog rt bp circleci-python-artifactory $CIRCLE_BUILD_NUM

CircleCI project Github repo

Full config file

SWOT (or how does it affect ActiveState?)

Strengths (that we can take advantage of)

  • Focuses on a specific problem and appears to do it well
  • Standardizes the way you interact with a wide range of package managers and open source repositories. Provides one place to manage a wide variety of artifacts.
  • Supports a variety of use cases from open source to high-availbility, global deployments

Weaknesses

  • No specific support of Perl (CPAN) or Tcl (Gutter) repositories. For Perl it is possible to configure a generic repository to cache modules after they are downloaded from CPAN.
  • Doesn’t tell administrators why is a package needed (i.e. what apps use a package, who will care if it’s removed)

Opportunities

  • Accelerate Platform development by providing some of the low level plumbing for interacting with repositories, archiving builds, and managing artifacts.
  • Provides a complete API and CLI client that we can integrate with
  • jFrog doesn’t seem too focused on the individual developer beyond supporting the various package management tools. More devops focused.

Threats (from jFrog as a competitor)

  • jFrog has established a foothold in lots of companies
  • They’re building a platform that is capable of complementing the ActiveSate Platform, but is also in some feature areas a direct competitor. For example, they have a product for scanning binaries (X-Ray) and also have a Black Duck integration.

Resources

Project Examples

Frog’s Blog

Observations

  1. Interesting to look at how they’ve segmented paid vs free/OSS features and Pro vs. Enterprise
  2. Support for on prem, hosted, and cloud deployments
  3. Integration with existing tooling (e.g. pip), jfrog cli, and API access

Misc Notes

Setup tools

“If you do not have a CI integration or your organizational procedures require you to configure package managers and build tools on individual workstations, Artifactory provides you with an easy to use Set Me Up feature. This feature will auto-generate the configuration snippets and commands you need.

If you have a CI/CD setup, you can integrate Artifactory into your build ecosystem and gain visibility of artifacts deployed, dependencies and information on the build environment. You can connect your workstations to Artifactory, with all the necessary configuration of the various package managers and clients, using our CI server integrations.”

Implementation flow

  1. Setup Artifactory alongside your current system.
  2. Configure your builds to start pushing content, including builds and artifacts, to Artifactory while still maintaining your old system. i.e retrieve/deploy to your legacy system but in parallel also deploy to Artifactory.
  3. Start deploying and retrieving content directly to/from Artifactory. But at the same time continue to deploy your content to the legacy system, for the consumption of other teams/projects.
  4. Once you are ready and confident with your Artifactory integration, start migrating your projects one by one or in bulk.

Leveraging Artifactory

Some of the inherent added value of introducing Artifactory into your environment is improving the build process. For example, utilizing your build info and metadata will allow you to have traceable builds and easily configured retention procedures.

Having traceable builds will prevent the delay of build process by stages like compliance, security and release management. On the one hand, it will allow you to separate the development process from the post-development stages such as QA, compliance, and security. And on the other hand still maintain bidirectional connection between all of this post-development steps.