Use Your Private Python Libraries from Artifactory in Zepl

Zack Shainsky

December 11, 2020

Problem

Oftentimes data science teams build their own custom libraries and store them in common repositories such as Artifactory or Github. These libraries are the culmination of many hours of work and are critical for the data science process. The inevitable need to move from local notebook development environments to cloud-based resources is pushing teams to consider how they will continue to access their private python libraries. This blog will show you how you can access these libraries securely from the Zepl notebook.

Please note:
This blog does not focus on public libraries hosted on the public PyPI server. Zepl has an easy to download and build reusable images from a list of required libraries through our Custom Image builder.

Solution

We will use Zepl’s flexible notebook interface, to access the container shell to integrate our team’s Artifactory Python libraries in Zepl. This same approach can be used to access other types of repositories such as Maven (Java) and CRAN (R).

How to Guide

Step 1. Set up an Artifactory PyPI repo & Python package

Set up an Artifactory PyPI repository and added Python packages. If this is something you have never done before, you can learn more about doing so here:

Python: Build Python Packages
Artifactory: Create PyPI Repositories & Upload Packages

Step 2. Access the Container Shell

Each Zepl notebook runs on a linux container. A useful tool in Zepl is accessing the container’s shell to run Operating System commands. This is handled through the python interpreter (%python) and using the ‘!’ symbol.

%python

!<linux shell command here>

In Python, installing libraries is handled by a package manager called, pip. Pip searches for library packages based on a package name and index-url provided. The default index-url is https://pypi.python.org/simple, however we will update this value in step 3 to match your Artifactory repository index-url value.

Step 3. Install Custom Library

In one single line, here is how you can install a Package from your Artifactory repository:

%python

!pip install <LIBRARYNAME>==<LIBRARYVERSION> –index https://<USERNAME>:<PASSWORD>@<JFROG ACCOUNT URL>/artifactory/api/pypi/<YOUR REPO NAME>/simple

Replace these variables in your code:

<USERNAME> : Your Artifactory username
<PASSWORD> : Your Artifactory user password
<JFROG ACCOUNT URL>: Your Artifactory account url
<YOUR REPO NAME>: Repository with desired package to install

In order to take address security concerns of exposing your user password in clear text, take advantage Zepl’s new Secret Store:

We can verify that the package was installed by running this command:

%python
!pip show

Step 4. Import Custom Library

This is the final step! Make sure that the install statement worked by importing the library into your python code. My example installs a library called “capitalize” and here is what it looks like in Python:

%python
import capitalize
capitalize.capital_mod.capitalize_line("print all words in this sentence as capitals")

Open in Zepl

This Zepl notebook shows an extended example. Feel free to copy or import our sample notebook here.

Conclusion

Installing from Artifactory isn’t much harder than installing a public package. With one single line of code we are able to easily connect to a private Artifactory PyPI repository, install the library in a Zepl container!

Use Your Private Python Libraries from Artifactory in Zepl

Blog