Tag Archives: package

How to Create a Julia Package from Scratch

By: Great Lakes Consulting

Re-posted from: https://blog.glcs.io/package-creation

This post was written by Steven Whitaker.

The Julia programming languageis a high-level languagethat is known, at least in part,for its excellent package managerand outstanding composability.(See another blog post that illustrates this composability.)

Julia makes it super easyfor anybody to create their own package.Julia’s package manager enables easy development and testing of packages.The ease of package developmentencourages developers to split reusable chunks of codeinto individual packages,further enhancing Julia’s composability.

In this post,we will learn what comprises a Julia package.We will also discuss toolsthat automate the creation of packages.Finally,we will talk about the basics of package developmentand walk through how to publish (register) a packagefor others to use.

This post assumes you are comfortable navigating the Julia REPL.If you need a refresher,check out our post on the Julia REPL.

Components of a Package

Packages are easy enough to use:just install them with add PkgName in the package promptand then run using PkgName in the julia prompt.But what actually goes into a package?

Packages must follow a specific directory structureand include certain informationto be recognized as a package by Julia.

Suppose we are creating a package called PracticePackage.jl.First, we create a directory called PracticePackage.This directory is the package root.Within the root directory we need a file called Project.tomland another directory called src.

The Project.toml requires the following information:

name = "PracticePackage"uuid = "11111111-2222-3333-aaaa-bbbbbbbbbbbb"authors = ["Your Name <youremail@email.com>"]version = "0.1.0"
  • uuid stands for universally unique identifier,and can be generated in Julia withusing UUIDs; uuid4().The purpose of a UUID is to allow different packages of the same name to coexist.
  • version should be set to whatever version is appropriate for your package,typically "0.1.0" or "1.0.0" for an initial release.The versioning of Julia packages follows SemVer.
  • The Project.toml will also include informationabout package dependencies,but more on that later.

The src directory requires one Julia filenamed PracticePackage.jlthat defines a module named PracticePackage:

module PracticePackage# Package code goes here.end

So, the directory structure of the packagelooks like the following:

PracticePackage Project.toml src     PracticePackage.jl

And that’s all there is to a package!(Well, at least minimally.)

Some Technicalities

Feel free to skip this section,but if you are curious about some technicalitiesfor what comprises a valid package,read on.

  • The Project.toml only needs the name and uuid fieldsfor Julia to recognize the package.Without the version field,Julia treats the version as v0.0.0.
    • However, the version and authors fields are neededto register the package.
  • The name of the package root directory doesn’t matter,meaning it doesn’t have to match the package name.However, the name field in Project.tomldoes have to match the name of the moduledefined in src/PracticePackage.jl,and the file name of src/PracticePackage.jl also has to match.
    • For example,we could change the name of the packageby setting name = "Oops" in Project.toml,renaming src/PracticePackage.jl to src/Oops.jl,and defining module Oops in that file.We would not have to rename the package root directoryfrom PracticePackage to Oops(though that would be a good idea to avoid confusion).

Automatically Generating Packages

The basic structure of a package is pretty simple,so there ought to be a way to automate it, right?(I mean, who wants to manually generate a UUID?)Good news: package creation can be automated!

Package generate Command

Julia comes with a generate package command built-in.First, change directoriesto where the package root directory should live,then run generate in the Julia package prompt:

pkg> generate PracticePackage

This command creates the package root directory PracticePackageand the Project.toml and src/PracticePackage.jl files.Some notes:

  • The Project.toml is pre-filled with the correct fields and values,including an automatically generated UUID.When I ran generate on my computer,it also pre-filled the authors fieldwith my name and email from my ~/.gitconfig file.
  • src/PracticePackage.jl is pre-filledwith a definition for the module PracticePackage.It also defines a function greet in the module,but typically you will replace that with your own code.

PkgTemplates.jl

The generate command works fine,but it’s barebones.For example,if you are planning on hosting your package on GitHub,you might want to include a GitHub Actionfor continuous integration (CI),so it would be niceto automate the creation of the appropriate .yml file.This is where PkgTemplates.jl comes in.

PkgTemplates.jl is a normal Julia package,so install it as usual and run using PkgTemplates.Then we can create our PracticePackage.jl:

t = Template(; dir = ".")t("PracticePackage")

Running this code creates the packagewith the following directory structure:

PracticePackage .git    .github    dependabot.yml    workflows        CI.yml        CompatHelper.yml        TagBot.yml .gitignore LICENSE Manifest.toml Project.toml README.md src    PracticePackage.jl test     runtests.jl

As you can see,PkgTemplates.jl automatically generates a lot of filesthat aid in following package development best practices,like adding CI and tests.

Note that many optionscan be supplied to Templateto customize what files are generated.See the PkgTemplates.jl docs for all the options.

Checklist of settings

Basic Package Development

Once your package is set up,the next step is to actually add code.Add the functions, types, constants, etc.that your package needsdirectly in the PracticePackage module in src/PracticePackage.jl,or add additional files in the src directoryand include them in the module.(See a previous blog post for more information about modules,though note that using modules directly works slightly differentlythan using packages.)

To add dependencies for your package to use,you will need to activate your project’s package environmentand then add packages.For example,if you want your package to use the DataFrames.jl package,start Julia and navigate to your package root directory.Then, activate the package environment and add the package:

(@v1.X) pkg> activate .(PracticePackage) pkg> add DataFrames

After this,you will be able to include using DataFramesin your package codeto enable the functionality provided by DataFrames.jl.

Adding packages after activating the package environmentedits the package’s Project.toml file.It adds a [deps] sectionthat lists the added packages and their UUIDs.In the example above,adding DataFrames.jladds the following lines to the Project.toml file:

[deps]DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"

(And (PracticePackage) pkg> rm DataFrames would remove the DataFrames = ... line,so it is best not to edit the [deps] section manually.)

Finally,to try out your package,activate your package environment (as above)and then load your package as usual:

julia> using PracticePackage # No need to `add PracticePackage` first.

Note that by default Julia will have to be restartedto reload any changes you make to your package code.If you want to avoid restarting Juliawhenever you make changes,check out Revise.jl.

Publishing/Registering a Package

Once your package is in working order,it is natural to want to publish the packagefor others to use.

A package can be publishedby registering it in a package registry,which basically is a map that tells the Julia package managerwhere to find a packageso it can be downloaded.

Treasure map

The General registry is the largest registryas well as the default registry used by Julia;most, if not all, of the most popular open-source packages(DataFrames.jl, Plots.jl, StaticArrays.jl, ModelingToolkit.jl, etc.)exist in General.Once a package is registered in General,it can be installed with pkg> add PracticePackage.

(Note that if registering a package is not desired for some reason,a package can be added via URL, e.g.,pkg> add https://github.com/username/PracticePackage.jl,assuming the package is in a public git repository.However,the package manager has limited abilityto manage packages added in this way;in particular,managing package versions must be done manually.)

The most common wayto register a package in Generalis to use Registrator.jl as a GitHub App.See the README for detailed instructions,but the process basically boils down to:

  1. Write/test package code.
  2. Update the version field in the Project.toml(e.g., to "0.1.0" or "1.0.0" for the first registered version).
  3. Add a comment with @JuliaRegistrator registerto the latest commit that should be includedin the registered version of the package.

Note that there are additional steps for preparing a package for publishingthat we did not discuss in this post(such as specifying compatible versionsof Julia and package dependencies).Refer to the General registry’s documentation and links therein for details.

Summary

In this post,we discussed creating Julia packages.We learned what comprises a package,how to automate package creation,and how to register a package in Julia’s General registry.

What package development tips do you have?Let us know in the comments below!

Additional Links

Cover image background provided by www.proflowers.com athttps://www.flickr.com/photos/127365614@N08/16011252136.

Treasure map image source: https://openclipart.org/detail/299283/x-marks-the-spot

]]>

Tips and tricks to register your first Julia package

By: Sören Dobberschütz

Re-posted from: http://sdobber.github.io/juliapackage/

FluxArchitectures has finally been published as a Julia package! Installation is now as easy as typing ] to activate the package manager, and then

add FluxArchitectures

at the REPL prompt.

While preparing the package, I noticed that a lot of good and useful information about some of the details of registering a package is spread out in different documentations and not easily accessible. In this post, I try to give a walkthrough of the different steps. There is also an older (but highly recommended) video from Chris Rackauckas explaining some of the process:

Package Template

Start out by creating a package using the PkgTemplate.jl-package. This package takes care of most of the internals for setting everything up. I decided to have automatic workflows for running tests and preparing the documentation done on Github, so the code for creating the template was

using PkgTemplates

t = Template(; 
    user="UserName",
    dir="~/Code/PkgLocation/",
    julia=v"1.6",
    plugins=[
        Git(; manifest=true, ssh=true),
        GitHubActions(; x86=true),
        Codecov(),
        Documenter{GitHubActions}(),
    ],
)

t("PkgName")

Of course, change UserName to your Github user name, PkgLocation and PkgName to the locaction and package name of your choice. A good starting point is this place of the PkgTemplates.jl documentation. There are a lot of options available, which are described in the documentation.

Depending on which Julia version you want to run your tests, you might need to edit the .github/workflows/CI.yml file. For example, to run tests also on the latest Julia nightly, edit the matrix section:

matrix:
        version:
          - '1.6'
          - 'nightly'

To allow for errors in the nightly version, edit the steps section to include a continue-on-error statement:

    - uses: julia-actions/julia-runtest@v1
      continue-on-error: ${{ matrix.version == 'nightly' }}

(The full file can be found here.)

Develop Code and Tests

The next step is of course to fill the template with Julia code (which goes into the src folder) and tests for your code (residing in the test folder).

Documentation

A package should have some documentation describing its features. It goes into the docs folder in the form of Markdown files. The make.jl file takes care of preparing everything, though we need to tell it the desired sidebar structure of our documentation. This goes into the pages keyword:

pages=[
        "Home" => "index.md",
        "Examples" => "examples/examples.md",
        "Exported Functions" => "functions.md",
        "Models" =>
                    ["Model 1" => "models/model1.md",
                     "Model 2" => "models/model2.md"],
        "Reference" => "reference.md",
    ]

All the files on the right side of the pairs need to exist in the docs/src-directory (or subfolders thereof). It is strongly advised to run the make.jl file locally and see if everything works.

See the Documenter.jl-documentation on how to automatically add docstrings from the package source code, add links to other sections etc. One can also learn a lot by looking at the code for other packages’ documentations.

External Data

In case your package includes some larger files with example data etc., it is a good idea to include them via Julia’s Artifact system. This consists of the following steps:

  • Create a .tar.gz archive of your files.
  • Upload the files to an accessible location (e.g. in a separate GitHub repository). For FluxArchitectures, I used a FA_data repository.
  • Create an Artifacts.toml file in your package folder containing information about the files to download.
  • Access your files in Julia by finding their location through the Artifact system – Julia will automatically take care of downloading them, storing them and making them accesible. This works like 
      using Pkg.Artifacts
  rootpath = artifact"DatasetName"

These steps are described in detail in the Pkg-documentation. For the toml-file, a file SHA and git tree SHA are needed. They can be produced by Julia itself – see the linked documentation. If one adds the lazy = true keyword to the section containing the git tree SHA, the data is only downloaded when the user requests it for the first time.

A common mistake is to “just” copy-paste the GitHub URL to the archive into the Artifacts.toml file, which will not work. Make sure to use a link to the “raw” data, which usually can be obtained by inserting a raw into the GitHub URL, for example as in https://github.com/sdobber/FA_data/raw/main/data.tar.gz.

Check Requirements

It is a good idea to check the requirements for new packages, which can be found in the RegistryCI.jl-documentation. This documents gives some hints about proper naming etc.

Publish to GitHub & Add JuliaRegistrator

If not done already, make sure that your package is available on GitHub.

Click on the “install app” button on JuliaRegistrator’s page and allow the bot to access your package repository.

Stable vs Dev Documentation

The CI.yml file created in the package template contains a workflow that will build your documentation and make it available in your repository. The documentation is pushed to a new branch called gh-pages. You might need to tell GitHub to use this branch as a “Github Pages” Site. Follow the instructions in this GitHub documentation, and set the “publishing source” to the gh-pages branch.

With the default settings, only documentation for the current development version of the package will be created. If you also want to create and keep documentation for each tagged version, you need to create and add a key pair to the DOCUMENTER_KEY secret of the GitHub repository. The easiest way I found for producing those is to install the package called DocumenterTools.jl and run

using DocumenterTools
DocumenterTools.genkeys()

in the Julia REPL.

The REPL-output will present you with two strings that need to be pasted into different places:

  • The first key needs to be added as a public key to your repository, see this documentation and start at step 2.
  • The second key needs to be added as a repository secret. Follow this document. The name of the secret needs to be DOCUMENTER_KEY, and the value is the output string from Julia.

Register Package

On GitHub, open a new issue in your repository, and write @JuliaRegistrator register in the comment area. The JuliaRegistrator bot will pick this up, and after a 3 day waiting period, your new package hopefully gets added to the general Julia registry.

Developing your Julia package

By: DSB

Re-posted from: https://medium.com/coffee-in-a-klein-bottle/developing-your-julia-package-682c1d309507?source=rss-8bd6ec95ab58------2

A Tutorial on how to quickly and easily develop your own Julia package

We’ll show step-by-step how to develop your first package in Julia. To do this, we use the very useful PkgTemplate.jl, and we base our Tutorial on this guide by Quantecon and this video by Chris Rackauckas.

First things first. Start by creating the folder to store your package. In our case, we’ll be creating a package named “VegaGraphs”, which is a package that I’m developing. So, inside this folder, open your Julia REPL and install PkgTemplate.jl.

Installing PkgTemplates

1. Creating your Package Template

First, we need to create a template for our package. This template will include things like Licensing, Plugins, Authors, etc. For our example, we’ll be using a simple template with the MIT License. The plugins used will be GitHub Actions and Codecov. I’ll not be diving into these plugins, but just for the sake of clarity, I’ll briefly explain what they do.

  • GitHub Actions is a plugin that automatically builds a virtual machine and tests your code, hence, you can control the machine configurations necessary to running your package.
  • Codecov is a plugin that analysis your code, and evaluates how much of it is covered in your tests. So, for example, suppose that you wrote 3 different functions, but forgot to write a test for one of them. Then, Codecov will point out that there is no tests for such function.

Still inside the REPL, run the following commands:

t = Template(;user="YourUserNameOnGithub", plugins = [GitHubActions(), Codecov()], manifest = true)
generate("VegaGraphs.jl",t)

The first line of code defines the template, while the second one will generate your package. Note that I didn’t specify a folder, so the package will be created in a default location, which will be ~/.julia/dev (for Linux).

Now, just copy the files from the ~/.julia/dev/VegaGraphs to the folder where you will be working from, and then setup your git repository by running the following commands in the terminal:

# inside the VegaGraphs working folder
git init
git add -A
git commit -m "first commit"
git branch -M master
git remote add origin git@github.com:YOURUSERNAME/VegaGraphs.git
git push -u origin master

Taking a look inside the generated folder, you’ll have two folders and four files:

./src
./test
README.md
LICENSE
Manifest.toml
Project.toml
  • /src : This folder is where you will write the code for your package per se;
  • /test : Here is for storing your tests;
  • Project.toml: This is where you will store information such as the author, dependencies, julia version, etc;
  • Manifest.toml : This is a machine generated file, and you should just leave it be;

The other files are self explanatory.

2. Writing Code

We are ready to start coding our package. Note that inside the /src folder we already have a file named VegaGraphs.jl , which is the main file of our package. We can do all our coding directly inside VegaGraphs.jl , but as our code gets large, this might become messy.

Instead, we can write many different files for organizing our code, and then use VegaGraphs.jl to join everything together. Let’s do an example. We’ll code a very simple function, which we’ll store in another file, called graph_functions.jl , that will also be inside the ./src folder.

Here is some example code:

# code inside graph_functions.jl
function sum_values(x,y)
  return x+y
end

The code above is a simple implementation of a function. Below I show how to actually make this function available to users. One just needs to “include” the graph_functions.jl file, and to export the plot_scatter . Once exported, the function is now available to anyone who imports our package.

# code inside VegaGraphs.jl
module VegaGraphs
using VegaLite
export sum_values
include("graph_functions.jl")
end

Note that, besides including our function, I’ve also imported the VegaLite package. Hence, I need to specify VegaLite.jl as a dependency. We’ll do this by using the REPL again.

Go to the root of your package and open the REPL by running the command julia in the terminal.Now, press ] . This will put you on “package mode”. Next, write activate . , which will activate the Julia environment to your current folder. Finally, write add VegaLite , and this will add VegaLite.jl to your dependencies inside the Project.toml file.

Adding dependency to your package

3. Creating and running tests

So we’ve implemented a function and added a dependency to our package. The following step is to write a test, to guarantee that our code is indeed working. The code is self-explanatory, we just write our test inside a testset. You can write as many as you like to guarantee that your function is working properly.

# code inside ./test/runtests.jl
using VegaGraphs
using Test
@testset "VegaGraphs.jl" begin
    x = 2
    y = 2
    @test VegaGraphs.sum_values(x,y) == 4
end

Once the test is written, we have to run it and see if everything is working. Again, go to the root of the project and open your REPL. Guarantee that your environment is activated and run the tests as shown in the image below:

Running tests for your package

This will run your tests and see if everything passes. Once everything passes, we can trust that our code is working properly, and we can move on to more implementations.

4. Workflow – Text editor + Jupyter Notebook

With everything shown up until now, you are already ready to develop your package in Julia. Still, you might be interested on how to develop an efficient workflow. There are many possibilities here, such as using IDEs such as Juno and VsCode. I prefer to use Jupyter Notebooks with Vim (my text editor of choice), and doing everything from the terminal.

To use your developing package in your Notebook, you will need to activate your environment in a similar way that we’ve been doing up until now. As you open a new Notebook, run the following code in the very first cell.

Using Jupyter Notebook for developing packages

Note here that besides activating the environment, we also imported a package called Revise. This package is very helpful, and it should be imported right in the beginning of your notebook, before you actually import your own package, otherwise it won’t work properly!

Every time you modify the code in your package, to load the changes to your notebook you would have to restart the kernel. But when you use Revise.jl, you don’t need to restart your kernel, just import your package again, and the modifications will be applied.

Now, my workflow is very simple. I use Vim to modify the code in my package, and use the Notebook for trying things out. Once I get everything working as it should, I write down some tests, and test the whole thing.

5. Registering/Publishing your Package

Finally, suppose that you’ve finished writing your package, and you are ready to share it with the Julia community. Like with other packages, you want users to be able to write a simple Pkg.add("MyPackage") and install it. This process is called registering.

To do this, first go to Registrator.jl and install the app to your Github account.

In the Registrator.jl Gitub page, click in the “install app”

Next, go to your package Github’s page and enter the Issues tab. Create a new issue, an write @JuliaRegistrator register() , as shown in the image below.

Registering your package

Once you’ve done this, the JuliaRegistrator bot will open a pull request for your package to be registered. Unless you actually want your package to be registered, don’t actually submit the issue.

And that’s all.

Developing your Julia package was originally published in Coffee in a Klein Bottle on Medium, where people are continuing the conversation by highlighting and responding to this story.