Repos
What is a repo?
A repo (also known as 'repository') is a collection of packages and some metadata describing a variety of attributes about the packages (for example: package versions, operating system version, dependencies, processor architecture, etc).
What is a packagecloud repo?
A packagecloud repo is a more advanced take on the traditional repo.
A single packagecloud repo can:
- Contain packages of any (or all!) supported types; for example Debian, RPM, RubyGem, and Python packages can all coexist in the same packagecloud repo.
- Have packages for multiple Linux distributions; for example, if you have a Debian package that works for 2 versions of Ubuntu and 1 version of Debian you only need 1 packagecloud repo (see pushing packages for more info).
- Issue read tokens to identify specific nodes and control access to a repo by specific node.
Why are repos useful?
Repos are useful because they:
- Allow you store several versions of the same package (useful for debugging purposes, security analysis, etc).
- Typically include some verification mechanism (like GPG).
- Offer a way for many remote machines to download and install the same packages.
- Provide a way for software vendors to offer stable releases of software to customers.
How do I create repos?
There are a few open source tools that you can use to create a repo. Some of the mostly commonly used tools are:
- createrepo for creating and managing RPM package repos (also called YUM repos).
- reprepro for creating and managing DEB package repos (also called APT repos).
- gem for creating and managing RubyGem servers.
Unfortunately, many of these tools are buggy, poorly documented, or difficult to use in production.
An excellent alternative is packagecloud.io ;) - you can read more about how to create repos with our command line tool here. You can also create repos on packagecloud.io by logging in and clicking the New Repository button or by clicking here.
Disabling filelists metadata
APT and YUM repositories have metadata which contains a list of every file in every package. This metadata can be used by users to query a given a filename and determine which package wrote that file to the filesystem.
The major downside of this feature is that repositories with huge numbers of packages or packages with huge numbers of files (like Omnibus packages) will have extremely large file list metadata. This can cause long wait times for clients because their apt and yum clients will download this metadata to the system when refreshing the cached metadata for a repository. This excessive metadata download can also increase the length of configuration management runs and take up significant space on the user's disk.
In order to deal with this, repository owners can click the 'edit' link to the right of their repository on their homepage and check the checkbox which says "Disable filelist metadata". Save your changes by clicking update. This will trigger a re-index for your repository and your contents metadata will be empty.
Debian repository component support
Since we make it easy to create and manage as many repositories as you want, we feel there is no real need to support Debian repository components.
Security Features
GPG keys
A GPG key can be used to generate a digital signature (read more about how GPG generates and verifies digital signatures).
Many package manage systems that come with Linux support two different types of GPG signatures. GPG signatures on packages themselves and GPG signatures on repository metadata. The distincition is very subtle, but very important.
GPG signed packages
If you want to GPG sign your packages, you should do so before uploading it to packagecloud. packagecloud does not modify your uploaded data in anyway, whatsoever. If you sign your packages before uploading them, they will be signed when downloaded.
packagecloud allows repository owners to upload the GPG key used to sign the packages in a repository. When you do this, the key URL will be included in the package manager config that is generated for the client host. This ensures the key will be downloaded, added to the keyring, and used to verify package signatures. Note that packagecloud only stores public keys. If you happen to upload a secret key, we'll discard the secret key data and store only the public key when we process your GPG key upload.
GPG signed repository metadata
packagecloud does GPG sign repository metadata that we generate for your repository. We do this to ensure to our users that your repository metadata was generated by us.
If you've installed your packagecloud repository using the Bash script, Chef cookbook, or Puppet module there is nothing extra you need to do. All of those methods automatically install the correct GPG key for your repository.
Adding and removing GPG keys
For more information about adding or removing the GPG keys for a repository, visit the repository's page and click GPG. Read the next section to learn about how to export a GPG key that you've used to sign your packages so it can be uploaded to packagecloud.
Exporting GPG keys
To export a GPG key that you've used to sign packages uploaded to packagecloud, follow these instructions.
- Begin by running
gpg --list-keys --keyid-format=long. This will print information about each GPG key in your keyring. It will look like this:
pub 2048R/F2BB309992AE12C3 2015-03-14
uid Fake Name (test key)
sub 2048R/5F042FCD59BF0D31 2015-03-14
- Find the key ID for the package signing key you are using. In the example above, the key ID is
F2BB309992AE12C3.
- Export the public key by running:
gpg --armor --export keyid, substituting keyid for your GPG key ID.
- Copy and paste this output into the package key dialog located in the Package signing key section of the GPG tab on your repository.
Note that if you accidentally upload a secret key, packagecloud will extract the public key and discard your secret key. We never store your secret key data even if uploaded to us accidentally.
Learn more
To learn more about the technical inner workings of GPG signing packages vs repository metadata, check out our blog posts about GPG signing and verifying Debian packages and APT repositories and GPG signing and verifying RPM packages and YUM repositories.
Public / Private Repos
packagecloud allows you to create as many public repos as you like. Depending on your plan you may also be able to create private repos, as well.
Public repos are:
- Read accessible by anyone.
- Best suited for companies and individuals who are intending to distribute open source software.
- The repo owner and the specified collaborators may push packages to the repo.
Private repos are:
- Read accessible only with a read token
- Best suited for companies and individuals who have proprietary software.
- The repo owner and the specified collaborators may push packages to the repo.
Collaborators
packagecloud allows you to authorize other packagecloud users to push packages to and yank packages from your
repos, as well as promote packages to other repositories.
You can add or remove collaborators by clicking the edit link found next to a repo name when you log in to packagecloud.
Token Auth
We support a sophisticated multi-level authentication system that facilitates a wide variety of use cases for private repos. There are two levels of tokens: master tokens and read tokens.
Master Tokens
Master tokens are at the top level and their only purpose is generating descendent read tokens.
Master tokens can't be used for reading from repos, pushing packages, modifying configurations, or doing anything else. That means that you can safely give master tokens to customers, embed them in configuration management manifests, or otherwise distribute them to untrusted parties.
Each repo comes with a default master token. If you don't have a need for maintaining fine-grained access control to your repos, then all packagecloud functionality will automatically use the default master token transparently to you.
Master tokens can be generated and destroyed with the package_cloud command line client. For example, to create a new token named test, you would run the following command:
> package_cloud master_token create username/reponame master_token_name
Read Tokens
Read tokens are what's used to authenticate against the packagecloud repository. In other words, the package manager on your system will send this token along with its requests for repo metadata and packages.
These tokens are read-only, meaning that they can't be used to modify the repo in any way or authenticate against any packagecloud endpoints aside from the package servers themselves.We recommend using one read token per node to maximize isolation.
Read tokens are generated automatically by our repo installation scripts. Our scripts automatically associate read tokens with a node's hostname. If you write your own config management manifests, generating and associating read tokens is a simple matter of making a call to a REST API.
Revocation
If you want to revoke access to one node at a time, you can revoke its read token using the CLI. This is one of the benefits of generating one read-token per node.
NOTE: Use the master token name and read token name NOT the actual token values
> package_cloud read_token destroy username/reponame master_token_name/read_token_name
By revoking a master token, you can revoke all of its read token descendents at once, making it straightforward to — at once — revoke an entire customer, datacenter, or other logical grouping that exists in your system.
NOTE: Use the master token name NOT the actual token value
> package_cloud master_token destroy username/reponame master_token_name
Generating a Read Token for Private Repo Access
Generating a read token is a matter of making a REST call to the tokens endpoint for your repository. The following example will generate and return a read token for the private repository.
The ${master_token} in the example is referring to the master token associated to a repository. See Master Tokens.
NOTE: The installation scripts provided by packagecloud automatically generate read tokens for private repositories when they're executed. Take a look at the installation instructions for any packagecloud repository.
UNIQUE_ID=`hostname -f` && curl -XPOST --data "name=${UNIQUE_ID}" https://${master_token}:@packagecloud.io/install/repositories/${username}/${repository}/tokens.text
Continuous Integration
CircleCI
To push packages to packagecloud from CircleCI you must:
- Set an environment variable named
PACKAGECLOUD_TOKEN in your project's settings and the value must match your packagecloud API access token.
- Create a circle.yml file which installs the package_cloud gem and pushes the package to the OS and version of your choice.
For more info on your API access token, please refer to the API Tokens section in API docs.
Questions? We're here to help! Email support@packagecloud.io
Just an example; get your real API token here.
PACKAGECLOUD_TOKEN: f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0
Example circle.yml file
dependencies:
pre:
- gem install package_cloud
deployment:
production:
branch: master
commands:
- package_cloud push user/repo/os/version /path/to/pkg.ext
Travis CI
To push packages to packagecloud from Travis CI you must:
- Set an environment variable named
PACKAGECLOUD_TOKEN in your project's settings and the value must match your packagecloud API access token.
- Create a .travis.yml file which installs the package_cloud gem and pushes the package to the OS and version of your choice.
For more info on your API access token, please refer to the API Tokens section in API docs.
Questions? We're here to help! Email support@packagecloud.io
Just an example; get your real API token here.
PACKAGECLOUD_TOKEN: f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0
Example .travis.yml file
before_install:
- gem install package_cloud
script:
- package_cloud push user/repo/os/version /path/to/package.ext
Jenkins
To push packages to packagecloud from Jenkins you'll need to install and configure the packagecloud Jenkins plugin.
The step by step instructions can be found on the packagecloud Jenkins plugin wiki page.
There are 3 steps to the configuration process:
- Plugin installation: begin by finding and installing the Jenkins plugin via the Jenkins UI
- Credentials setup: next, use Jenkins' credentials system to store your packagecloud username and API token
- Configure post-build job: finally, add a post-build step to your Jenkins job, enter the repository to push to, and select the operating system and version
You'll need to use your packagecloud username and API token when configuring credentials in Jenkins.
Please log in to packagecloud to view your username and API token.
Questions? We're here to help! Email support@packagecloud.io
Buildkite
To push packages to packagecloud from Buildkite you must:
- Ensure the package_cloud gem is installed in your agent environment
- Expose the
PACKAGECLOUD_TOKEN environment variable in your Buildkite environment hook
- Add a build script to push your build artifact to packagecloud
For more info on your API access token, please refer to the API Tokens section in API docs.
Questions? We're here to help! Email support@packagecloud.io
Just an example; get your real API token here.
#!/bin/bash
export PACKAGECLOUD_TOKEN=f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0
Example build script that pushes a previously built build artifact to packagecloud:
#!/bin/bash
set -e
if [[ $# -eq 0 ]]; then
echo "Usage: $0 username/myrepo path/to/package/artifact"
exit 1
fi
# Download package artifact from a previous build step
buildkite-agent artifact download "$1" .
# Push package to packagecloud
package_cloud push "$2" "$(basename $1)"
Example build pipeline:
Command Line Client
Installation
Once you get the package_cloud CLI tool installed, you can create repositories and push packages quickly and easily.
- If you don't already have Ruby, you need to install it. You can find install instructions for most platforms here.
- Install the package_cloud Ruby gem by running:
> sudo gem install package_cloud
The package_cloud CLI has built-in help that can be accessed by running:
> package_cloud help
Creating repos
There are two ways to create a new repo:
- Using the web UI: https://packagecloud.io/repositories/new
- Using the package_cloud CLI:
> package_cloud repository create myrepo
If your plan allows you to create private repos, you can create one by:
- Selecting Private in the web UI on the repo create page: https://packagecloud.io/repositories/new
- Passing a flag to the package_cloud CLI:
> package_cloud repository create myrepo --private
Pushing packages
You can push a package using the package_cloud CLI:
> package_cloud push username/myrepo /path/to/packagefile
Some package types can be associated to specific versions of particular Linux distributions. If the package type you are pushing needs to be associated with a particular version you will be prompted by the package_cloud CLI.
You can also specify the version in the command line, for example, to push a package to Ubuntu 12.04 Precise, you would use:
> package_cloud push username/reponame/ubuntu/precise /path/to/packagefile
Supplying/Overriding Maven Coordinates
Sometimes (like for "fatjars"), we are unable to automatically detect the maven coordinates for a particular package, so you'll have to supply it on the command line like so:
> package_cloud push username/reponame/java /path/to/package.jar --coordinates=com.mygroup:packagename:1.0.2
Yanking packages (Delete a package)
You can remove a package from a repo by using the package_cloud CLI:
> package_cloud yank username/myrepo/[distro/version] packagename.ext
The packagename.ext must be the full filename of the package, including the extension.
Some packages (RPMs, Debs, Python packages, and JARs) are pushed to a specific distribution and version. When those packages are removed, the distribution and version that should be removed must be specified.
Other package types (RubyGems) can be removed without specifying a distribution or version, as these do not apply to RubyGem packages.
Examples
Remove a package called jake-1.0-2.src.rpm from a CentOS 6 repo named myrepo:
> package_cloud yank username/myrepo/el/6 jake-1.0-2.src.rpm
Remove a package called redis_2.8.1-1_amd64.deb from an Ubuntu Precise (12.04) repo named myrepo:
> package_cloud yank username/myrepo/ubuntu/precise redis_2.8.1-1_amd64.deb
Remove a package called packagecloud_test-0.0.1.whl from a Python repo named myrepo:
> package_cloud yank username/myrepo/python/1 packagecloud_test-0.0.1.whl
Remove a package called passenger-4.0.43.gem from a RubyGem repo named myrepo:
> package_cloud yank username/myrepo passenger-4.0.43.gem
Promoting packages (Move a package)
You can move a package to a different epo by using the package_cloud CLI:
> package_cloud promote username/myrepo/[distro/version] packagename.ext username/destination_repo
The packagename.ext must be the full filename of the package, including the extension.
Some packages (RPMs, Debs, Python packages, and JARs) are pushed to a specific distribution and version. When those packages are moved, the distribution/version must be specified and they will be moved to the same distribution/version in the destination repository.
Other package types (RubyGems) can be moved without specifying a distribution or version, as these do not apply to RubyGem packages.
Both repository owners and collaborators may move packages between repositories.
This operation is a move, not a copy, thus the source repository will no longer have a copy of the package when the operation has completed. Note that the download statistics
for the moved package will be cleared.
Examples
Move a packaged called jake-1.0-2.src.rpm from a CentOS 6 repo name myrepo to repo2:
> package_cloud promote username/myrepo/el/6 jake-1.0-2.src.rpm username/repo2
Move a package called redis_2.8.1-1_amd64.deb from an Ubuntu Precise (12.04) repo named myrepo to repo2:
> package_cloud promote username/myrepo/ubuntu/precise redis_2.8.1-1_amd64.deb username/repo2
Move a package called packagecloud_test-0.0.1.whl from a Python repo named myrepo to repo2:
> package_cloud promote username/myrepo/python/1 packagecloud_test-0.0.1.whl username/repo2
Move a package called passenger-4.0.43.gem from a RubyGem repo named myrepo:
> package_cloud promote username/myrepo passenger-4.0.43.gem username/repo2
Installing a repo
There are two methods you can use to install a repo:
- You can visit the repo page and click the installation tab on the left side to get Bash scripts, a Chef cookbook, a Puppet module, and manual installation instructions.
- You can also use the package_cloud command line client. You will need to specify the type of repo you want installed on your system (deb, rpm, or gem). For example, to install a deb repo you would run:
> package_cloud repository install username/reponame deb
OS versions for pushing and yanking packages
When pushing or yanking a package you can specify an OS and a version. If you do not specify one, the command line client will
walk you through choosing the correct OS and version.
The following table shows the OS and version strings that are valid for pushing and yanking:
RubyGems
No OS or version is required. RubyGem packages may be pushed like this:
package_cloud push user/repo example-1.0.1.gem
Python
Python wheels, eggs, and source distributons can be pushed by specifying python. Python packages may be pushed like this:
package_cloud push user/repo/python example-1.0.1.whl
Amazon Linux
Packages for Amazon Linux must use the string for Enterprise Linux 6 (el/6) when pushing, or yanking, from a packagecloud repository.
package_cloud push user/repo/el/6 amz-linux-package-1.0-2.el6.x86_64.rpm
package_cloud yank user/repo/el/6 amz-linux-package-1.0-2.el6.x86_64.rpm
{{key}}
| Version |
Push/Yank string |
|
{{ item.os_dist }} |
Example: pushing a package to {{ value[0]["display_name"] }}:
package_cloud push user/repo/{{value[0]["os_dist"]}} {{ useRpmExampleFile(value[0]["package_type"]) ? "test-1.0-2.el6.x86_64.rpm" : "testpkg_1.0-2_amd64.deb" }}
Example: yanking a package from {{ value[0]["display_name"] }}:
package_cloud yank user/repo/{{value[0]["os_dist"]}} {{ useRpmExampleFile(value[0]["package_type"]) ? "test-1.0-2.el6.x86_64.rpm" : "testpkg_1.0-2_amd64.deb" }}
When and how am I billed?
You are billed the price for your plan at the start of the month. At the end of the month, your bandwidth and storage watermark calcuated. Any overage amounts will be added to the bill that you receive at the start of the next month.
How can I view my current or previous usage information?
Your current bandwidth usage and storage watermark is available in the usage section of account settings. You'll also find previous months' usage underneath, sorted by date.
Will I get notified when I'm nearing my limits?
Yes. We'll notify you when you've reached 50%, 90% and 99% of either your bandwidth, storage or maximum overage amount. You can turn these off in usage under account settings.
How exactly is bandwidth usage calculated?
Uploads to packagecloud are free.
Bandwidth usage for downloads is updated every 5 minutes. We examine our request logs and attribute any bytes sent from packagecloud to the corresponding repository owner for the requested object. Conditional GET's, HEAD requests or cancelled requests aren't counted. You are only charged for any outgoing bytes.
What is a 'storage watermark' and how is it calculated?
Every time a package is pushed to, or yanked from packagecloud, we credit or debit a storage usages table for that user and repository, with the time that action took place. Then, we sum up all the bytes that were pushed and come up with a high watermark for any given billing period. Thus, if you upload 100GB of packages, and delete 90GB of them, you'll still have used 100GB for that billing period, and 10G for the next.
How can I control my usage/overage cost?
In the usage section of account settings, you can put the maximum overage amount you'd like to pay before the account gets disabled.
Note: Due to how bandwidth is calculated, there is a small window of about an hour between exceeding your maximum overage and us disabling the account. For high traffic repositories, you might need to budget accordingly. For example: if you send 10GB of bandwidth an hour, on a plan with 30cents/GB for overage and have set $1 as your maximum overage, you might end up paying $3. (30 cents/GB * 10GB).
My account just got disabled! How do I re-enable it?
Accounts are automatically re-enabled when: you have entered a new billing period, you have upgraded your plan, or you have increased the maximum overage allowed for your account.
Can I use packagecloud for free?
Yes, we offer a limited Free Plan and also offer 14 day trials of any of our paid plans.
Do your prices include taxes?
Yes.
We can't pay by credit card. Can you invoice my company?
We can invoice your company, but we'll still need a credit card on file to bill for overages. Contact support@packagecloud.io and we'll work with you to figure this out.
I think my usage will be way above your largest plan, can we get on custom plan?
Absolutely. Please contact support@packagecloud.io and we'll get you going.
What are the paid plan overage prices?
The following are the overage prices for the paid plans on packagecloud.io.
| $20/month |
$50/month |
$100/month |
$500/month |
| 22¢ per GB of Storage |
20¢ per GB of Storage |
18¢ per GB of Storage |
15¢ per GB of Storage |
| 65¢ per GB of Bandwidth |
60¢ per GB of Bandwidth |
55¢ per GB of Bandwidth |
45¢ per GB of Bandwidth |
How do plan upgrades work with different overage prices?
Overage rates for usage are calculated at the end of your billing cycle. This means that it's possible to get cheaper overages (or no overages at all) by upgrading to a larger plan before your billing cycle ends.