Welcome to Lago’s documentation!

Lago is an add-hoc virtual testing environment framework

Getting started

Getting started

Hello, this describes how to get started with Lago.

Installation

In order to install the framework, you’ll need to build RPMs or acquire them from a repository.

Latest lago RPMs are built by jenkins job and you can find them in the ci jobs:

http://jenkins.ovirt.org/job/lago_master_build-artifacts-$DIST-x86_64

Where $DIST is either el7, fc21, fc22 or fc23 (this list might be outdated, take a look at the repo to see the supported distros).

Or you can use the yum repo (it’s updated often right now, and a bit unstable), you can add it as a repository creating a file under /etc/yum.repos.d/lago.repo with the following content:

For Fedora:

[lago]
baseurl=http://resources.ovirt.org/repos/lago/stable/0.0/rpm/fc$releasever
name=Lago
enabled=1
gpgcheck=0

For EL distros (such as CentOS, RHEL, etc.):

[lago]
baseurl=http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el$releasever
name=Lago
enabled=1
gpgcheck=0

TODO: point to the release rpm once it’s implemented, and use gpgcheck=1

Once you have them, install the following packages:

$ yum install python-lago lago python-lago-ovirt lago-ovirt

This will install all the needed packages.

TODO:explain each package contents and goals

Machine set-up

Virtualization and nested virtualization support

1. Make sure that virtualization extension is enabled on the CPU, otherwise, you might need to enable it in the BIOS. Generally, if virtualization extension is disabled, dmesg log would contain a line similar to:

   kvm: disabled by BIOS
   

   NOTE: you can wait until everyithing is setup to reboot and change the bios, to make sure that everyithing will persist after reboot

2. To make sure that nested virtualization is enabled, run:

   $ cat /sys/module/kvm_intel/parameters/nested
   

   This command should print Y if nested virtualization is enabled, otherwise, enable it the following way:

3. Edit /etc/modprobe.d/kvm-intel.conf and add the following line:

   options kvm-intel nested=y
   

4. Reboot, and make sure nested virtualization is enabled.

libvirt

Make sure libvirt is configured to run:

$ systemctl enable libvirtd
$ systemctl start libvirtd

SELinux

At the moment, this framework might encounter problems running while SELinux policy is enforced.

To disable SELinux on the running system, run:

$ setenforce 0

To disable SELinux from start-up, edit /etc/selinux/config and set:

SELINUX=permissive

User setup

Running lago requires certain permissions, so the user running it should be part of certain groups.

Add yourself to lago and qemu groups:

$ usermod -a -G lago USERNAME
$ usermod -a -G qemu USERNAME

It is also advised to add qemu user to your group (to be able to store VM files in home directory):

$ usermod -a -G USERNAME qemu

For the group changes to take place, you’ll need to re-login to the shell. Make sure running id returns all the aforementioned groups.

Make sure that the qemu user has execution rights to the dir where you will be creating the prefixes, you can try it out with:

$ sudo -u qemu ls /path/to/the/destination/dir

If it can’t access it, make sure that all the dirs in the path have your user or qemu groups and execution rights for the group, or execution rights for other (highly recommended to use the group instead, if the dir did not have execution rights for others already)

It’s very common for the user home directory to not have group execution rights, to make sure you can just run:

$ chmod g+x $HOME

And, just to be sure, let’s refresh libvirtd service to ensure that it refreshes it’s permissions and picks up any newly created users:

$ sudo service libvirtd restart

NOTE: if you just added your user, make sure to restart libvirtd service

Preparing the workspace

Create a directory where you’ll be working, make sure qemu user can access it.

We will be using the example configurations of lago, for a custom setup you might want to create your own.

Running lago

This tests require that you have at least 36GB of free space under the /var/lib/lago directory and an extra 200MB wherever you are running them.

If you don’t have enough disk space on /var (for e.g, a default fedora install only has 20G), you can change the default path for downloading the images and repos on the lago.conf file. You can change the default values from:

$ cat /etc/lago.d/lago.conf
[lago]
log_level = debug
template_store = /var/lib/lago/store
template_repos = /var/lib/lago/repos
default_root_password = 123456

to use your homedir, for e.g:

$ vim /etc/lago.d/lago.conf
[lago]
log_level = debug
template_store = /home/USERNAME/lago/store
template_repos = /home/USERNAME/lago/repos
default_root_password = 123456

As an example, we will use the basic suite of the ovirt tests, so we have to download them, you can run the following to get a copy of the repository:

$ git clone git://gerrit.ovirt.org/ovirt-system-tests

As the tests that we are going to run are for ovirt-engine 3.5, we have to add the oVirt 3.5 release repository to our system so it will pull in the sdk package, the following works for any centos/fedora distro:

$ yum install -y http://resources.ovirt.org/pub/yum-repo/ovirt-release35.rpm

Once you have the code and the repo, you can run the run_suite.sh script to run any of the suites available (right now, only 3.5 and 3.6 basic_suites are fully working):

$ cd ovirt-system-tests
$ ./run_suite.sh basic_suite_3.5

NOTE: this will download a lot of vm images the first time it runs, check the section “template-repo.json: Sources for templates” on how to use local mirrors if available.

Remember that you don’t need root access to run it, if you have permission issues, make sure you followed the guidelines in the section “user setup” above

This will take a while, as first time execution downloads a lot of stuff, like downloading OS templates, where each one takes at least 1G of data. If you are still worried that its stuck, please refer to the FAQ to see if the issue you’re seeing is documented.

Once it is done, you will get the results in the directory deployment-basic_suite_3.5, that will include an initialized prefix with a 3.5 engine vm with all the hosts and storages added.

To access it, log in to the web-ui at

• URL: https://192.168.200.2/
• Username: admin@internal
• Password: 123

If you’re running the framework on a remote machine, you can tunnel a local port directly to the destination machine:

$ ssh -L 8443:192.168.200.2:443 remote-user@remote-ip
        ---- =================             ~~~~~~~~~
        (*)   (**)                         (***)

(*)   - The port on localhost that the tunnel will be available at.
(**)  - The destination where the remote machine will connect when local machine
        connects to the local end of the tunnel.
(***) - Remote machine through which we'll connect to the remote end of the
        tunnel.

After creating the tunnel, web-ui will be available at https://localhost:8443/

Poke around in the env

You can now open a shell to any of the vms, start/stop them all, etc.:

$ cd deployment-basic_suite_3.5
$ lagocli shell engine
[root@engine ~]# exit

$ lagocli stop
2015-11-03 12:11:52,746 - root - INFO - Destroying VM engine
2015-11-03 12:11:52,957 - root - INFO - Destroying VM storage-iscsi
2015-11-03 12:11:53,167 - root - INFO - Destroying VM storage-nfs
2015-11-03 12:11:53,376 - root - INFO - Destroying VM host3
2015-11-03 12:11:53,585 - root - INFO - Destroying VM host2
2015-11-03 12:11:53,793 - root - INFO - Destroying VM host1
2015-11-03 12:11:54,002 - root - INFO - Destroying VM host0
2015-11-03 12:11:54,210 - root - INFO - Destroying network lago

$ lagocli start
2015-11-03 12:11:46,377 - root - INFO - Creating network lago
2015-11-03 12:11:46,712 - root - INFO - Starting VM engine
2015-11-03 12:11:47,261 - root - INFO - Starting VM storage-iscsi
2015-11-03 12:11:47,726 - root - INFO - Starting VM storage-nfs
2015-11-03 12:11:48,115 - root - INFO - Starting VM host3
2015-11-03 12:11:48,573 - root - INFO - Starting VM host2
2015-11-03 12:11:48,937 - root - INFO - Starting VM host1
2015-11-03 12:11:49,296 - root - INFO - Starting VM host0

Cleanup

Once you’re done with the environment, run:

$ cd deployment-basic_suite_3.5
$ lagocli cleanup

That will stop any running vms and remove the lago metadata in the prefix, it will not remove any other files (like disk images) or anything though, so you can play with them for further investigation if needed, but once executed, it’s safe to fully remove the prefix dir if you want to.

Step by step now

As the above script has become a bit complicated, and it’s not (yet) part of lago itself, this section will do the same as the script, but step by step with lago only command to give you a better idea of what you have to do in a usual project.

So, let’s get back to the root of the ovirt-system-tests repo, and cd into the basic_suite_3.5 dir:

cd ovirt-system-tests/basic_suite_3.5

Let’s take a look to what is in there:

$ tree
.
├── control.sh
├── deploy-scripts
│   ├── add_local_repo.sh
│   ├── bz_1195882_libvirt_workaround.sh
│   ├── setup_container_host.sh
│   ├── setup_engine.sh
│   ├── setup_host.sh
│   ├── setup_storage_iscsi.sh
│   └── setup_storage_nfs.sh
├── engine-answer-file.conf
├── init.json.in
├── reposync-config.repo
├── template-repo.json
└── test-scenarios
    ├── 001_initialize_engine.py
    ├── 002_bootstrap.py
    ├── 003_create_clean_snapshot.py
    └── 004_basic_sanity.py

We can ignore the control.sh script, as it’s used by the run_suite.sh and we don’t care about that in this readme.

init.json.in: The heart of lago, virt configurations

This init.json.in file, is where we will describe all the virtual elements of our test environment, usually, vms and networks.

In this case, as the file is shared between suites, it’s actually a template and we will have to change the @SUITE@ string inside it by the path to the current suite:

$ suite_path=$PWD
$ sed -e "s/@SUITE@/$suite_path/g" init.json.in > init.json

Now we have a full init.json file :), but we have to talk about another file before being able to create the prefix:

Note that lago supports json and yaml formats for that file.

template-repo.json: Sources for templates

This file contains information about the available disk templates and repositories to get them from, we can use it as it is, but if you are in Red Hat office in Israel, you might want to use the Red Hat internal mirrors there, for that use the common/template-repos/office.json file instead, see next for the full command line.

NOTE: You can use any other template repo if you specify your own json file there

TODO: document the repo store json file format

Initializing the prefix

Now we have seen all the files needed to initialize our test prefix (aka, the directory that will contain our env). To do so we have to run this:

$ lagocli init \
    --template-repo-path=template-repo.json \
    deployment-basic_suite_3.5 \
    init.json

Remember that if you are in the Red Hat office, you might want to use the repo mirror that’s hosted there, if so, run this command instead:

$ lagocli init \
    --template-repo-path=common/template-repos/office.json \
    deployment-basic_suite_3.5 \
    init.json

This will create the deployment-basic_suite_3.5 directory and populate it with all the disks defined in the init.json file, and some other info (network info, uuid... not relevant now).

This will take a while the first time, but the next time it will use locally cached images and will take only a few seconds!

If you are using run_suite.sh

To use an alternate repository template file when running run_suite.sh, you’ll have to edit it for now, search for the init command invocation and modify it there, at the time of writing this, if you want to use the Red Hat Israel office mirror, you have to change this:

38 env_init () {
39     $CLI init \
40         $PREFIX \
41         $SUITE/init.json \
42         --template-repo-path $SUITE/template-repo.json
43 }

by:

env_init () {
    $CLI init \
        $PREFIX \
        $SUITE/init.json \
        --template-repo-path common/template-repos/office.json
}

reposync-config.repo: yum repositories to make available to the vms

This file contains a valid yum repos definition, it’s the list of all the yum repos that will be enabled on the vms to pull from. If you want to use any custom repos just add the yum repo entry of your choice there and it will be make accessible to the vms.

The internal repository is built from one or several ‘sources’, there are 2 types of sources:

• External RPM repositories:

  A yum .repo file can be passed to the verb, and all the included repositories will be downloaded using ‘reposync’ and added to the internal repo.

This is used by the ovirt reposetup verb. To prefetch and generate the local repo, we have to run it:

$ lagocli ovirt reposetup --reposync-yum-config="reposync-config.repo"

This might take a while the first time too, as it has to fetch a few rpms from a few repos, next time it will also use a cache to speed things up considerably.

NOTE: From now on, all the lagocli command will be run inside the prefix, so cd to it:

$ cd deployment-basic_suite_3.5

Bring up the virtual resources

We are ready to start powering up vms!

# make sure you are in the prefix
$ pwd
/path/to/ovirt-system-tests/deployment-basic_suite_3.5
$ lagocli start

This starts all resources (VMs, bridges), at any time, you can use the stop verb to stop all active resources.

Run oVirt initial setup scripts

Once all of our vms and network are up and running, we have to run any setup scripts that will configure oVirt in the machines, as we already described in the init.json what scripts should be executed, the only thing left is to trigger it:

$ lagocli ovirt deploy

This should be relatively fast, around a minute or two, for everything to get installed and configured

Running the tests

Okok, so now we have our environment ready for the tests!! \o/

Lets get it on, remember that they should be executed in order:

$ lagocli ovirt runtest 001_initialize_engine.py
...
$ lagocli ovirt runtest 002_bootstrap.py
...
$ lagocli ovirt runtest 003_create_clean_snapshot.py
...
$ lagocli ovirt runtest 004_basic_sanity.py
...

This tests run a simple test suite on the environment:

• Create a new DC and cluster
• Deploy all the hosts
• Add storage domains
• Import templates

The tests are written in python and interact with the environment using the python SDK.

Collect the logs

So now we want to collect all the logs from the vms, to troubleshoot and debug if needed (or just to see if they show what we expect). To do so, you can just:

$ lagocli ovirt collect \
    --output "test_logs"

We can run that command anytime, you can run it in between the tests also, specifying different output directories if you want to see the logs during the process or compare later with the logs once the tests finish.

You can see all the logs now in the dir we specified:

$ tree test_logs
test_logs/
├── engine
│   └── _var_log_ovirt-engine
│       ├── boot.log
│       ├── console.log
│       ├── dump
│       ├── engine.log
│       ├── host-deploy
│       ├── notifier
│       ├── ovirt-image-uploader
│       ├── ovirt-iso-uploader
│       ├── server.log
│       └── setup
│           └── ovirt-engine-setup-20151029122052-7g9q2k.log
├── host0
│   └── _var_log_vdsm
│       ├── backup
│       ├── connectivity.log
│       ├── mom.log
│       ├── supervdsm.log
│       ├── upgrade.log
│       └── vdsm.log
├── host1
│   └── _var_log_vdsm
│       ├── backup
│       ├── connectivity.log
│       ├── mom.log
│       ├── supervdsm.log
│       ├── upgrade.log
│       └── vdsm.log
├── host2
│   └── _var_log_vdsm
│       ├── backup
│       ├── connectivity.log
│       ├── mom.log
│       ├── supervdsm.log
│       ├── upgrade.log
│       └── vdsm.log
├── host3
│   └── _var_log_vdsm
│       ├── backup
│       ├── connectivity.log
│       ├── mom.log
│       ├── supervdsm.log
│       ├── upgrade.log
│       └── vdsm.log
├── storage-iscsi
└── storage-nfs

Cleaning up

As before, once you have finished playing with the prefix, you will want to clean it up (remember to play around!), to do so just:

$ lagocli cleanup

FAQ

1. How do I know if the run_suite.sh is stuck or still running?

   Sometimes the script is downloading very big files which might Seem to someone as the script is stuck. One hacky way of making sure the script is still working is to check the size and content of the store dir:

   $ ls -la /var/lib/lago/store
   

   This will show any templates being downloaded and file size changes.

Developing

CI Process

Here is described the usual workflow of going through the CI process from starting a new branch to getting it merged and released in the unstable repo.

Starting a branch

First of all, when starting to work on a new feature or fix, you have to start a new branch (in your fork if you don’t have push rights to the main repo). Make sure that your branch is up to date with the project’s master:

git checkout -b my_fancy_feature
# in case that origin is already lago-project/lago
git reset --hard origin/master

Then, once you can just start working, doing commits to that branch, and pushing to the remote from time to time as a backup.

Once you are ready to run the ci tests, you can create a pull request to master branch, if you have hub installed you can do so from command line, if not use the ui:

$ hub pull-request

That will automatically trigger a test run on ci, you’ll see the status of the run in the pull request page. At that point, you can keep working on your branch, probably just rebasing on master regularly and maybe amending/squashing commits so they are logically meaningful.

A clean commit history

An example of not good pull request history:

• Added right_now parameter to virt.VM.start function
• Merged master into my_fancy_feature
• Added tests for the new parameter case
• Renamed right_now parameter to sudo_right_now
• Merged master into my_fancy_feature
• Adapted test to the rename

This history can be greatly improved if you squashed a few commits:

• Added sudo_right_now parameter to virt.VM.start function
• Added tests for the new parameter case
• Merged master into my_fancy_feature
• Merged master into my_fancy_feature

And even more if instead of merging master, you just rebased:

• Added sudo_right_now parameter to virt.VM.start function
• Added tests for the new parameter case

That looks like a meaningful history :)

Rerunning the tests

While working on your branch, you might want to rerun the tests at some point, to do so, you just have to add a new comment to the pull request with one of the following as content:

• ci test please
• ci :+1:
• ci :thumbsup:

Asking for reviews

If at any point, you see that you are not getting reviews, please add the label ‘needs review’ to flag that pull request as ready for review.

Getting the pull request merged

Once the pull request has been reviewed and passes all the tests, an admin can start the merge process by adding a comment with one of the following as content:

• ci merge please
• ci :shipit:

That will trigger the merge pipeline, that will run the tests on the merge commit and deploy the artifacts to the unstable repo on success.

Environment setup

Here are some guidelines on how to set up your development of the lago project.

Requirements

You’ll need some extra packages to get started with the code for lago, assuming you are runnig Fedora:

> sudo dnf install git mock libvirt-daemon qemu-kvm autotools

And you’ll need also a few Python libs, which you can install from the repos or use venv or similar, for the sake of this example we will use the repos ones:

> sudo dnf install python-flake8 python-nose python-dulwich yapf

Yapf is not available on older Fedoras or CentOS, you can get it from the official yapf repo or try on copr.

Now you are ready to get the code:

> git clone git@github.com:lago-project/lago.git

From now on all the commands will be based from the root of the cloned repo:

> cd lago

Style formatting

We will accept only patches that pass pep8 and that are formatted with yapf. More specifically, only patches that pass the local tests:

> make check-local

It’s recommended that you setup your editor to check automatically for pep8 issues. For the yapf formatting, if you don’t want to forget about it, you can install the pre-commit git hook that comes with the project code:

> ln -s scripts/pre-commit.style .git/pre-commit

Now each time that you run git commit it will automatically reformat the code you changed with yapf so you don’t have any issues when submitting a patch.

Testing your changes

Once you do some changes, you should make sure they pass the checks, there’s no need to run on each edition but before submitting a patch for review you should do it.

You can run them on your local machine, but the tests themselves will install packages and do some changes to the os, so it’s really recommmended that you use a vm, or as we do on the CI server, use mock chroots. If you don’t want to setup mock, skip the next section.

Hopefully in a close future we can use lago for that ;)

Setting up mock_runner.sh with mock (fedora)

For now we are using a script developed by the oVirt devels to generate chroots and run tests inside them, it’s not packaged yet, so we must get the code itself:

> cd ..
> git clone git://gerrit.ovirt.org/jenkins

As an alternative, you can just download the script and install them in your $PATH:

> wget https://gerrit.ovirt.org/gitweb?p=jenkins.git;a=blob_plain;f=mock_configs/mock_runner.sh;hb=refs/heads/master

We will need some extra packages:

> sudo dnf install mock

And, if not running as root (you shouldn’t!) you have to add your user to the newly created mock group, and make sure the current session is in that group:

> sudo usermod -a -G mock $USER
> newgrp mock
> id  # check that mock is listed

Running the tests inside mock

Now we have all the setup we needed, so we can go back to the lago repo and run the tests, the first time you run them, it will take a while to download all the required packages and install them in the chroot, but on consecutive runs it will reuse all the cached chroots.

The mock_runner.sh script allows us to test also different distributions, any that is supported by mock, for example, to run the tests for fedora 23 you can run:

> ../jenkins/mock_runner.sh -p fc23

That will run all the check-patch.sh (the -p option) tests inside a chroot, with a minimal fedora 23 installation. It will leave any logs under the logs directory and any generated artifacts under exported-artifacts.

Getting started developing

Everyone is welcome to send patches to lago, but we know that not everybody knows everything, so here’s a reference list of technologies and methodologies that lago uses for reference.

Python!

Lago is written in python 2.7 (for now), so you should get yourself used to basic-to-medium python constructs and technics like:

• Basic python: Built-in types, flow control, pythonisms (import this)
• Object oriented programming (OOP) in python: Magic methods, class inheritance

Some useful resources:

• Base docs: https://docs.python.org/2.7/

• Built-in types: https://docs.python.org/2.7/library/stdtypes.html

• About classes: https://docs.python.org/2.7/reference/datamodel.html#new-style-and-classic-classes

• The Zen of Python:

  > python -c "import this"
  
  The Zen of Python, by Tim Peters
  
  Beautiful is better than ugly.
  Explicit is better than implicit.
  Simple is better than complex.
  Complex is better than complicated.
  Flat is better than nested.
  Sparse is better than dense.
  Readability counts.
  Special cases aren't special enough to break the rules.
  Although practicality beats purity.
  Errors should never pass silently.
  Unless explicitly silenced.
  In the face of ambiguity, refuse the temptation to guess.
  There should be one-- and preferably only one --obvious way to do it.
  Although that way may not be obvious at first unless you're Dutch.
  Now is better than never.
  Although never is often better than *right* now.
  If the implementation is hard to explain, it's a bad idea.
  If the implementation is easy to explain, it may be a good idea.
  Namespaces are one honking great idea -- let's do more of those!
  

Bash

Even though there is not much bash code, the functional tests and some support scripts use it, so better to get some basics on it. We will try to follow the same standards for it than the oVirt project has.

Libvirt + qemu/kvm

As we are using intesively libvirt and qemu/kvm, it’s a good idea to get yourself familiar with the main commands and services:

• libvirt: http://libvirt.org
• virsh client: http://libvirt.org/virshcmdref.html
• qemu (qemu-img is useful to deal with vm disk images): https://en.wikibooks.org/wiki/QEMU/Images

Also, there’s a library and a set of tools from the libguestfs project that we use to prepare templates and are very useful when debugging, make sure you play at least with virt-builder, virt-customize, virt-sparsify and guestmount.

Git + Github

We use git as code version system, and we host it on Github right now, so if you are not familiar with any of those tools, you should get started with them, specially you should be able to:

• Clone a repo from github
• Fork a repo from github
• Create/delete/move to branches (git checkout)
• Move to different points in git history (git reset)
• Create/delete tags (git tag)
• See the history (git log)
• Create/amend commits (git commit)
• Retrieve changes from the upstream repository (git fetch)
• Apply your changes on top of the retrieved ones (git rebase)
• Apply your changes as a merge commit (git merge)
• Squash/reorder existing commits (git rebase –interactive)
• Send your changes to the upstream (git push)
• Create a pull request

You can always go to the git docs though there is a lot of good literature on it too.

Unit tests with py.test

Lately we decided to use py.test for the unit tests, and all the current unit tests were migrated to it. We encourage adding unit tests to any pull requests you send.

Functional tests with bats

For the functional tests, we decided to use bats framework. It’s completely written in bash, and if you are modifying or adding any functionality, you should add/modify those tests accordingly. It has a couple of custom constructs, so take a look to the bats docs while reading/writing tests.

Packaging

Our preferred distribution vector is though packages. Right now we are only building for rpm-based system, so right now you can just take a peek on how to build rpms. Keep in mind also that we try to move as much of the packaging logic as posible to the python packaging system itself too, worth getting used to it too.

Where to go next

You can continue setting up your environment and try running the examples in the readme to get used to lago. Once you get familiar with it, you can pick any of the existing issues and send a pull request to fix it, so you get used to the ci process we use to get stuff developed flawlessly and quickly, welcome!

Contents

lago package

Subpackages

lago.plugins package

lago.plugins.PLUGIN_ENTRY_POINTS = {'cli': 'lago.plugins.cli', 'out': 'lago.plugins.output'}

Map of plugin type string -> setuptools entry point

class lago.plugins.Plugin

Bases: object

Base class for all the plugins

lago.plugins.load_plugins(namespace)

Loads all the plugins for the given namespace

Parameters:namespace (str) – Namespace string, as in the setuptools entry_points Returns:Returns the list of loaded plugins already instantiated Return type:dict of str, object

Submodules

lago.plugins.cli module

About CLIPlugins

A CLIPlugin is a subcommand of the lagocli command, it’s ment to group actions together in a logical sense, for example grouping all the actions done to templates.

To create a new subcommand for testenvcli you just have to subclass the CLIPlugin abstract class and declare it in the setuptools as an entry_point, see this module’s setup.py/setup.cfg for an example:

class NoopCLIplugin(CLIPlugin):
    init_args = {
        'help': 'dummy help string',
    }

    def populate_parser(self, parser):
        parser.addArgument('--dummy-flag', action='store_true')

    def do_run(self, args):
        if args.dummy_flag:
            print "Dummy flag passed to noop subcommand!"
        else:
            print "Dummy flag not passed to noop subcommand!"

You can also use decorators instead, an equivalent is:

@cli_plugin_add_argument('--dummy-flag', action='store_true')
@cli_plugin(help='dummy help string')
def my_fancy_plugin_func(dummy_flag, **kwargs):
    if dummy_flag:
        print "Dummy flag passed to noop subcommand!"
    else:
        print "Dummy flag not passed to noop subcommand!"

Or:

@cli_plugin_add_argument('--dummy-flag', action='store_true')
def my_fancy_plugin_func(dummy_flag, **kwargs):
    "dummy help string"
    if dummy_flag:
        print "Dummy flag passed to noop subcommand!"
    else:
        print "Dummy flag not passed to noop subcommand!"

Then you will need to add an entry_points section in your setup.py like:

setup(
    ...
    entry_points={
        'lago.plugins.cli': [
            'noop=noop_module:my_fancy_plugin_func',
        ],
    }
    ...
)

Or in your setup.cfg like:

[entry_points]
lago.plugins.cli =
    noop=noop_module:my_fancy_plugin_func

Any of those will add a new subcommand to the lagocli command that can be run as:

$ lagocli noop
Dummy flag not passed to noop subcommand!

TODO: Allow per-plugin namespacing to get rid of the **kwargs parameter

class lago.plugins.cli.CLIPlugin

Bases: lago.plugins.Plugin

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 25

_abc_registry = <_weakrefset.WeakSet object>

do_run(args)

Execute any actions given the arguments

Parameters:args (Namespace) – with the arguments Returns:None

init_args

Dictionary with the argument to initialize the cli parser (for example, the help argument)

populate_parser(parser)

Add any required arguments to the parser

Parameters:parser (ArgumentParser) – parser to add the arguments to Returns:None

class lago.plugins.cli.CLIPluginFuncWrapper(do_run=None, init_args=None)

Bases: lago.plugins.cli.CLIPlugin

Special class to handle decorated cli plugins, take into account that the decorated functions have some limitations on what arguments can they define actually, if you need something complicated, used the abstract class CLIPlugin instead.

Keep in mind that right now the decorated function must use **kwargs as param, as it will be passed all the members of the parser, not just whatever it defined

__call__(*args, **kwargs)

Keep the original function interface, so it can be used elsewhere

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 25

_abc_registry = <_weakrefset.WeakSet object>

add_argument(*argument_args, **argument_kwargs)

do_run(args)

init_args

populate_parser(parser)

set_help(help=None)

set_init_args(init_args)

lago.plugins.cli.cli_plugin(func=None, **kwargs)

Decorator that wraps the given function in a CLIPlugin

Parameters:

• func (callable) – function/class to decorate
• **kwargs – Any other arg to use when initializing the parser (like help, or prefix_chars)

Returns:

cli plugin that handles that method

Return type:

CLIPlugin

Notes

It can be used as a decorator or as a decorator generator, if used as a decorator generator don’t pass any parameters

Examples

>>> @cli_plugin
... def test(**kwargs):
...     print 'test'
...
>>> print test.__class__
<class 'cli.CLIPluginFuncWrapper'>

>>> @cli_plugin()
... def test(**kwargs):
...     print 'test'
>>> print test.__class__
<class 'cli.CLIPluginFuncWrapper'>

>>> @cli_plugin(help='dummy help')
... def test(**kwargs):
...     print 'test'
>>> print test.__class__
<class 'cli.CLIPluginFuncWrapper'>
>>> print test.init_args['help']
'dummy help'

lago.plugins.cli.cli_plugin_add_argument(*args, **kwargs)

Decorator generator that adds an argument to the cli plugin based on the decorated function

Parameters:

• *args – Any args to be passed to argparse.ArgumentParser.add_argument()
• *kwargs – Any keyword args to be passed to argparse.ArgumentParser.add_argument()

Returns:

Decorator that builds or extends the cliplugin for the decorated function, adding the given argument definition

Return type:

function

Examples

>>> @cli_plugin_add_argument('-m', '--mogambo', action='store_true')
... def test(**kwargs):
...     print 'test'
...
>>> print test.__class__
<class 'cli.CLIPluginFuncWrapper'>
>>> print test._parser_args
[(('-m', '--mogambo'), {'action': 'store_true'})]

>>> @cli_plugin_add_argument('-m', '--mogambo', action='store_true')
... @cli_plugin_add_argument('-b', '--bogabmo', action='store_false')
... @cli_plugin
... def test(**kwargs):
...     print 'test'
...
>>> print test.__class__
<class 'cli.CLIPluginFuncWrapper'>
>>> print test._parser_args 
[(('-b', '--bogabmo'), {'action': 'store_false'}),
 (('-m', '--mogambo'), {'action': 'store_true'})]

lago.plugins.cli.cli_plugin_add_help(help)

Decorator generator that adds the cli help to the cli plugin based on the decorated function

Parameters:help (str) – help string for the cli plugin Returns:Decorator that builds or extends the cliplugin for the decorated function, setting the given help Return type:function

Examples

>>> @cli_plugin_add_help('my help string')
... def test(**kwargs):
...     print 'test'
...
>>> print test.__class__
<class 'cli.CLIPluginFuncWrapper'>
>>> print test.help
my help string

>>> @cli_plugin_add_help('my help string')
... @cli_plugin()
... def test(**kwargs):
...     print 'test'
>>> print test.__class__
<class 'cli.CLIPluginFuncWrapper'>
>>> print test.help
my help string

lago.plugins.output module

About OutFormatPlugins

An OutFormatPlugin is used to format the output of the commands that extract information from the perfixes, like status.

class lago.plugins.output.DefaultOutFormatPlugin

Bases: lago.plugins.output.OutFormatPlugin

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 25

_abc_registry = <_weakrefset.WeakSet object>

format(info_obj, indent='')

indent_unit = ' '

class lago.plugins.output.JSONOutFormatPlugin

Bases: lago.plugins.output.OutFormatPlugin

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 25

_abc_registry = <_weakrefset.WeakSet object>

format(info_dict)

class lago.plugins.output.OutFormatPlugin

Bases: lago.plugins.Plugin

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 25

_abc_registry = <_weakrefset.WeakSet object>

format(info_dict)

Execute any actions given the arguments

Parameters:info_dict (dict) – information to reformat Returns:String representing the formatted info Return type:str

class lago.plugins.output.YAMLOutFormatPlugin

Bases: lago.plugins.output.OutFormatPlugin

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 25

_abc_registry = <_weakrefset.WeakSet object>

format(info_dict)

Submodules

lago.brctl module

lago.brctl._brctl(command, *args)

lago.brctl._set_link(name, state)

lago.brctl.create(name, stp=True)

lago.brctl.destroy(name)

lago.brctl.exists(name)

lago.cmd module

lago.cmd.check_group_membership()

lago.cmd.create_parser(cli_plugins, out_plugins)

lago.cmd.main()

lago.config module

lago.config._get_environ()

lago.config._get_from_dir(path, key)

lago.config._get_from_env(key)

lago.config._get_from_files(paths, key)

lago.config._get_providers()

lago.config.get(key, default=<object object>)

lago.constants module

lago.dirlock module

lago.dirlock._lock_path(path)

lago.dirlock.lock(path, excl, key_path)

Waits until the given directory can be locked

Parameters:

• path (str) – Path of the directory to lock
• excl (bool) – If the lock should be exclusive
• key_path (str) – path to the file that contains the uid to use when locking

Returns:

None

lago.dirlock.trylock(path, excl, key_path)

Tries once to get a lock to the given dir

Parameters:

• path (str) – path to the directory to lock
• excl (bool) – If the lock should be exclusive
• key_path (str) – path to the file that contains the uid to use when locking

Returns:

True if it did get a lock, False otherwise

Return type:

bool

lago.dirlock.unlock(path, key_path)

Removes the lock of the uid in the given key file

Parameters:

• path (str) – Path of the directory to lock
• key_path (str) – path to the file that contains the uid to remove the lock of

Returns:

None

lago.libvirt_utils module

Utilities to help deal with the libvirt python bindings

lago.libvirt_utils.DOMAIN_STATES = {<class 'sphinx.ext.autodoc.VIR_DOMAIN_SHUTDOWN'>: 'beign shut down', <class 'sphinx.ext.autodoc.VIR_DOMAIN_CRASHED'>: 'crashed', <class 'sphinx.ext.autodoc.VIR_DOMAIN_PAUSED'>: 'paused', <class 'sphinx.ext.autodoc.VIR_DOMAIN_RUNNING'>: 'running', <class 'sphinx.ext.autodoc.VIR_DOMAIN_BLOCKED'>: 'blocked', <class 'sphinx.ext.autodoc.VIR_DOMAIN_NOSTATE'>: 'no state', <class 'sphinx.ext.autodoc.VIR_DOMAIN_SHUTOFF'>: 'shut off', <class 'sphinx.ext.autodoc.VIR_DOMAIN_PMSUSPENDED'>: 'suspended'}

Mapping of domain statuses values to human readable strings

class lago.libvirt_utils.Domain

Bases: object

Class to namespace libvirt domain related helpers

static resolve_state(state_number)

Get a nice description from a domain state number

Parameters:state_number (list of int) – State number as returned by libvirt.virDomain.state() Returns:small human readable description of the domain state, unknown if the state is not in the known list Return type:str

lago.log_utils module

This module defines the special logging tools that lago uses

lago.log_utils.ALWAYS_SHOW_REG = <_sre.SRE_Pattern object>

Regexp that will match the above template

lago.log_utils.ALWAYS_SHOW_TRIGGER_MSG = 'force-show:%s'

Message template that will always shoud the messago

class lago.log_utils.ColorFormatter(fmt=None, datefmt=None)

Bases: logging.Formatter

Formatter to add colors to log records

CRITICAL = '\x1b[31m'

CYAN = '\x1b[36m'

DEBUG = ''

DEFAULT = '\x1b[0m'

ERROR = '\x1b[31m'

GREEN = '\x1b[32m'

INFO = '\x1b[36m'

NONE = ''

RED = '\x1b[31m'

WARNING = '\x1b[33m'

WHITE = '\x1b[37m'

YELLOW = '\x1b[33m'

classmethod colored(color, message)

Small function to wrap a string around a color

Parameters:

• color (str) – name of the color to wrap the string with, must be one of the class properties
• message (str) – String to wrap with the color

Returns:

the colored string

Return type:

str

format(record)

Adds colors to a log record and formats it with the default

Parameters:record (logging.LogRecord) – log record to format Returns:The colored and formatted record string Return type:str

class lago.log_utils.ContextLock

Bases: object

Context manager to thread lock a block of code

lago.log_utils.END_TASK_MSG = 'Success'

Message to be shown when a task is ended

lago.log_utils.END_TASK_REG = <_sre.SRE_Pattern object>

Regexp that will match the above template

lago.log_utils.END_TASK_TRIGGER_MSG = 'end task %s'

Message template that will trigger a task end

class lago.log_utils.LogTask(task, logger=<module 'logging' from '/usr/lib/python2.7/logging/__init__.pyc'>, level='info', propagate_fail=True)

Bases: object

Context manager for a log task

Example

>>> with LogTask('mytask'):
...     pass

lago.log_utils.START_TASK_MSG = ''

Message to be shown when a task is started

lago.log_utils.START_TASK_REG = <_sre.SRE_Pattern object>

Regexp that will match the above template

lago.log_utils.START_TASK_TRIGGER_MSG = 'start task %s'

Message template that will trigger a task

class lago.log_utils.Task(name, *args, **kwargs)

Bases: collections.deque

Small wrapper around deque to add the failed status and name to a task

name

str

name for this task

failed

bool

If this task has failed or not (if there was any error log shown during it’s execution)

force_show

bool

If set, will show any log records generated inside this task even if it’s out of nested depth limit

elapsed_time()

class lago.log_utils.TaskHandler(initial_depth=0, task_tree_depth=-1, buffer_size=2000, dump_level=40, level=0, formatter=<class 'lago.log_utils.ColorFormatter'>)

Bases: logging.StreamHandler

This log handler will use the concept of tasks, to hide logs, and will show all the logs for the current task if there’s a logged error while running that task.

It will hide any logs that belong to nested tasks that have more than task_tree_depth parent levels, and for the ones that are above that level, it will show only the logs that have a loglevel above level.

You can force showing a log record immediately if you use the log_always() function bypassing all the filters.

If there’s a log record with log level higher than dump_level it will be considered a failure, and all the logs for the current task that have a log level above level will be shown no matter at which depth the task belongs to. Also, all the parent tasks will be tagged as error.

formatter

logging.LogFormatter

formatter to use

initial_depth

int

Initial depth to account for, in case this handler was created in a subtask

tasks_by_thread (dict of str

OrderedDict of str: Task): List of thread names, and their currently open tasks with their latest log records

dump_level

int

log level from which to consider a log record as error

buffer_size

int

Size of the log record deque for each task, the bigger, the more records it can show in case of error but the more memory it will use

task_tree_depth

int

number of the nested level to show start/end task logs for, if -1 will show always

level

int

Log level to show logs from if the depth limit is not reached

main_failed

bool

used to flag from a child thread that the main should fail any current task

_tasks_lock

ContextLock

Lock for the tasks_by_thread dict

_main_thread_lock

ContextLock

Lock for the main_failed bool

TASK_INDICATORS = ['@', '#', '*', '-', '~']

List of chars to show as task prefix, to ease distinguishing them

am_i_main_thread

Returns:if the current thread is the main thread Return type:bool

close_children_tasks(parent_task_name)

Closes all the children tasks that were open

Parameters:parent_task_name (str) – Name of the parent task Returns:None

cur_depth_level

Returns:depth level for the current task Return type:int

cur_task

Returns:the current active task Return type:str

cur_thread

Returns:Name of the current thread Return type:str

emit(record)

Handle the given record, this is the entry point from the python logging facility

Params:

record (logging.LogRecord): log record to handle

Returns:None

get_task_indicator(task_level=None)

Parameters:task_level (int or None) – task depth level to get the indicator for, if None, will use the current tasks depth Returns:char to prepend to the task logs to indicate it’s level Return type:str

get_tasks(thread_name)

Parameters:thread_name (str) – name of the thread to get the tasks for Returns:list of task names and log records for each for the given thread Return type:OrderedDict of str, Task

handle_closed_task(task_name, record)

Do everything needed when a task is closed

Params:

task_name (str): name of the task that is finishing record (logging.LogRecord): log record with all the info

Returns:None

handle_error()

Handles an error log record that should be shown

Returns:None

handle_new_task(task_name, record)

Do everything needed when a task is starting

Params:

task_name (str): name of the task that is starting record (logging.LogRecord): log record with all the info

Returns:None

mark_main_tasks_as_failed()

Flags to the main thread that all it’s tasks sholud fail

Returns:None

mark_parent_tasks_as_failed(task_name, flush_logs=False)

Marks all the parent tasks as failed

Parameters:

• task_name (str) – Name of the child task
• flush_logs (bool) – If True will discard all the logs form parent tasks

Returns:

None

pretty_emit(record, is_header=False, task_level=None)

Wrapper around the logging.StreamHandler emit method to add some decoration stuff to the message

Parameters:

• record (logging.LogRecord) – log record to emit
• is_header (bool) – if this record is a header, usually, a start or end task message
• task_level (int) – If passed, will take that as the current nested task level instead of calculating it from the current tasks

Returns:

None

should_show_by_depth(cur_level=None)

Parameters:cur_level (int) – depth level to take into account Returns:True if the given depth level should show messages (not taking into account the log level) Return type:bool

should_show_by_level(record_level, base_level=None)

Parameters:

• record_level (int) – log level of the record to check
• base_level (int or None) – log level to check against, will use the object’s dump_level if None is passed

Returns:

True if the given log record should be shown according to the log level

Return type:

bool

tasks

Returns:list of task names and log records for each for the current thread Return type:OrderedDict of str, Task

lago.log_utils.end_log_task(task, logger=<module 'logging' from '/usr/lib/python2.7/logging/__init__.pyc'>, level='info')

Ends a log task

Parameters:

• task (str) – name of the log task to end
• logger (logging.Logger) – logger to use
• level (str) – log level to use

Returns:

None

lago.log_utils.hide_paramiko_logs()

lago.log_utils.log_always(message)

Wraps the given message with a tag that will make it be always logged by the task logger

Parameters:message (str) – message to wrap with the tag Returns:tagged message that will get it shown immediately by the task logger Return type:str

lago.log_utils.log_task(task, logger=<module 'logging' from '/usr/lib/python2.7/logging/__init__.pyc'>, level='info', propagate_fail=True)

Parameterized decorator to wrap a function in a log task

Example

>>> @log_task('mytask')
... def do_something():
...     pass

lago.log_utils.setup_prefix_logging(logdir)

Sets up a file logger that will create a log in the given logdir (usually a lago prefix)

Parameters:logdir (str) – path to create the log into, will be created if it does not exist Returns:None

lago.log_utils.start_log_task(task, logger=<module 'logging' from '/usr/lib/python2.7/logging/__init__.pyc'>, level='info')

Starts a log task

Parameters:

• task (str) – name of the log task to start
• logger (logging.Logger) – logger to use
• level (str) – log level to use

Returns:

None

lago.paths module

class lago.paths.Paths(prefix)

Bases: object

images(*path)

logs()

metadata()

prefix_lagofile()

This file represents a prefix that’s initialized

prefixed(*args)

scripts(*args)

ssh_id_rsa()

ssh_id_rsa_pub()

uuid()

virt(*path)

lago.prefix module

class lago.prefix.Prefix(prefix)

Bases: object

A prefix is a directory that will contain all the data needed to setup the environment.

_prefix

str

Path to the directory of this prefix

_paths

lago.path.Paths

Path handler class

_virt_env

lago.virt.VirtEnv

Lazily loaded virtual env handler

_metadata

dict

Lazily loaded metadata

_add_nic_to_mapping(net, dom, nic)

Populates the given net spec mapping entry with the nicks of the given domain

Parameters:

• net (dict) – Network spec to populate
• dom (dict) – libvirt domain specification
• nic (str) – Name of the interface to add to the net mapping from the domain

Returns:

None

_allocate_ips_to_nics(conf)

For all the nics of all the domains in the conf that have dynamic ip, allocate one and addit to the network mapping

Parameters:conf (dict) – Configuration spec to extract the domains from Returns:None

_allocate_subnets(conf)

Allocate all the subnets needed by the given configuration spec

Parameters:conf (dict) – Configuration spec where to get the nets definitions from Returns:tuple – allocated subnets and modified conf Return type:list, dict

static _check_predefined_subnets(conf)

Checks if all of the nets defined in the config are inside the allowed range, throws exception if not

Parameters:conf (dict) – Configuration spec where to get the nets definitions from Returns:None Raises:RuntimeError – If there are any subnets out of the allowed range

_config_net_topology(conf)

Initialize and populate all the network related elements, like reserving ips and populating network specs of the given confiiguration spec

Parameters:conf (dict) – Configuration spec to initalize Returns:None

_copy_delpoy_scripts(scripts)

Copy the given deploy scripts to the scripts dir in the prefix

Parameters:scripts (list of str) – list of paths of the scripts to copy to the prefix Returns:list with the paths to the copied scripts, with a prefixed with $LAGO_PREFIX_PATH so the full path is not hardcoded Return type:list of str

_copy_deploy_scripts_for_hosts(domains)

Copy the deploy scripts for all the domains into the prefix scripts dir

Parameters:domains (dict) – spec with the domains info as when loaded from the initfile Returns:None

_create_disk(name, spec, template_repo=None, template_store=None)

Creates a disc with the given name from the given repo or store

Parameters:

• name (str) – Name of the domain to create the disk for
• spec (dict) – Specification of the disk to create
• template_repo (TemplateRepository or None) – template repo instance to use
• template_store (TemplateStore or None) – template store instance to use

Returns:

Tuple – Path to the disk and disk metadata

Return type:

str, dict

Raises:

RuntimeError – If the type of the disk is not supported or failed to create the disk

_create_disks(domain_name, disks_specs, template_repo, template_store)

_create_ssh_keys()

Generate a pair of ssh keys for this prefix

Returns:None Raises:RuntimeError – if it fails to create the keys

_create_virt_env()

Create a new virt env from this prefix

Returns:virt env created from this prefix Return type:lago.virt.VirtEnv

_deploy_host(host)

static _generate_disk_name(host_name, disk_name, disk_format)

_generate_disk_path(disk_name)

_get_metadata()

Retrieve the metadata info for this prefix

Returns:metadata info Return type:dict

_get_scripts(host_metadata)

Temporary method to retrieve the host scripts

TODO:

remove once the “ovirt-scripts” option gets deprecated

Parameters:host_metadata (dict) – host metadata to retrieve the scripts for Returns:deploy scripts for the host, empty if none found Return type:list

_handle_empty_disk(host_name, disk_spec)

_handle_file_disk(disk_spec)

_handle_lago_template(disk_path, template_spec, template_store, template_repo)

_handle_ova_image(domain_spec)

_handle_qcow_template(disk_path, template_spec)

_handle_template(host_name, template_spec, template_store=None, template_repo=None)

static _init_net_specs(conf)

Given a configuration specification, initializes all the net definitions in it so they can be used comfortably

Parameters:conf (dict) – Configuration specification Returns:the adapted new conf Return type:dict

_ova_to_spec(filename)

Retrieve the given ova and makes a template of it. Creates a disk from network provided ova. Calculates the needed memory from the ovf. The disk will be cached in the template repo

Parameters:filename (str) – the url to retrive the data from

TODO:

• Add hash checking against the server for faster download and latest version
• Add config script running on host - other place
• Add cloud init support - by using cdroms in other place
• Handle cpu in some way - some other place need to pick it up
• Handle the memory units properly - we just assume MegaBytes

Returns:

list with the disk specification int: VM memory, None if none defined int: Number of virtual cpus, None if none defined

Return type:

list of dict

Raises:

• RuntimeError – If the ova format is not supported
• TypeError – If the memory units in the ova are noot supported (currently only ‘MegaBytes’)

_prepare_domain_image(domain_spec, prototypes, template_repo, template_store)

_prepare_domains_images(conf, template_repo, template_store)

_register_preallocated_ips(conf)

Parse all the domains in the given conf and preallocate all their ips into the networks mappings, raising exception on duplicated ips or ips out of the allowed ranges

See also

lago.subnet_lease

Parameters:conf (dict) – Configuration spec to parse Returns:None Raises:RuntimeError – if there are any duplicated ips or any ip out of the allowed range

_retrieve_disk_url(disk_url, disk_dst_path=None)

static _run_qemu(qemu_cmd, disk_path)

_save_metadata()

Write this prefix metadata to disk

Returns:None

_set_scripts(host_metadata, scripts)

Temporary method to set the host scripts

TODO:

remove once the “ovirt-scripts” option gets deprecated

Parameters:host_metadata (dict) – host metadata to set scripts in Returns:the updated metadata Return type:dict

_use_prototype(spec, prototypes)

Populates the given spec with the values of it’s declared prototype

Parameters:

• spec (dict) – spec to update
• prototypes (dict) – Configuration spec containing the prototypes

Returns:

updated spec

Return type:

dict

cleanup(*args, **kwargs)

Stops any running entities in the prefix and uninitializes it, usually you want to do this if you are going to remove the prefix afterwards

Returns:None

collect_artifacts(*args, **kwargs)

create_snapshots(name)

Creates one snapshot on all the domains with the given name

Parameters:name (str) – Name of the snapshots to create Returns:None

deploy(*args, **kwargs)

destroy()

Destroy this prefix, running any cleanups and removing any files inside it.

fetch_url(url)

Retrieves the given url to the prefix

Parameters:url (str) – Url to retrieve Returns:path to the downloaded file Return type:str

get_nets()

Retrieve info on all the nets from all the domains

Returns:dict of str->list – dictionary with net_name -> net list Return type:str

get_snapshots()

Retrieve info on all the snapshots from all the domains

Returns:list(str): dictionary with vm_name -> snapshot list Return type:dict of str

get_vms()

Retrieve info on all the vms

Returns:dict of str->list – dictionary with vm_name -> vm list Return type:str

initialize(*args, **kwargs)

Initialize this prefix, this includes creating the destination path, and creating the uuid for the prefix, for any other actions see Prefix.virt_conf()

Will safely roll back if any of those steps fail

Returns:None Raises:RuntimeError – If it fails to create the prefix dir

classmethod is_prefix(path)

Check if a path is a valid prefix

Parameters:path (str) – path to be checked Returns:True if the given path is a prefix Return type:bool

classmethod resolve_prefix_path(start_path=None)

Look for an existing prefix in the given path, in a path/.lago dir, or in a .lago dir under any of it’s parent directories

Parameters:start_path (str) – path to start the search from, if None passed, it will use the current dir Returns:path to the found prefix Return type:str Raises:RuntimeError – if no prefix was found

revert_snapshots(name)

Revert all the snapshots with the given name from all the domains

Parameters:name (str) – Name of the snapshots to revert Returns:None

save()

Save this prefix to persistent storage

Returns:None

start(vm_names=None)

Start this prefix

Parameters:vm_names (list of str) – List of the vms to start Returns:None

stop(vm_names=None)

Stop this prefix

Parameters:vm_names (list of str) – List of the vms to stop Returns:None

virt_conf(conf, template_repo=None, template_store=None, do_bootstrap=True)

Initializes all the virt infrastructure of the prefix, creating the domains disks, doing any network leases and creating all the virt related files and dirs inside this prefix.

Parameters:

• conf (dict) – Configuration spec
• template_repo (TemplateRepository) – template repository intance
• template_store (TemplateStore) – template store instance

Returns:

None

virt_conf_from_stream(conf_fd, template_repo=None, template_store=None, do_bootstrap=True)

Initializes all the virt infrastructure of the prefix, creating the domains disks, doing any network leases and creating all the virt related files and dirs inside this prefix.

Parameters:

• conf_fd (File) – File like object to read the config from
• template_repo (TemplateRepository) – template repository intance
• template_store (TemplateStore) – template store instance

Returns:

None

virt_env

Getter for this instance’s virt env, creates it if needed

Returns:virt env instance used by this prefix Return type:lago.virt.VirtEnv

lago.prefix._create_ip(subnet, index)

Given a subnet or an ip and an index returns the ip with that lower index from the subnet (255.255.255.0 mask only subnets)

Parameters:

• subnet (str) – Strign containing the three first elements of the decimal representation of a subnet (X.Y.Z) or a full ip (X.Y.Z.A)
• index (int or str) – Last element of a decimal ip representation, for example, 123 for the ip 1.2.3.123

Returns:

The dotted decimal representation of the ip

Return type:

str

lago.prefix._ip_in_subnet(subnet, ip)

Checks if an ip is included in a subnet.

Note

only 255.255.255.0 masks allowed

Parameters:

• subnet (str) – Strign containing the three first elements of the decimal representation of a subnet (X.Y.Z) or a full ip (X.Y.Z.A)
• ip (str or int) – Decimal ip representation

Returns:

True if ip is in subnet, False otherwise

Return type:

bool

lago.subnet_lease module

Module that handles the leases for the subnets of the virtual network interfaces.

Note

Currently only /24 ranges are handled, and all of them under the 192.168.MIN_SUBNET to 192.168.MAX_SUBNET ranges

The leases are stored under LEASE_DIR as json files with the form:

[
    "/path/to/prefix/uuid/file",
    "uuid_hash",
]

Where the uuid_hash is the 32 char uuid of the prefix (the contents of the uuid file at the time of doing the lease)

lago.subnet_lease.LEASE_DIR = '/var/lib/lago/subnets/'

Path to the directory where the net leases are stored

lago.subnet_lease.LOCK_FILE = '/var/lib/lago/subnets/leases.lock'

Path to the net leases lock

lago.subnet_lease.MAX_SUBNET = 209

Upper range for the allowed subnets

lago.subnet_lease.MIN_SUBNET = 200

Lower range for the allowed subnets

lago.subnet_lease._acquire(*args, **kwargs)

Lease a free network for the given uuid path

Parameters:uuid_path (str) – Path to the uuid file of a lago.Prefix Returns:the third element of the dotted ip of the leased network or None if no lease was available Return type:int or None

Todo

Raise exception or something instead of returning None so the caller can handle the failure case

lago.subnet_lease._lease_owned(path, current_uuid_path)

Checks if the given lease is owned by the prefix whose uuid is in the given path

Note

The prefix must be also in the same path it was when it took the lease

Parameters:

• path (str) – Path to the lease
• current_uuid_path (str) – Path to the uuid to check ownersip of

Returns:

True if the given lease in owned by the prefix, False otherwise

Return type:

bool

lago.subnet_lease._lease_valid(path)

Checs if the given lease still has a prefix that owns it

Parameters:path (str) – Path to the lease Returns:True if the uuid path in the lease still exists and is the same as the one in the lease Return type:bool

lago.subnet_lease._locked(func)

Decorator that will make sure that you have the exclusive lock for the leases

lago.subnet_lease._release(*args, **kwargs)

Free the lease of the given subnet index

Parameters:index (int) – Third element of a dotted ip representation of the subnet, for example, for 1.2.3.4 it would be 3 Returns:None

lago.subnet_lease._take_lease(path, uuid_path)

Persist to the given leases path the prefix uuid that’s in the uuid path passed

Parameters:

• path (str) – Path to the leases file
• uuid_path (str) – Path to the prefix uuid

Returns:

None

lago.subnet_lease._validate_lease_dir_present(func)

Decorator that will ensure that the lease dir exists, creating it if necessary

lago.subnet_lease.acquire(uuid_path)

Lease a free network for the given uuid path

Parameters:uuid_path (str) – Path to the uuid file of a lago.Prefix Returns:the dotted ip of the gateway for the leased net Return type:str

Todo

_aquire might return None, this will throw a TypeError

lago.subnet_lease.is_leasable_subnet(subnet)

Checks if a given subnet is inside the defined provisionable range

Parameters:subnet (str) – Subnet or ip in dotted decimal format Returns:True if subnet is inside the range, False otherwise Return type:bool

lago.subnet_lease.release(subnet)

Free the lease of the given subnet

Parameters:subnet (str) – dotted ip or network to free the lease of Returns:None

lago.sysprep module

lago.sysprep._config_net_interface(iface, **kwargs)

lago.sysprep._upload_file(local_path, remote_path)

lago.sysprep._write_file(path, content)

lago.sysprep.add_ssh_key(key, with_restorecon_fix=False)

lago.sysprep.config_net_interface_dhcp(iface, hwaddr)

lago.sysprep.set_hostname(hostname)

lago.sysprep.set_iscsi_initiator_name(name)

lago.sysprep.set_root_password(password)

lago.sysprep.set_selinux_mode(mode)

lago.sysprep.sysprep(disk, mods, backend='direct')

lago.templates module

This module contains any disk template related classes and functions, including the repository store manager classes and template providers, some useful definitions:

• Template repositories:

  Repository where to fetch templates from, as an http server

• Template store:

  Local store to cache templates

• Template:

  Unititialized disk image to use as base for other disk images

• Template version:

  Specific version of a template, to allow getting updates without having to change the template name everywhere

class lago.templates.FileSystemTemplateProvider(root)

Handles file type templates, that is, getting a disk template from the host’s filesystem

_prefixed(*path)

Join all the given paths prefixed with this provider’s base root path

Parameters:*path (str) – sections of the path to join, passed as positional arguments Returns:Joined paths prepended with the provider root path Return type:str

download_image(handle, dest)

Copies over the handl to the destination

Parameters:

• handle (str) – path to copy over
• dest (str) – path to copy to

Returns:

None

get_hash(handle)

Returns the associated hash for the given handle, the hash file must exist (handle + '.hash').

Parameters:handle (str) – Path to the template to get the hash from Returns:Hash for the given handle Return type:str

get_metadata(handle)

Returns the associated metadata info for the given handle, the metadata file must exist (handle + '.metadata').

Parameters:handle (str) – Path to the template to get the metadata from Returns:Metadata for the given handle Return type:dict

class lago.templates.HttpTemplateProvider(baseurl)

This provider allows the usage of http urls for templates

download_image(handle, dest)

Downloads the image from the http server

Parameters:

• handle (str) – url from the self.baseurl to the remote template
• dest (str) – Path to store the downloaded url to, must be a file path

Returns:

None

static extract_if_needed(path)

get_hash(handle)

Get the associated hash for the given handle, the hash file must exist (handle + '.hash').

Parameters:handle (str) – Path to the template to get the hash from Returns:Hash for the given handle Return type:str

get_metadata(handle)

Returns the associated metadata info for the given handle, the metadata file must exist (handle + '.metadata'). If the given handle has an .xz extension, it will get removed when calculating the handle metadata path

Parameters:handle (str) – Path to the template to get the metadata from Returns:Metadata for the given handle Return type:dict

open_url(url, suffix='', dest=None)

Opens the given url, trying the compressed version first. The compressed version url is generated adding the .xz extension to the url and adding the given suffix after that .xz extension. If dest passed, it will download the data to that path if able

Parameters:

• url (str) – relative url from the self.baseurl to retrieve
• suffix (str) – optional suffix to append to the url after adding the compressed extension to the path
• dest (str or None) – Path to save the data to

Returns:

response object to read from (lazy read), closed if no dest passed

Return type:

urllib.addinfourl

Raises:

RuntimeError – if the url gave http error when retrieving it

class lago.templates.Template(name, versions)

Disk image template class

name

str

Name of this template

_versions (dict(str

TemplateVersion)): versions for this template

get_latest_version()

Retrieves the latest version for this template, the latest being the one with the newest timestamp

Returns:TemplateVersion

get_version(ver_name=None)

Get the given version for this template, or the latest

Parameters:ver_name (str or None) – Version to retieve, None for the latest Returns:The version matching the given name or the latest one Return type:TemplateVersion

class lago.templates.TemplateRepository(dom)

A template repository is a single source for templates, that uses different providers to actually retrieve them. That means for example that the ‘ovirt’ template repository, could support the ‘http’ and a theoretical ‘gluster’ template providers.

_dom

dict

Specification of the template

_providers

dict

Providers instances for any source in the spec

_get_provider(spec)

Get the provider for the given template spec

Parameters:spec (dict) – Template spec Returns:A provider instance for that spec Return type:HttpTemplateProvider or FileSystemTemplateProvider

classmethod from_url(path)

Instantiate a TemplateRepository instance from the data in a file or url

Parameters:path (str) – Path or url to the json file to load Returns:A new instance Return type:TemplateRepository

get_by_name(name)

Retrieve a template by it’s name

Parameters:name (str) – Name of the template to retrieve Raises:KeyError – if no template is found

name

Getter for the template repo name

Returns:the name of this template repo Return type:str

class lago.templates.TemplateStore(path)

Local cache to store templates

The store uses various files to keep track of the templates cached, access and versions. An example template store looks like:

$ tree /var/lib/lago/store/
/var/lib/lago/store/
├── in_office_repo:centos6_engine:v2.tmp
├── in_office_repo:centos7_engine:v5.tmp
├── in_office_repo:fedora22_host:v2.tmp
├── phx_repo:centos6_engine:v2
├── phx_repo:centos6_engine:v2.hash
├── phx_repo:centos6_engine:v2.metadata
├── phx_repo:centos6_engine:v2.users
├── phx_repo:centos7_engine:v4.tmp
├── phx_repo:centos7_host:v4.tmp
└── phx_repo:storage-nfs:v1.tmp

There you can see the files:

• *.tmp

  Temporary file created while downloading the template from the repository (depends on the provider)

• ${repo_name}:${template_name}:${template_version}

  This file is the actual disk image template

• *.hash

  Cached hash for the template disk image

• *.metadata

  Metadata for this template image in json format, usually this includes the distro and root-password

__contains__(temp_ver)

Checks if a given version is in this store

Parameters:temp_ver (TemplateVersion) – Version to look for Returns:True if the version is in this store Return type:bool

_init_users(temp_ver)

Initializes the user access registry

Parameters:temp_ver (TemplateVersion) – template version to update registry for Returns:None

_prefixed(*path)

Join the given paths and prepend this stores path

Parameters:*path (str) – list of paths to join, as positional arguments Returns:all the paths joined and prepended with the store path Return type:str

download(temp_ver, store_metadata=True)

Retrieve the given template version

Parameters:

• temp_ver (TemplateVersion) – template version to retrieve
• store_metadata (bool) – If set to False, will not refresh the local metadata with the retrieved one

Returns:

None

get_path(temp_ver)

Get the path of the given version in this store

Parameters:TemplateVersion (temp_ver) – version to look for Returns:The path to the template version inside the store Return type:str Raises:RuntimeError – if the template is not in the store

get_stored_hash(temp_ver)

Retrieves the hash for the given template version from the store

Parameters:temp_ver (TemplateVersion) – template version to retrieve the hash for Returns:hash of the given template version Return type:str

get_stored_metadata(temp_ver)

Retrieves the metadata for the given template version from the store

Parameters:temp_ver (TemplateVersion) – template version to retrieve the metadata for Returns:the metadata of the given template version Return type:dict

mark_used(temp_ver, key_path)

Adds or updates the user entry in the user access log for the given template version

Parameters:

• temp_ver (TemplateVersion) – template version to add the entry for
• key_path (str) – Path to the prefix uuid file to set the mark for

class lago.templates.TemplateVersion(name, source, handle, timestamp)

Each template can have multiple versions, each of those is actually a different disk template file representation, under the same base name.

download(destination)

Retrieves this template to the destination file

Parameters:destination (str) – file path to write this template to Returns:None

get_hash()

Returns the associated hash for this template version

Returns:Hash for this version Return type:str

get_metadata()

Returns the associated metadata info for this template version

Returns:Metadata for this version Return type:dict

timestamp()

Getter for the timestamp

lago.templates._locked(func)

Decorator that ensures that the decorated function has the lock of the repo while running, meant to decorate only bound functions for classes that have lock_path method.

lago.templates.find_repo_by_name(name, repo_dir=None)

Searches the given repo name inside the repo_dir (will use the config value ‘template_repos’ if no repo dir passed), will rise an exception if not found

Parameters:

• name (str) – Name of the repo to search
• repo_dir (str) – Directory where to search the repo

Returns:

path to the repo

Return type:

str

Raises:

RuntimeError – if not found

lago.utils module

class lago.utils.CommandStatus

Bases: lago.utils.CommandStatus

class lago.utils.EggTimer(timeout)

elapsed()

class lago.utils.ExceptionTimer(timeout)

Bases: object

start()

stop()

class lago.utils.LockFile(path, timeout=None, **kwargs)

Bases: object

Context manager that creates a lock around a directory, with optional timeout in the acquire operation

Parameters:

• path (str) – path to the dir to lock
• timeout (int) – timeout in seconds to wait while acquiring the lock
• **kwargs (dict) – Any other param to pass to lockfile.LockFile

__enter__()

Start the lock with timeout if needed in the acquire operation

Raises:TimerException – if the timeout is reached before acquiring the lock

class lago.utils.RollbackContext(*args)

Bases: object

A context manager for recording and playing rollback. The first exception will be remembered and re-raised after rollback

Sample usage: > with RollbackContext() as rollback: > step1() > rollback.prependDefer(lambda: undo step1) > def undoStep2(arg): pass > step2() > rollback.prependDefer(undoStep2, arg)

More examples see tests/utilsTests.py @ vdsm code

__exit__(exc_type, exc_value, traceback)

If this function doesn’t return True (or raises a different exception), python re-raises the original exception once this function is finished.

clear()

defer(func, *args, **kwargs)

prependDefer(func, *args, **kwargs)

exception lago.utils.TimerException

Bases: exceptions.Exception

Exception to throw when a timeout is reached

class lago.utils.VectorThread(targets)

join_all(raise_exceptions=True)

start_all()

lago.utils._CommandStatus

alias of CommandStatus

lago.utils._read_nonblocking(f)

lago.utils._ret_via_queue(func, queue)

lago.utils._run_command(command, input_data=None, stdin=None, out_pipe=-1, err_pipe=-1, env=None, **kwargs)

Runs a command

Parameters:

• command (list of str) – args of the command to execute, including the command itself as command[0] as [‘ls’, ‘-l’]
• input_data (str) – If passed, will feed that data to the subprocess through stdin
• out_pipe (int or file) – File descriptor as passed to :ref:subprocess.Popen to use as stdout
• stdin (int or file) – File descriptor as passed to :ref:subprocess.Popen to use as stdin
• err_pipe (int or file) – File descriptor as passed to :ref:subprocess.Popen to use as stderr
• of str (env(dict) – str): If set, will use the given dict as env for the subprocess
• **kwargs – Any other keyword args passed will be passed to the :ref:subprocess.Popen call

Returns:

result of the interactive execution

Return type:

lago.utils.CommandStatus

lago.utils.add_timestamp_suffix(base_string)

lago.utils.deepcopy(original_obj)

Creates a deep copy of an object with no crossed referenced lists or dicts, useful when loading from yaml as anchors generate those cross-referenced dicts and lists

Parameters:original_obj (object) – Object to deep copy Returns:deep copy of the object Return type:object

lago.utils.drain_ssh_channel(chan, stdin=None, stdout=<open file '<stdout>', mode 'w'>, stderr=<open file '<stderr>', mode 'w'>)

lago.utils.func_vector(target, args_sequence)

lago.utils.in_prefix(prefix_class, workdir_class)

lago.utils.interactive_ssh_channel(chan, command=None, stdin=<open file '<stdin>', mode 'r'>)

lago.utils.invoke_in_parallel(func, *args_sequences)

lago.utils.json_dump(obj, f)

lago.utils.load_virt_stream(virt_fd)

Loads the given conf stream into a dict, trying different formats if needed

Parameters:virt_fd (str) – file like objcect with the virt config to load Returns:Loaded virt config Return type:dict

lago.utils.rotate_dir(base_dir)

lago.utils.run_command(command, input_data=None, out_pipe=-1, err_pipe=-1, env=None, **kwargs)

Runs a command non-interactively

Parameters:

• command (list of str) – args of the command to execute, including the command itself as command[0] as [‘ls’, ‘-l’]
• input_data (str) – If passed, will feed that data to the subprocess through stdin
• out_pipe (int or file) – File descriptor as passed to :ref:subprocess.Popen to use as stdout
• err_pipe (int or file) – File descriptor as passed to :ref:subprocess.Popen to use as stderr
• of str (env(dict) – str): If set, will use the given dict as env for the subprocess
• **kwargs – Any other keyword args passed will be passed to the :ref:subprocess.Popen call

Returns:

result of the interactive execution

Return type:

lago.utils.CommandStatus

lago.utils.run_interactive_command(command, env=None, **kwargs)

Runs a command interactively, reusing the current stdin, stdout and stderr

Parameters:

• command (list of str) – args of the command to execute, including the command itself as command[0] as [‘ls’, ‘-l’]
• of str (env(dict) – str): If set, will use the given dict as env for the subprocess
• **kwargs – Any other keyword args passed will be passed to the :ref:subprocess.Popen call

Returns:

result of the interactive execution

Return type:

lago.utils.CommandStatus

lago.utils.service_is_enabled(name)

lago.utils.with_logging(func)

lago.virt module

class lago.virt.BridgeNetwork(env, spec)

Bases: lago.virt.Network

_libvirt_xml()

start()

stop()

lago.virt.LIBVIRT_URL = 'qemu:///system'

Url to the libvirt daemon

class lago.virt.NATNetwork(env, spec)

Bases: lago.virt.Network

_libvirt_xml()

class lago.virt.Network(env, spec)

Bases: object

_libvirt_name()

add_mapping(name, ip, save=True)

add_mappings(mappings)

alive()

gw()

is_management()

name()

resolve(name)

save()

start()

stop()

class lago.virt.ServiceState

ACTIVE = 2

INACTIVE = 1

MISSING = 0

class lago.virt.VM(env, spec)

Bases: object

VM properties: * name * cpus * memory * disks * metadata * network/mac addr

_artifact_paths()

_check_alive(func)

_check_defined(func)

_create_dead_snapshot(name)

_create_live_snapshot(name)

_detect_service_manager()

_extract_paths_dead(paths)

_extract_paths_live(paths)

_extract_paths_scp(paths)

_get_ssh_client(*args, **kwargs)

_libvirt_name()

_libvirt_xml()

classmethod _normalize_spec(spec)

_open_ssh_client()

_reclaim_disk(path)

_reclaim_disks()

_scp(*args, **kwds)

_template_metadata()

alive()

bootstrap()

collect_artifacts(host_path)

copy_from(remote_path, local_path, recursive=True)

copy_to(local_path, remote_path)

create_snapshot(name)

defined()

distro()

extract_paths(paths)

guest_agent()

has_guest_agent()

interactive_console(*args, **kwargs)

Opens an interactive console

Returns:result of the virsh command execution Return type:lago.utils.CommandStatus

interactive_ssh(*args, **kwargs)

ip()

iscsi_name()

metadata

name()

nets()

nics()

revert_snapshot(name)

root_password()

save(path=None)

service(*args, **kwargs)

ssh(command, data=None, show_output=True, propagate_fail=True, tries=None)

ssh_reachable()

ssh_script(path, show_output=True)

start()

state()

Return a small description of the current status of the domain

Returns:small description of the domain status, ‘down’ if it’s not defined at all. Return type:str

stop()

virt_env()

vnc_port(*args, **kwargs)

wait_for_ssh()

class lago.virt.VirtEnv(prefix, vm_specs, net_specs)

Bases: object

Env properties: * prefix * vms * net

• libvirt_con

_create_net(net_spec)

_create_vm(vm_spec)

bootstrap()

create_snapshots(*args, **kwargs)

classmethod from_prefix(prefix)

get_net(name=None)

get_nets()

get_snapshots(domains=None)

Get the list of snapshots for each domain

Parameters:

• domanins (list of str) – list of the domains to get the snapshots
• all will be returned if none or empty list passed (for,) –

Returns:

dict of str -> list – with the domain names and the list of snapshots for each

Return type:

str

get_vm(name)

get_vms()

libvirt_con

prefixed_name(unprefixed_name, max_length=0)

Returns a uuid pefixed identifier

Parameters:

• unprefixed_name (str) – Name to add a prefix to
• max_length (int) – maximum length of the resultant prefixed name, will adapt the given name and the length of the uuid ot fit it

Returns:

prefixed identifier for the given unprefixed name

Return type:

str

revert_snapshots(*args, **kwargs)

save(*args, **kwargs)

start(vm_names=None)

stop(vm_names=None)

virt_path(*args)

class lago.virt._Service(vm, name)

alive()

exists()

classmethod is_supported(vm)

start()

stop()

class lago.virt._SysVInitService(vm, name)

Bases: lago.virt._Service

BIN_PATH = '/sbin/service'

_request_start()

_request_stop()

state()

class lago.virt._SystemdContainerService(vm, name)

Bases: lago.virt._Service

BIN_PATH = '/usr/bin/docker'

HOST_BIN_PATH = '/usr/bin/systemctl'

_request_start()

_request_stop()

state()

class lago.virt._SystemdService(vm, name)

Bases: lago.virt._Service

BIN_PATH = '/usr/bin/systemctl'

_request_start()

_request_stop()

state()

lago.virt._gen_ssh_command_id()

lago.virt._guestfs_copy_path(g, guest_path, host_path)

lago.virt._ip_to_mac(ip)

lago.virt._path_to_xml(basename)

lago.workdir module

A workdir is the base directory where lago will store all the files it needs and that are unique (not shared between workdirs).

It’s basic structure is a directory with one soft link and multiple directories, one per prefix. Where the link points to the default prefix to use.

exception lago.workdir.MalformedWorkdir

Bases: lago.workdir.WorkdirError

exception lago.workdir.PrefixAlreadyExists

Bases: lago.workdir.WorkdirError

exception lago.workdir.PrefixNotFound

Bases: lago.workdir.WorkdirError

class lago.workdir.Workdir(path, prefix_class=<class 'lago.prefix.Prefix'>)

Bases: object

This class reperesents a base workdir, where you can store multiple prefixes

Properties:

path(str): Path to the workdir perfixes(dict of str->self.prefix_class): dict with the prefixes in the workdir, by name current(str): Name of the current prefix prefix_class(type): Class to use when creating prefixes

_set_current(new_current)

Change the current default prefix, for internal usage

Parameters:new_current (str) – Name of the new current prefix, it must already exist Returns:None Raises:PrefixNotFound – if the given prefix name does not exist in the workdir

_update_current()

Makes sure that a current is set

add_prefix(workdir, *args, **kwargs)

Adds a new prefix to the workdir.

Parameters:

• name (str) – Name of the new prefix to add
• *args – args to pass along to the prefix constructor
• *kwargs – kwargs to pass along to the prefix constructor

Returns:

The newly created prefix

Raises:

PrefixAlreadyExists – if the prefix name already exists in the workdir

destroy(workdir, *args, **kwargs)

Destroy all the given prefixes and remove any left files if no more prefixes are left

Parameters:

• prefix_names (list of str) – list of prefix names to destroy, if None
• passed (default) –

get_prefix(workdir, *args, **kwargs)

Retrieve a prefix, resolving the current one if needed

Parameters:name (str) – name of the prefix to retrieve, or current to get the current one Returns:instance of the prefix with the given name Return type:self.prefix_class

initialize(prefix_name='default', *args, **kwargs)

Initializes a workdir by adding a new prefix to the workdir.

Parameters:

• prefix_name (str) – Name of the new prefix to add
• *args – args to pass along to the prefix constructor
• *kwargs – kwargs to pass along to the prefix constructor

Returns:

The newly created prefix

Raises:

PrefixAlreadyExists – if the prefix name already exists in the workdir

classmethod is_workdir(path)

Check if the given path is a workdir

Parameters:path (str) – Path to check Returns:True if the given path is a workdir Return type:bool

join(*args)

Gets a joined path prefixed with the workdir path

Parameters:*args (str) – path sections to join Returns:Joined path prefixed with the workdir path Return type:str

load()

Loads the prefixes that are available is the workdir

Returns:None Raises:MalformedWorkdir – if the wordir is malformed

classmethod resolve_workdir_path(start_path='.')

Look for an existing workdir in the given path, in a path/.lago dir, or in a .lago dir under any of it’s parent directories

Parameters:start_path (str) – path to start the search from, if None passed, it will use the current dir Returns:path to the found prefix Return type:str Raises:RuntimeError – if no prefix was found

set_current(workdir, *args, **kwargs)

Change the current default prefix

Parameters:new_current (str) – Name of the new current prefix, it must already exist Returns:None Raises:PrefixNotFound – if the given prefix name does not exist in the workdir

exception lago.workdir.WorkdirError

Bases: exceptions.RuntimeError

Base exception for workdir errors, catch this one to catch any workdir error

lago.workdir.workdir_loaded(func)

Decorator to make sure that the workdir is loaded when calling the decorated function

lago_template_repo package

class lago_template_repo.TemplateRepoCLI

Bases: lago.plugins.cli.CLIPlugin

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 25

_abc_registry = <_weakrefset.WeakSet object>

do_run(args)

init_args = {'help': 'Utility for system testing template management'}

populate_parser(parser)

class lago_template_repo.Verbs

ADD = 'add'

UPDATE = 'update'

lago_template_repo.do_add(args)

lago_template_repo.do_update(args)

ovirtlago package

class ovirtlago.OvirtPrefix(*args, **kwargs)

Bases: lago.prefix.Prefix

_activate()

_create_rpm_repository(dists, repos_path, repo_names, projects_list=None)

_create_virt_env()

_deactivate()

create_snapshots(name, restore=True)

deploy(*args, **kwargs)

prepare_repo(*args, **kwargs)

revert_snapshots(name)

run_test(*args, **kwargs)

serve(*args, **kwargs)

start()

stop()

class ovirtlago.OvirtWorkdir(*args, **kwargs)

Bases: lago.workdir.Workdir

ovirtlago._activate_all_hosts(api)

ovirtlago._activate_all_storage_domains(*args, **kwargs)

ovirtlago._activate_storage_domains(api, sds)

ovirtlago._build_engine_rpms(engine_dir, output_dir, dists, build_gwt=False)

ovirtlago._build_ioprocess_rpms(source_dir, output_dir, dists)

ovirtlago._build_rpms(name, script, source_dir, output_dir, dists, env=None)

ovirtlago._build_vdsm_jsonrpc_java_rpms(source_dir, output_dir, dists)

ovirtlago._build_vdsm_rpms(vdsm_dir, output_dir, dists)

ovirtlago._deactivate_all_hosts(api)

ovirtlago._deactivate_all_storage_domains(*args, **kwargs)

ovirtlago._deactivate_storage_domains(api, sds)

ovirtlago._fix_reposync_issues(reposync_out, repo_path)

Fix for the issue described at::

https://bugzilla.redhat.com/show_bug.cgi?id=1332441

ovirtlago._git_revision_at(path)

ovirtlago._sync_rpm_repository(repo_path, yum_config, repos)

ovirtlago._with_repo_server(func)

Submodules

ovirtlago.cmd module

class ovirtlago.cmd.OvirtCLI

Bases: lago.plugins.cli.CLIPlugin

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 25

_abc_registry = <_weakrefset.WeakSet object>

do_run(args)

init_args = {'help': 'oVirt related actions'}

populate_parser(parser)

ovirtlago.cmd._populate_parser(cli_plugins, parser)

ovirtlago.constants module

ovirtlago.merge_repos module

ovirtlago.merge_repos._fastcopy(source, dest)

ovirtlago.merge_repos.merge(output_dir, input_dirs)

ovirtlago.paths module

class ovirtlago.paths.OvirtPaths(prefix)

Bases: lago.paths.Paths

build_dir(*path)

internal_repo(*path)

test_logs(*args)

ovirtlago.testlib module

class ovirtlago.testlib.LogCollectorPlugin(prefix)

Bases: nose.plugins.base.Plugin

_addFault(test, err)

addError(test, err)

addFailure(test, err)

configure(options, conf)

name = 'log-collector-plugin'

options(parser, env=None)

class ovirtlago.testlib.TaskLogNosePlugin(*args, **kwargs)

Bases: nose.plugins.base.Plugin

configure(options, conf)

name = 'tasklog-plugin'

options(parser, env)

startTest(test)

stopTest(test)

ovirtlago.testlib._instance_of_any(obj, cls_list)

ovirtlago.testlib._vms_capable(vms, caps)

ovirtlago.testlib.assert_true_within(func, timeout, allowed_exceptions=None)

ovirtlago.testlib.assert_true_within_long(func, allowed_exceptions=None)

ovirtlago.testlib.assert_true_within_short(func, allowed_exceptions=None)

ovirtlago.testlib.continue_on_failure(func)

ovirtlago.testlib.engine_capability(caps)

ovirtlago.testlib.get_test_prefix()

ovirtlago.testlib.host_capability(caps)

ovirtlago.testlib.test_sequence_gen(test_list)

ovirtlago.testlib.with_ovirt_api(func)

ovirtlago.testlib.with_ovirt_prefix(func)

ovirtlago.utils module

ovirtlago.utils._create_http_server(listen_ip, listen_port, root_dir)

Starts an http server with an improved request handler

Parameters:

• listen_ip (str) – Ip to listen on
• port (int) – Port to register on
• root_dir (str) – path to the directory to serve

Returns:

instance of the http server, already running on a thread

Return type:

BaseHTTPServer

ovirtlago.utils.generate_request_handler(root_dir)

Factory for _BetterHTTPRequestHandler classes

Parameters:root_dir (path) – Path to the dir to serve Returns:A ready to be used improved http request handler Return type:_BetterHTTPRequestHandler

ovirtlago.utils.repo_server_context(*args, **kwds)

Context manager that starts an http server that serves the given prefix’s yum repository. Will listen on constants.REPO_SERVER_PORT and on the first network defined in the previx virt config

Parameters:prefix (ovirtlago.OvirtPrefix) – prefix to start the server for Returns:None

ovirtlago.virt module

class ovirtlago.virt.EngineVM(*args, **kwargs)

Bases: lago.virt.VM

_artifact_paths()

_create_api()

_get_api(wait)

add_iso(path)

engine_setup(config=None)

get_api(wait=True)

stop()

class ovirtlago.virt.HostVM(env, spec)

Bases: lago.virt.VM

_artifact_paths()

class ovirtlago.virt.NodeVM(env, spec)

Bases: lago.virt.VM

_artifact_paths()

collect_artifacts(host_path)

wait_for_ssh()

class ovirtlago.virt.OvirtVirtEnv(prefix, vm_specs, net_spec)

Bases: lago.virt.VirtEnv

_create_vm(vm_spec)

engine_vm()

host_vms()

Releases

Release process

Versioning

For lago we use a similar approach to semantic versioning, that is:

X.Y.Z

For example:

0.1.0
1.2.123
2.0.0
2.0.1

Where:

• Z changes for each patch (number of patches since X.Y tag)
• Y changes from time to time, with milestones (arbitrary bump), only for backwards compatible changes
• X changes if it’s a non-backwards compatible change or arbitrarily (we don’t like Y getting too high, or big milestone reached, ...)

The source tree has tags with the X.Y versions, that’s where the packaging process gets them from.

On each X or Y change a new tag is created.

For now we have only one branch (master) and we will try to keep it that way as long as possible, if at some point we have to support old versions, then we will create a branch for each X version in the form:

vX

For example:

v0
v1

There’s a helper script to resolve the current version, based on the last tag and the compatibility breaking commits since then, to get the version for the current repo run:

$ scripts/version_manager.py . version

RPM Versioning

The rpm versions differ from the generic version in that they have the -1 suffix, where the -1 is the release for that rpm (usually will never change, only when repackaging without any code change, something that is not so easy for us but if there’s any external packagers is helpful for them)

Repository layout

Tree schema of the repository:

lago
├── stable <-- subdirs for each major version to avoid accidental
│   │          non-backwards compatible ugrade
│   │
│   ├── 0.0  <-- Contains any 0.* release for lago
│   │   ├── ChangeLog_0.0.txt
│   │   ├── rpm
│   │   │   ├── el6
│   │   │   ├── el7
│   │   │   ├── fc22
│   │   │   └── fc23
│   │   └── sources
│   ├── 1.0
│   │   ├── ChangeLog_1.0.txt
│   │   ├── rpm
│   │   │   ├── el6
│   │   │   ├── el7
│   │   │   ├── fc22
│   │   │   └── fc23
│   │   └── sources
│   └── 2.0
│       ├── ChangeLog_2.0.txt
│       ├── rpm
│       │   ├── el6
│       │   ├── el7
│       │   ├── fc22
│       │   └── fc23
│       └── sources
└── unstable <-- Multiple subdirs are needed only if branching
    ├── 0.0  <-- Contains 0.* builds that might or might not have
    │   │        been released
    │   ├── latest  <--- keeps the latest build from merged, static
    │   │                url
    │   ├── snapshot-lago_0.0_pipeline_1
    │   ├── snapshot-lago_0.0_pipeline_2
    │   │         ^ contains the rpms created on the pipeline build
    │   │           number 2 for the 0.0 version, this is needed to
    │   │           ease the automated testing of the rpms
    │   │
    │   └── ... <-- this is cleaned up from time to time to avoid
    │               using too much space
    ├── 1.0
    │   ├── latest
    │   ├── snapshot-lago_1.0_pipeline_1
    │   ├── snapshot-lago_pipeline_2
    │   └── ...
    └── 2.0
        ├── latest
        ├── snapshot-lago_2.0_pipeline_1
        ├── snapshot-lago_2.0_pipeline_2
        └── ...

Promotion of artifacts to stable, aka. releasing

The goal is to have an automated set of tests, that check in depth lago, and run them in a timely fashion, and if passed, deploy to stable. As right now that’s not yet possible, so for now the tests and deploy is done manually.

The promotion of the artifacts is done in these cases:

• New major version bump (X+1.0, for example 1.0 -> 2.0)
• New minor version bump (X.Y+1, for exampyre 1.1 -> 1.2)
• If it passed certain time since the last X or Y version bumps (X.Y.Z+n, for example 1.0.1 -> 1.0.2)
• If there are blocking/important bugfixes (X.Y.Z+n)
• If there are important new features (X.Y+1 or X.Y.Z+n)

How to mark a major version

Whenever there’s a commit that breaks the backwards compatibility, you should add to it the pseudo-header:

Sem-Ver: api-breaking

And that will force a major version bump for any package built from it, that is done so in the moment when you submit the commit in gerrit, the packages that are build from it have the correct version.

After that, make sure that you tag that commit too, so it will be easy to look for it in the future.

The release procedure on the maintainer side

1. Select the snapshot repo you want to release

2. Test the rpms, for now we only have the tests from projects that use it:

   • Run all the ovirt tests on it, make sure it does not break anything, if there are issues -> open bug

   • Run vdsm functional tests, make sure it does not break anything, if

     there are issues -> open bug

3. On non-major version bump X.Y+1 or X.Y.Z+n

   • Create a changelog since the base of the tag and keep it aside

4. On Major version bump X+1.0

   • Create a changelog since the previous .0 tag (X.0) and keep

     it aside

5. Deploy the rpms from snapshot to dest repo and copy the ChangeLog from the tarball to ChangeLog_X.0.txt in the base of the stable/X.0/ dir

6. Send email to lago-devel with the announcement and the changelog since the previous tag that you kept aside, feel free to change the body to your liking:

   Subject: [day-month-year] New lago release - X.Y.Z
   
   Hi everyone! There's a new lago release with version X.Y.Z ready for you to
   upgrade!
   
   Here are the changes:
       <CHANGELOG HERE>
   
   Enjoy!
   

Changelog

Here you can find the full changelog for this version

Indices and tables

• Index
• Module Index
• Search Page