Regular users of Ansible are probably familiar with the first_found lookup plugin. It’s incredibly useful for dealing with distribution variance. Say you’re writing a role which installs a few packages but the name of the package isn’t consistent across distributions. For example the start of your role’s main.yml might be something like the following:

- name: source distribution dependent variables
  include_vars: {{ item }}
  with_first_found:
    - "{{ ansible_distribution }}-{{ ansible_distribuion_major_version }}.yml"
    - "{{ ansible_distribution }}.yml"
    - "{{ ansible_os_family }}.yml"
    - "defaults.yml"

A not often used feature of this module is that you can actually pass this module undefined variables. For people who are using older versions of Ansible this might not seem weird at all but newer versions of Ansible will throw you an error if you attempt to template an undefined variable. For example the following task will fail if apache_module_packages is undefined.

- name: install apache modules
  package: name={{ item }} state=installed
  with_items: "{{ apache_module_packages }}"

You can either fix this by adding a default value in defaults/main.yml or by using the default filter. But first_found has no such trouble handling undefined variables; in the following example if either more_specific_conf or less_specific_conf are undefined then the lookup will happily proceede to the next file. As an end user, you might not notice that this is actually invoking weird behavior because it’s relatively intuitive that an undefeind variable should be skipped.

- name: copy configuration file
  copy: src={{ item }} dest=/etc/myapp.conf
  with_first_found:
    - "{{ more_specific_conf }}"
    - "{{ less_specific_conf }}"
    - "generic.conf"

In my work with Ansible I found myself wanting for a lookup plug-in similar to first_found but searching for variables rather than files – let’s call it first_defined.

My Use Case⌗

Since bringing a feature into a Free Software project usually means committing to the maintenance of that feature for an indefinite amount of time, one has to demonstrate a high degree of usefulness. I’m not entirely sure if my simple plug-in passes this test, but it has served me well in my environment.

In my shop we group our systems by class, group, and host where the class is the type of system (e.g. fileserver, webserver, or workstation), the group is the department that owns the machine, and the host is the machine itself. Sometimes when we apply a configuration we want to merge the options from higher in the hierarchy; for example, the packages installed for the accounting group’s webserver should have the packages common to all webservers, the packages that the accounting group needs, and the packages specific to that host. But other times we want to pick the most specific configuration; for example, choosing the appropriate RedHat Satellite activation key.

In a simpler case we might be able implement this type of behavior using Ansible’s default variable precedence but since our group and class variables are both represented as Ansible groups in ./group_vars we can’t (or shouldn’t) try to make Ansible prefer the variables from some groups over others. However, even if we could use variable precedence to get this behavior Ansible’s official documentation strongly recommends against doing this because it makes debugging more difficult.

So the pattern I’ve been using is postfixing the variable with its hierarchal identifier, for example the package installation example would be something like the following with each variable being defined in its respective inventory file.

- name: install web server packages
  package: name={{ item }} state=installed
  with_items:
    - "{{ web_packages_common }}"
    - "{{ web_packages_class }}"
    - "{{ web_packages_group }}"
    - "{{ web_packages_host }}"

Approaching The Problem⌗

Since what we’re looking for is a modified version of first_found we should probably start with its source in lib/ansible/plguins/lookup/first_found.py. It’s not a particularly long plug-in and most of it is devoted to the file searching aspect, but here’s the relevant excerpt.

for fn in total_search:
    try:
        fn = self._templar.template(fn)
    except (AnsibleUndefinedVariable, UndefinedError) as e:
        continue

    if os.path.isabs(fn) and os.path.exists(fn):
        return [fn]
    else:
        if roledir is not None:
            # check the templates and vars directories too,if they exist
            for subdir in ('templates', 'vars', 'files'):
                path = self._loader.path_dwim_relative(roledir, subdir, fn)
                if os.path.exists(path):
                    return [path]

        # if none of the above were found, just check the
        # current filename against the current dir
        path = self._loader.path_dwim(fn)
        if os.path.exists(path):
            return [path]

So we follow this code as a template and come up with out first version of first_defined.

def run(self, terms, variables, **kwargs):

    skip = False
    all_expressions = []

    for term in terms:
        # Check if we're using the alternate syntax.
        if isinstance(term, dict):
            expressions = term.get('expr',[])
            skip = boolean(term.get('skip', False))

            for expr in expressions:
                all_expressions.append(expr)
        else:
            all_expressions.append(term)

    for expr in all_expressions:
        try:
            expr = self._templar.template(expr)
            return [expr]
        except (AnsibleUndefinedVariable, UndefinedError):
            continue

    # We didn't find any valid expressions. Should we skip the task?
    if skip:
        return []
    else:
        raise AnsibleLookupError(self.lookup_error_message)

So we’re done right? Let’s try and run it!

- name: test first defined
  debug: var=item
  with_first_defined:
    - "{{ undefined_variable }}"
    - "{{ defined_variable }}"
    - "default_value"

[DEPRECATION WARNING]: Skipping task due to undefined Error, in the future this
will be a fatal error.: 'undefined_variable' is undefined.

So what went wrong? Rather than recant the time spent figuring out how first_found works I’ll just give you the answer. If you check the source for the Ansible task executor here you’ll see the following.

if self._task.loop == 'first_found':
    # first_found loops are special.  If the item is undefined
    # then we want to fall through to the next value rather
    # than failing.
    loop_terms = listify_lookup_plugin_terms(terms=self._task.loop_args, templar=templar, loader=self._loader, fail_on_undefined=False, convert_bare=True)
    loop_terms = [t for t in loop_terms if not templar._contains_vars(t)]
else:
    try:
        loop_terms = listify_lookup_plugin_terms(terms=self._task.loop_args, templar=templar, loader=self._loader, fail_on_undefined=True, convert_bare=True)
    except AnsibleUndefinedVariable as e:
        display.deprecated("Skipping task due to undefined Error, in the future this will be a fatal error.: %s" % to_bytes(e))
        return None

Apparently, every other plug-in has its variables templated before prior to passing control, but first_found is magic. To their credit they’re clearly trying to remove this behavior but it’s incredibly frustrating when upstream plug-ins can do things that user provided plug-ins can’t.

Working Around the Problem⌗

After some contemplation I came to the conclusion that I actually like this behavior, but it does lead a fairly awkward syntax for specifying a hard coded default value. Rather then pass a jinja2 block complete with {{ }} we instead just pass the inner contents as strings which my plug-in will have no trouble templating. Thus our previous example becomes the following.k

- name: test first defined
  debug: var=item
  with_first_defined:
    - "undefined_variable"
    - "defined_variable"
    - "'default value'"

I personally think this is a little ugly, but you can avoid the double quoting by putting a variable in defaults/main.yml and then simply specifying the name as the default parameter. I would consider this a best practice since all your default values will be stored in a single place rather then strewn throughout your playbook.

This actually only requires one simple modification to our earlier code.

def run(self, terms, variables, **kwargs):

    skip = False
    all_expressions = []

    for term in terms:
        # Check if we're using the alternate syntax.
        if isinstance(term, dict):
            expressions = term.get('expr',[])
            skip = boolean(term.get('skip', False))

            for expr in expressions:
                all_expressions.append(expr)
        else:
            all_expressions.append(term)

    for expr in all_expressions:
        try:
            # Pass the templar an expression so it isn't treated as a literal
            # string.
-           expr = self._templar.template(expr)
+           expr = self._templar.template("{{ %s }}" % expr)
            return [expr]
        except (AnsibleUndefinedVariable, UndefinedError):
            continue

    # We didn't find any valid expressions. Should we skip the task?
    if skip:
        return []
    else:
        raise AnsibleLookupError(self.lookup_error_message)

If you like my plug-in and think it could be useful in your environment the source code for the latest version is available on GitHub estheruary/ansible-with-first-defined.