Perfect shape. Handle Complex Forms in Python with WTForms.

[Hacker]

The content of the article

• Why is this needed?
• Installation
• Form creation
• Working with a form
• Form generation (GET /users/new)
• Payload parsing (POST /users)
• Options for Partial Parsing of Payload
• Validators
• Dynamically changing properties of form fields
• Prefabs and inherited forms
• Populating relational fields (one-to-many, many-to-many)
• Custom widgets and extensions
• Instead of a conclusion

Why is this needed?
To understand what problem we are solving, let's take a look at a simple example. Imagine that our web application has a form to create users.

<form action="">
    <!-- personal info -->
    <input type="text" id="f_name" name="f_name" placeholder="John" />
    <input type="text" id="l_name" name="l_name" placeholder="Dow" />

    <!-- account info -->
    <input type="email" id="email" name="email" placeholder="[email protected]" />
    <input type="password" id="password" name="password" placeholder="**********" />

    <!-- meta info -->
    <select name="gender" id="gender">
        <option value="0">Male</option>
        <option value="1" selected>Female</option>
    </select>
    <input type="city" id="city" name="city" placeholder="Saint-Petersburg" />
    <textarea name="signature" id="signature" cols="30" rows="10"></textarea>

    <input type="submit" value="Create user!" />
</form>

This form looks simple. However, using it in a real application will add a number of tasks.

1. For each field (or in one block), you need to display information about errors that may appear during form validation.
2. Most likely, we will want hints for some fields.
3. Surely we will need to hang one or more CSS classes for each field, or even do it dynamically.
4. Some of the fields should contain pre-filled data from the backend - previous attempts to submit a form or data for drop-down lists. The special case with the field genderis simple, but the options for the selection can be generated by queries to the database.

Etc. All these tweaks will at least double our shape.

Now let's look at how we will process this form on the server. For each field, we must do the following.

1. Correctly map it by name.
2. Check the range of acceptable values - validate the form.
3. If there were errors, save them by returning the edit form back to the client side.
4. If everything is OK, then map them to a database object or a structure similar in properties for further processing.

In addition, when creating the user you as the administrator have to fill only part of the data ( emailand password) the rest of the user fills in the profile itself. In this case, you will most likely have to copy the template, delete some of the fields, create an identical form handler on the server, or insert checks into the current one for different form variants. The field validation logic will either have to be copied or moved into a separate function. In this case, you need not to get confused in the names of the fields coming from the client, otherwise the data will simply be lost.

But users not only need to be created, but also edited using the same form! Moreover, for the administrator and the user, these forms will be different, with a partially overlapping set of fields.
All these requirements dramatically increase the number of templates, handlers, validators, which, at best, will be placed in a common module, and most likely will be copied and pasted quickly. And if you need to change one field in the form, you will have to shovel the entire application, catching errors and typos.

It would be more convenient to describe the form in some kind of declarative format, for example, in the form of a Python class, once describing all parameters, classes, validators, handlers, and at the same time providing for the possibility of its inheritance and extension. This is where the WTForms library comes in handy.

INFO
If you've used large frameworks like Django or Rails, you've already come across similar functionality in one form or another. However, not every task requires huge Django. It is convenient to use WTForms in conjunction with lightweight micro-frameworks or in highly specialized applications with the need to process web forms, where using Django is unjustified.

Installation
First, let's install the library itself. I will show examples in Python 3. Where context is needed, the code is executed in the aiohttp framework handler. This does not change the essence - the examples will work with Flask, Sanic or any other module. Jinja2 is used as a template engine .Install via pip:

pip install wtforms

Checking the version.

import wtforms
wtforms.__version__
# '2.2.1'

Let's try to rewrite the form above in WTForms and process it.

Form creation
WTForms has a number of built-in classes for describing forms and their fields. A shape definition is a class that inherits from a library built-in class Form. Form fields are described by class attributes, each of which, when created, is assigned an instance of the class of the field of the type corresponding to the type of the form field. It sounds complicated, but it's actually easier.

from wtforms import Form, StringField, TextAreaField, SelectField, validators

class UserForm(Form):
    first_name = StringField('First name', [validators.Length(min=5, max=30)])
    last_name = StringField('Last name', [validators.Length(min=5, max=30)])

    email = StringField('Email', [validators.Email()])
    password = StringField('Password')

    # meta
    gender = SelectField('Gender', coerce=int, choices=[  # cast val as int
        (0, 'Male'),
        (1, 'Female'),
    ])
    city = StringField('City')
    signature = TextAreaField('Your signature', [validators.Length(min=10, max=4096)])

Here's what we did:

• created a class UserFormfor our form. It inherits from built-in Form(s BaseForm);
• each of the form fields was described with a class attribute, assigning an object of the type built into the lib Field.

In most of the form fields, we have used an imported class StringField. As you might guess, the field genderrequires a different type of input - a limited set of values (m / f), so we used SelectField. It is also better to accept a user signature not in a regular one input, but in textarea, so we used TextAreaField, whose HTML representation (widget) is a tag <textarea>. If we needed to handle a numeric value, we would import the built-in class IntegerFieldand describe the field to it.

WWW
WTForms has a lot of built-in classes for describing fields, you can see everything here. You can also create a custom class field.

You need to know the following about fields.

1. Each field can take a set of arguments common to all field types.
2. Almost every field has an HTML representation, a so-called widget.
3. For each field, you can specify a set of validators.
4. Some fields can take additional arguments. For example, SelectFieldyou can specify a set of possible values for.
5. Fields can be added to existing forms. And you can modify, change the values on the fly. This is especially useful when you need to slightly change the behavior of a form for one specific case, without creating a new form class.
6. Fields can provoke validation errors according to the specified rules, they will be stored in form.field.errors.

Working with a form
Let's try to display the form. A typical workflow for working with forms consists of two stages.

1. GET request for the page where we need to display our form. At this point, we must create an instance of our form, configure it, if necessary, and pass it to the template engine in the context for rendering. This is usually done in the handler (action) of the GET request controller and something similar, depending on the HTTP framework you are using (or not using, for WTForms this is not a problem). In other words, in a route handler like GET /users/new. By the way, in Django or Rails, you do similar things. In the first, you create the same form and pass it to the template engine in the template context, and in the second, you create a new, not yet saved object in the current context through the new (@user = User.new) method.
2. POST request for a page from which we must receive form data (for example POST /users) and somehow process: perform data validation, fill in the object fields from the form to save it to the database.

Form generation (GET /users/new)
Let's create an instance of our predefined shape:

user_form = UserForm()
type(user_form)
# __main__.UserForm

We can refer to each form field separately by its attribute:

type(form.first_name)
# wtforms.fields.core.StringField

In the simplest case, that's all. Now an instance of our form can be passed to the template engine for display:

def new(self, request):
    user_form = UserForm()

    render('new_user.html', {
        'form': user_form,
    })

The render method is, of course, specific. In your case, the rendering methods will be determined by the framework and template engine you are using.

Great, we passed our form to the template engine. How to render it in a template? Easy peasy. Let me remind you that we are considering the process using the example of Jinja2.

{{ form.first_name.label }}
{% if form.first_name.errors %}
    <ul class="errors">
        {% for error in form.first_name.errors %}
            <li>{{ error }}</li>{% endfor %}
    </ul>
{% endif %}
{{ form.first_name() }}

The above code with user_formas formwill be transformed by the template engine into the following markup.

<label for="first_name">First name</label>
<input id="first_name" name="first_name" type="text" value="">

Here's what happens.

1. On the first line, we turned to the labelfield attribute of first_nameour form. It contains the HTML code for the label of our field first_name. The text is taken from the description of the form class from the corresponding attribute of the field.
2. We then checked the contents of errorsour field's list . As you might guess, it contains errors. At the moment there are no errors in it, so the block did not output anything. However, if this form had already been filled in and was filled incorrectly (for example, the validator from 6 to 30 in length did not miss the value), then this error would be included in the list of the field. We will see the work of validators further.
3. Finally, on the last line, we render the tag itself by inputcalling a method on .first_name()our form instance.

Everything is very flexible. We can render all the attributes of the field, or just the input tag itself. It is easy to guess that now we can do the same for all other fields by rendering all the form fields or only part of them with the corresponding embedded HTML widgets.

Payload parsing ( POST /users)
The next step is to get the form data on the server and process it somehow. This stage consists of several steps.

1. Get POST data (this can happen in different ways depending on whether you are using a framework and which one, if you do).
2. Parse POST data through our form instance.
3. Check (validate) the correctness of filling. If something is wrong, return errors.
4. Fill in the required object with the form data. This is optional, but if you use an ORM, chances are high that you need to create an object in the database based on the form data.

In our case, the object in the database is a user, an object of the User class.

async def create(self, request):
    # Get payload. For aiohttp, this is not the most optimal
    # way for large payloads. Taken for brevity
    payload = await request.post()
    # Create a new instance of our form and fill it with data,
    # came from the client
    form = UserForm(payload)

    # If the data from the client is validated
    if form.validate():
        # Create a new User object
        user = User()
        # Fill its attributes with form data
        form.populate_obj(user)
        # ...
        # Save the user to the database, redirect further

We rendered the form, got the data back from the client, checked it and wrote it to the database. At the same time, we did not dive into the internals of HTML, field IDs, names and their mappings on the client and server. Isn't it convenient?

Options for Partial Parsing of Payload
If you carefully read the previous section, you certainly have a question: how is the user model filled with form data? After all, the form does not know anything about the ORM fields (which may not exist). So how is the mapping of form fields to an object in a function populatefrom WTForms done? The easiest way is to look at the code of this function.

Signature: form.populate_obj(obj)
Source:
    def populate_obj(self, obj):
        """
        Populates the attributes of the passed `obj`
        with data from the form's fields.

        :note: This is a destructive operation;
               Any attribute with the same name
               as a field will be overridden. Use with caution.
        """
        for name, field in iteritems(self._fields):
            field.populate_obj(obj, name)

As you can see, the function gets a list of all the fields in our form, and then iterates through the list and assigns values to the attributes of the provided object. On top of that, this happens recursively: this is needed for container fields - FormFields.
This works great in most cases. Even for association fields: a user can have a field whose value is a relational database, for example, a group to which the user belongs. In this case, use the class wtforms.fields.SelectField, passing it choices=[...]with a list of possible relational values, and on the server, if there is an ORM, it will be recognized without problems.

However, sometimes you still need to automatically fill in the class attributes only with part of the form fields, and somehow preprocess the rest. There are two options.

1. Do not use a built-in function populate_objat all and process all fields manually, accessing them through an attribute of .dataeach form field like form.f_name.data.
2. Write your own method to fill the object with form data.

I like the second option better (although it has limitations). For example, like this:

from wtforms.compat import iteritems, itervalues, with_metaclass

def populate_selective(form, obj, exclude=[]):
    for name, field in filter(lambda f: f[0] not in exclude, iteritems(form._fields)):
        field.populate_obj(obj, name)

Now you can use only those fields from the form that you need:

populate_selective(form, user, exclude=['f_name', 'l_name', 'city',])

And deal with the rest according to your own logic.

Validators
Another question, the answer to which you probably already understood from the context: how does the function work form.validate()? It checks exactly the very lists of validators with parameters that we specified when defining the form class. Let's try to fill in various values in string fields that the user will provide to our form in a real application from the client, and see how the validator reacts.

form = UserForm()

form.first_name.data = 'Johnny'
form.last_name.data = 'Do'
form.email.data = 'invalid_email'
form.password.data = 'super-secret-pass'

Let's try to validate this form.

form.validate()
# False

Validation failed. Do you remember that in each field there is a list errorsthat will contain errors if they occur while filling out the form. Let's take a look at them.

form.first_name.errors
# []

Everything is correct, there were no errors in the first field, the list of validators was [validators.Length(min=5, max=30)]passed, since the name Johnnymatches the only validator. Let's see others.

form.last_name.errors
# ['Field must be between 5 and 30 characters long.']
form.email.errors
# ['Invalid email address.']
form.password.errors
# []

In the second and third cases, the validators worked, and our template (below) will display a list of errors.

{% if form.first_name.errors %}
    <ul class="errors">
        {% for error in form.first_name.errors %}
            <li>{{ error }}</li>{% endfor %}
    </ul>
{% endif %}

INFO
Of course, for everything to work, to replenish the form, you need to send the same instance of the form to the template engine, and not create a new one. In addition to the list of errors, it will contain pre-filled fields from the previous attempt, so that the user does not have to enter everything over again.

A complete list of built-in validators can be found here , and if they are not enough, WTForms allows you to define your own.

Dynamically changing properties of form fields
You already know that form fields have a set of common attributes that can be specified across all field classes. For example, a description that comes as the first positional argument in any field. Other examples:

• id - the ID attribute of the HTML widget when rendering;
• name- the name of the widget (property namein HTML) by which the mapping will be done;
• errors, validators, and so on.

All this is possible due to the fact that all field classes inherit from the wtforms.fields.Field base class.

However, the cases are different. Sometimes, in an already defined form, you may need to change the value of the fields. For example,

• set a default value in one of the string fields;
• add a class for another field when rendering (because the same form is used in many places in the application and this requires a special class);
• for the third field, specify a data attribute for the client code that contains an API endpoint for dynamic data fetching.

It is better to configure all these points right before the very rendering of the form at the finished form instance: there is absolutely no need to drag this into the class definition. But how to do that? Let's remember that our form is a regular Python object and we can manipulate its attributes!

Let's set the default value of the field first_name(another option is through default):

form.first_name.data = 'Linus'

A field of any class has a dictionary render_kw. It provides a list of attributes that will be rendered in an HTML tag (widget).

# The field now looks good with Bootstrap!
form.last_name.render_kw['class'] = 'form-control'

Well, let's set a custom data attribute to check for duplicate accounts:

form.users.render_kw['data-url'] = request.app.router['api_users_search'].url_for()

Prefabs and inherited forms
At the very beginning of the article, we said that the same form can be used in different situations. Usually we move the form description into a separate module and then import it. But in one case, we should have only a minimal set of form fields (attributes), and in the other, an extended one. Inheritance will help us avoid duplicating definitions of form classes.

Let's define the base class of the form:

class UserBaseForm(Form):
    email = StringField('Email', [validators.Email()])
    password = StringField('Password')

It will contain only those fields that are required to create a user account. And then we define an extended one that will inherit from the base one:

class UserExtendedForm(UserBaseForm):
    first_name = StringField('First name', [validators.Length(min=4, max=25)])
    last_name = StringField('Last name', [validators.Length(min=4, max=25)])

Let's create two forms and see what fields they have.

base_form = UserBaseForm()
base_form._fields
# OrderedDict([('email', <wtforms.fields.core.StringField at 0x106b1df60>),
# ('password', <wtforms.fields.core.StringField at 0x106b1d630>)])

Now let's see what our extended form contains:

extended_from = UserExtendedForm()
extended_from._fields
# OrderedDict([('email', <wtforms.fields.core.StringField at 0x106b12a58>),
# ('password', <wtforms.fields.core.StringField at 0x106b12f60>),
# ('first_name', <wtforms.fields.core.StringField at 0x106b12e80>),
# ('last_name', <wtforms.fields.core.StringField at 0x106b12ef0>)])

As you can see, it contains not only the described fields, but also those that were defined in the base class. Thus, we can create complex forms, inheriting them from each other, and use the one that we need at the moment in the current controller.

Another way to create complex forms is the already mentioned FormField. This is a separate field class that can inherit from an existing form class. For example, along with Post, you can create a new User for this post by prefixing the field names.

Populating relational fields (one-to-many, many-to-many)
One (not) big problem with building forms is relational. They differ from ordinary fields in that their presentation in the database does not correspond as is to what should be displayed in the form field, and when saved, they may require preprocessing. And this problem is easy to solve with WTForms. Since we know that form fields can be changed dynamically, why not use this property to pre-populate it with objects in the desired format?

Let's look at a simple example: we have a form for creating a post and for it you need to specify a category and a list of authors. The post category is always one, and there can be several authors. By the way, a similar method is used directly on Xakep.ru (I use WTForms on the Hacker's backend, PHP with WP is only in the public part).

We can display relations in a form in two ways.

1. In the usual one <select>, which will be rendered as a drop-down list. This method is suitable when we have few possible values. For example, a list of post categories - there are no more than a dozen of them, including hidden ones.
2. In a dynamically loaded list, similar to the list of tags that you find on other sites. A simple trick will help us to implement it.

In the first version, our form has a category field, in the base it corresponds to a field category_id. To render this field in the template as select, we must create a categoryclass attribute on the form SelectField. When rendering, you need to transfer a list of possible values to it, which is formed by a query in the database (read: a list of possible categories for a post), and also set a default value.

# Import the template helper that represents the Category object
# as a string in the desired format, analogous to __str__. Needed for convenience
from admin.template_helpers.categories import humanize_category

# Select all categories from the database
categories = Category.select().all()

# Set the default value of the first one
form.categories.data = [str(categories[0].id)]

# Pass a list of all possible options for SelectField
# The template will render a <select> with the specified <option> selected
# Format - a list of tuples of the form (<identifier>, <human readable representation>)
form.categories.choices = [(c.id, humanize_category(c)) for c in categories]

As a result, the list field will have pre-filled values.

Pre-filled select with set value via WTForms.
With the authors of posts (users) or magazines, this trick will not work. We have about a hundred thousand of the first, and, of course, it will be impossible to render or search in such a giant select. One of the options for solving the problem is to use the Select2 library. It allows you to turn any inputinto a dynamically loaded list a la list of tags by simply assigning the desired class, and load the data at the provided URL. We already know how to do this through a familiar dictionary render_kw.

form.issues.render_kw['class'] = 'live_multiselect'
form.issues.render_kw['data-url'] = request.app.router['api_issues_search'].url_for()

And then, by simply adding a jQuery function to the template, we turn all input with the required class into dynamically loaded selectors (the search handler, of course, must be on the server):

$(".live_multiselect").each(function (index) {
    let url = $(this).data('url')
    let placeholder = $(this).data('placeholder')

    $(this).select2({
        tags: true,
        placeholder: placeholder,
        minimumInputLength: 3,
        ajax: {
            url: url,
            delay: 1000,
            dataType: 'json',
            processResults: function (data) {
                let querySet = { results: data };
                return querySet
            }
        }
    });
});

As a result, we get a convenient reusable widget.

Dynamically loaded selector using WTForms and Select2.

Custom widgets and extensions
The example above may seem specific, but it leads to an important problem. It's good that our task is solved by the Select2 plugin, which allows literally adding one class and a pinch of JS to get the necessary functionality. However, what if we need a completely custom template for a field, or even a completely our own complex field with a custom template, behavior and validators?

Fortunately, WTForms allows us to create our own widgets (HTML template generator classes for rendering fields). We can do this in two ways:

1. Create your own based on the existing ( class CustomWidget(TextInput):...), extending its behavior and overriding methods, including __call__. For example, wrap in an additional HTML template.
2. Create a completely custom widget without inheriting from existing built-in ones.

A list of built-in widgets can be found here, recommendations and an example of a fully custom one are also present in the documentation.

Integrating your own widget is easy too. Each field has an attribute widget. We can specify our widget as this keyword argument when defining a field in the form class, or, if a custom widget is not always needed, dynamically assign it to the field.

Besides custom widgets, we can create completely custom fields. An example of such a field is the WTForms-JSON extension, which is useful for processing JSON fields of models. It is also possible to define your own field, you will find an example in the documentation.

Instead of a conclusion
Perhaps after reading this article, it seemed to you that a separate library for generating and serving HTML forms is an unnecessary complication. And you will be right when it comes to small applications.

However, when you need to process a dozen complex forms, some of them reuse and form dynamically, a declarative way of describing fields and rules for parsing them allows you not to get confused in the endless noodles of names and IDs and get rid of monotonous work by shifting the writing of template code from the programmer to library. Agree, it's cool.