2012-10-06T05:43:26Z

The Flask Mega-Tutorial, Part IX: Pagination

(Great news! There is a new version of this tutorial!)

This is the ninth article in the series in which I document my experience writing web applications in Python using the Flask microframework.

The goal of the tutorial series is to develop a decently featured microblogging application that demonstrating total lack of originality I have decided to call microblog.

NOTE: This article was revised in September 2014 to be in sync with current versions of Python and Flask.

Here is an index of all the articles in the series that have been published to date:

Recap

In the previous article in the series we've made all the database changes necessary to support the 'follower' paradigm, where users choose other users to follow.

Today we will build on what we did last time and enable our application to accept and deliver real content to its users. We are saying goodbye to the last of our fake objects today!

Submission of blog posts

Let's start with something simple. The home page should have a form for users to submit new posts.

First we define a single field form object (file app/forms.py):

class PostForm(Form):
    post = StringField('post', validators=[DataRequired()])

Next, we add the form to the template (file app/templates/index.html):

<!-- extend base layout -->
{% extends "base.html" %}

{% block content %}
  <h1>Hi, {{ g.user.nickname }}!</h1>
  <form action="" method="post" name="post">
      {{ form.hidden_tag() }}
      <table>
          <tr>
              <td>Say something:</td>
              <td>{{ form.post(size=30, maxlength=140) }}</td>
              <td>
              {% for error in form.post.errors %}
              <span style="color: red;">[{{ error }}]</span><br>
              {% endfor %}
              </td>
          </tr>
          <tr>
              <td></td>
              <td><input type="submit" value="Post!"></td>
              <td></td>
          </tr>
      </table>
  </form>
  {% for post in posts %}
  <p>
    {{ post.author.nickname }} says: <b>{{ post.body }}</b>
  </p>
  {% endfor %}
{% endblock %}

Nothing earth shattering so far, as you can see. We are simply adding yet another form, like the ones we've done before.

Last of all, the view function that ties everything together is expanded to handle the form (file app/views.py):

from forms import LoginForm, EditForm, PostForm
from models import User, Post

@app.route('/', methods=['GET', 'POST'])
@app.route('/index', methods=['GET', 'POST'])
@login_required
def index():
    form = PostForm()
    if form.validate_on_submit():
        post = Post(body=form.post.data, timestamp=datetime.utcnow(), author=g.user)
        db.session.add(post)
        db.session.commit()
        flash('Your post is now live!')
        return redirect(url_for('index'))
    posts = [
        { 
            'author': {'nickname': 'John'}, 
            'body': 'Beautiful day in Portland!' 
        },
        { 
            'author': {'nickname': 'Susan'}, 
            'body': 'The Avengers movie was so cool!' 
        }
    ]
    return render_template('index.html',
                           title='Home',
                           form=form,
                           posts=posts)

Let's review the changes in this function one by one:

  • We are now importing the Post and PostForm classes
  • We accept POST requests in both routes associated with the index view function, since that is how we will receive submitted posts.
  • When we arrive at this view function through a form submission we insert a new Post record into the database. When we arrive at it via a regular GET request we do as before.
  • The template now receives an additional argument, the form, so that it can render the text field.

One final comment before we continue. Notice how after we insert a new Post into the detabase we do this:

return redirect(url_for('index'))

We could have easily skipped the redirect and allowed the function to continue down into the template rendering part, and it would have been more efficient. Because really, all the redirect does is return to this same view function to do that, after an extra trip to the client web browser.

So, why the redirect? Consider what happens after the user writes a blog post, submits it and then hits the browser's refresh key. What will the refresh command do? Browsers resend the last issued request as a result of a refresh command.

Without the redirect, the last request is the POST request that submitted the form, so a refresh action will resubmit the form, causing a second Post record that is identical to the first to be written to the database. Not good.

By having the redirect, we force the browser to issue another request after the form submission, the one that grabs the redirected page. This is a simple GET request, so a refresh action will now repeat the GET request instead of submitting the form again.

This simple trick avoids inserting duplicate posts when a user inadvertently refreshes the page after submitting a blog post.

Displaying blog posts

And now we get to the fun part. We are going to grab blog posts from the database and display them.

If you recall from a few articles ago, we created a couple of fake posts and we've been displaying those in our home page for a long time. The fake objects were created explicitly in the index view function as a simply Python list:

    posts = [
        { 
            'author': {'nickname': 'John'}, 
            'body': 'Beautiful day in Portland!' 
        },
        { 
            'author': {'nickname': 'Susan'}, 
            'body': 'The Avengers movie was so cool!' 
        }
    ]

But in the last article we created the query that allows us to get all the posts from followed users, so now we can simply replace the above with this (file app/views.py):

    posts = g.user.followed_posts().all()

And when you run the application you will be seeing blog posts from the database!

The followed_posts method of the User class returns a sqlalchemy query object that is configured to grab the posts we are interested in. Calling all() on this query just retrieves all the posts into a list, so we end up with a structure that is very much alike the fake one we've been using until now. It's so close that the template does not even notice.

At this point feel free to play with the application. You can create a few users, make them follow others, and finally post some messages to see how each user sees its blog post stream.

Pagination

The application is looking better than ever, but we have a problem. We are showing all of the followed posts in the home page. What happens if a user has a thousand followed posts? Or a million? As you can imagine, grabbing and handling such a large list of objects will be extremely inefficient.

Instead, we are going to show this potentially large number of posts in groups, or pages.

Flask-SQLAlchemy comes with very good support for pagination. If for example, we wanted to get the first three followed posts of some user we can do this:

    posts = g.user.followed_posts().paginate(1, 3, False).items

The paginate method can be called on any query object. It takes three arguments:

  • the page number, starting from 1,
  • the number of items per page,
  • an error flag. If True, when an out of range page is requested a 404 error will be automatically returned to the client web browser. If False, an empty list will be returned instead of an error.

The return value from paginate is a Pagination object. The items member of this object contains the list of items in the requested page. There are other useful things in the Pagination object that we will see a bit later.

Now let's think about how we can implement pagination in our index view function. We can start by adding a configuration item to our application that determines how many items per page we will display (file config.py):

# pagination
POSTS_PER_PAGE = 3

It is a good idea to have these global knobs that can change the behavior of our application in the configuration file all together, because then we can go to a single place to revise them all.

In the final application we will of course use a much larger number than 3, but for testing it is useful to work with small numbers.

Next, let's decide how the URLs that request different pages will look. We've seen before that Flask routes can take arguments, so we can add a suffix to the URL that indicates the desired page:

http://localhost:5000/         <-- page #1 (default)
http://localhost:5000/index    <-- page #1 (default)
http://localhost:5000/index/1  <-- page #1
http://localhost:5000/index/2  <-- page #2

This format of URLs can be easily implemented with an additional route added to our view function (file app/views.py):

from config import POSTS_PER_PAGE

@app.route('/', methods=['GET', 'POST'])
@app.route('/index', methods=['GET', 'POST'])
@app.route('/index/<int:page>', methods=['GET', 'POST'])
@login_required
def index(page=1):
    form = PostForm()
    if form.validate_on_submit():
        post = Post(body=form.post.data, timestamp=datetime.utcnow(), author=g.user)
        db.session.add(post)
        db.session.commit()
        flash('Your post is now live!')
        return redirect(url_for('index'))
    posts = g.user.followed_posts().paginate(page, POSTS_PER_PAGE, False).items
    return render_template('index.html',
                           title='Home',
                           form=form,
                           posts=posts)

Our new route takes the page argument, and declares it as an integer. We also need to add the page argument to the index function, and we have to give it a default value because two of the three routes do not have this argument, so for those the default will always be used.

And now that we have a page number available to us we can easily hook it up to our followed_posts query, along with the POSTS_PER_PAGE configuration constant we defined earlier.

Note how easy these changes are, and how little code is affected each time we make a change. We are trying to write each part of the application without making any assumptions regarding how the other parts work, and this enables us to write modular and robust applications that are easier to test and are less likely to fail or have bugs.

At this point you can try the pagination by entering URLs for the different pages by hand into your browser's address bar. Make sure you have more than three posts available so that you can see more than one page.

Page navigation

We now need to add links that allow users to navigate to the next and/or previous pages, and luckily this is extremely easy to do, Flask-SQLAlchemy does most of the work for us.

We are going to start by making a small change in the view function. In our current version we use the paginate method as follows:

posts = g.user.followed_posts().paginate(page, POSTS_PER_PAGE, False).items

By doing this we are only keeping the items member of the Pagination object returned by paginate. But this object has a number of other very useful things in it, so we will instead keep the whole object (file app/views.py):

posts = g.user.followed_posts().paginate(page, POSTS_PER_PAGE, False)

To compensate for this change, we have to modify the template (file app/templates/index.html):

<!-- posts is a Paginate object -->
{% for post in posts.items %}
<p>
  {{ post.author.nickname }} says: <b>{{ post.body }}</b>
</p>
{% endfor %}

What this change does is make the full Paginate object available to our template. The members of this object that we will use are:

  • has_next: True if there is at least one more page after the current one
  • has_prev: True if there is at least one more page before the current one
  • next_num: page number for the next page
  • prev_num: page number for the previous page

With these for elements we can produce the following (file app/templates/index.html):

<!-- posts is a Paginate object -->
{% for post in posts.items %}
<p>
  {{ post.author.nickname }} says: <b>{{ post.body }}</b>
</p>
{% endfor %}
{% if posts.has_prev %}<a href="{{ url_for('index', page=posts.prev_num) }}">&lt;&lt; Newer posts</a>{% else %}&lt;&lt; Newer posts{% endif %} | 
{% if posts.has_next %}<a href="{{ url_for('index', page=posts.next_num) }}">Older posts &gt;&gt;</a>{% else %}Older posts &gt;&gt;{% endif %}

So we have two links. First we have one labeled "Newer posts" that sends us to the previous page (keep in mind we show posts sorted by newest first, so the first page is the one with the newest stuff). Conversely, the "Older posts" points to the next page.

When we are looking at the first page we do not want to show a link to go to the previous page, since there isn't one. This is easy to detect because posts.has_prev will be False. We handle that case simply by showing the same text of the link but without the link itself. The link to the next page is handled in the same way.

Implementing the Post sub-template

Back in the article where we added avatar pictures we defined a sub-template with the HTML rendering of a single post. The reason we created this sub-template was so that we can render posts with a consistent look in multiple pages, without having to duplicate the HTML code.

It is now time to implement this sub-template in our index page. And, as most of the things we are doing today, it is surprisingly simple (file app/templates/index.html):

<!-- posts is a Paginate object -->
{% for post in posts.items %}
    {% include 'post.html' %}
{% endfor %}

Amazing, huh? We just discarded our old rendering code and replaced it with an include of the sub-template. Just with this, we get the nicer version of the post that includes the user's avatar.

Here is a screenshot of the index page of our application in its current state:

microblog profile page

The user profile page

We are done with the index page for now. However, we have also included posts in the user profile page, not posts from everyone but just from the owner of the profile. To be consistent the user profile page should be changed to match the index page.

The changes are similar to those we made on the index page. Here is a summary of what we need to do:

  • add an additional route that takes the page number
  • add a page argument to the view function, with a default of 1
  • replace the list of fake posts with the proper database query and pagination
  • update the template to use the pagination object

Here is the updated view function (file app/views.py):

@app.route('/user/<nickname>')
@app.route('/user/<nickname>/<int:page>')
@login_required
def user(nickname, page=1):
    user = User.query.filter_by(nickname=nickname).first()
    if user is None:
        flash('User %s not found.' % nickname)
        return redirect(url_for('index'))
    posts = user.posts.paginate(page, POSTS_PER_PAGE, False)
    return render_template('user.html',
                           user=user,
                           posts=posts)

Note that this function already had an argument (the nickname of the user), so we add the page number as a second argument.

The changes to the template are also pretty simple (file app/templates/user.html):

<!-- posts is a Paginate object -->
{% for post in posts.items %}
    {% include 'post.html' %}
{% endfor %}
{% if posts.has_prev %}<a href="{{ url_for('user', nickname=user.nickname, page=posts.prev_num) }}">&lt;&lt; Newer posts</a>{% else %}&lt;&lt; Newer posts{% endif %} | 
{% if posts.has_next %}<a href="{{ url_for('user', nickname=user.nickname, page=posts.next_num) }}">Older posts &gt;&gt;</a>{% else %}Older posts &gt;&gt;{% endif %}

Final words

Below I'm making available the updated version of the microblog application with all the pagination changes introduced in this article.

Download microblog-0.9.zip.

As always, a database isn't provided so you have to create your own. If you are following this series of articles you know how to do it. If not, then go back to the database article to find out.

As always, I thank you for following my tutorial. I hope to see you again in the next one!

Miguel

116 comments

  • #76 Miguel Grinberg said 2014-12-24T17:55:48Z

    @Gaurav: paginate() is a feature of Flask-SQLAlchemy, it is not available when you run queries directly through SQLAlchemy. Try to run the query through the model:

    new_posts = Post.query.filter_by(status=1).order_by(Post.post_id.DESC()) paginate_data = new_posts.paginate(1,1,False)

    The alternative is for you to look at the source code for the paginate() method in Flask-SQLAlchemy and write a similar query using SQLAlchemy filters.

  • #77 Jacopo Notarstefano said 2015-01-17T10:07:39Z

    Hi Miguel!

    First of all, thank you for this tutorial. Next month I'm starting a job working on a Flask application, and this tutorial is helping me immensely getting up to speed with this framework. I will definitely buy your book as a "thank you"!

    I only have a minor nitpick/clarification: shouldn't the "<<" and ">>" in the pagination navigation be "<<" and ">>" respectively? Or is Jinja inferring that those characters are not part of an HTML tag, thus converting them for us?

    Cheers, Jacopo Notarstefano

  • #78 Miguel Grinberg said 2015-01-18T06:42:14Z

    @Jacopo: interesting that I missed that, good catch! Jinja2 does not help with this, but I'm guessing the HTML renderer in the browser is smart enough to detect that "<<" does not indicate any valid HTML construct and renders it as text. The correct approach would be to use << and >> as you suggest. Thanks!

  • #79 Sven Hergenhahn said 2015-03-20T12:48:16Z

    Hi Miguel,

    you helped me immenseley with your book and the mega tutorial, so many thanks.

    I have one question:

    I use pagination on a page that has a form for filtering. So when I'm e.g. on index/3 and filter my data, then I end up on page index/3 again which is empty, because the filtered data doesn't fill three pages.

    I was able to kind of fix this by setting 'page = 1' in the view where the filtering case is handled. This will display the first paginated page, but the URL in the browser location line is still index/3.

    Is there a way to re-set the URL as well when clicking a button, i.e. set it to say index/1 ?

    I hope I was able to make myself clear. Thanks for any suggestions or ideas.

    Sven

  • #80 Miguel Grinberg said 2015-03-21T21:59:11Z

    @Sven: Are you using Ajax to update your table of data? If you are not, then the route that handles the filter POST request can return a redirect to index/1. I think in your current implementation you are not doing a redirect, that explains the URL not changing. Responding to POST requests with a redirect is also a good practice regardless of your URL problem, so I recommend that you do that. See http://en.wikipedia.org/wiki/Post/Redirect/Get for information about the POST/Redirect/GET pattern.

  • #81 Hyun Jun, An said 2015-03-30T09:44:14Z

    I guess this function doesn't work on py3.

    def followed_posts(self): return Post.query.join(followers, (followers.c.followed_id == \ Post.user_id)).filter(followers.c.follower_id == self.id). \ order_by(Post.timestamp.desc())

    i changed the self.id code to 1 of integer for testing.

    the return is this query code: SELECT post.id AS post_id, post.body AS post_body, post.timestamp AS post_timestamp, post.user_id AS post_user_id FROM post JOIN followers ON followers.followed_id = post.user_id WHERE followers.follower_id = :follower_id_1 ORDER BY post.timestamp DESC

    but if we wanna get some posts of followed, we should change from above query to below query.: SELECT post.id AS post_id, post.body AS post_body, post.timestamp AS post_timestamp, post.user_id AS post_user_id FROM post JOIN followers ON followers.followed_id = post.user_id WHERE followers.follower_id = 1 ORDER BY post.timestamp DESC

    so how can i fix it?

  • #82 Miguel Grinberg said 2015-03-31T00:36:58Z

    @Hyun: what is the difference? The self.id is an argument that is inserted in the query at run time. Seems the two queries are identical to me.

  • #83 Peng Xu said 2015-05-29T06:05:25Z

    Really nice article! Thanks very much!

  • #84 Sven Hergenhahn said 2015-07-09T11:37:13Z

    Hi Miguel,

    re #80 - I finally came around implementing this - it was actually only very few lines I had to change but assinging the values from the form to the session keys and then using a redirect(url_for('.index')) solved all my problems I had with the form - so - thanks a lot, you are doing a tremendous job here!

    Cheers, Sven

  • #85 Dirk said 2015-10-18T14:33:30Z

    Hi Miguel, thank you very much for that awesome tutorial!

    I do have a question: I have linked the username on the post.html template to link to the user profile page. Is there a possibility to avoid these links, if this is displayed on the profile page? So in the normal overview I can access all profiles of the users, having posted something, but on their profile page, it would link to the profile I'm currently viewing.

    Kind regards Dirk

  • #86 Miguel Grinberg said 2015-10-18T22:29:35Z

    @Dirk: yes, you can add a conditional in the template, that checks if the user profile being viewed is the one of the logged in user, and in that case suppress the link.

  • #87 Edward said 2015-10-20T18:46:30Z

    I'm having some issues with flickr login After logging in for the first time with flickr yesterday after attempting to loginthe page now refreshes and does not log me in. I can still login fine with Yahoo. Thanks for the help

  • #88 Miguel Grinberg said 2015-10-22T05:18:45Z

    @Edward: can't really say what the problem is from your description. You will need to debug it to find out why the server isn't logging you in.

  • #89 Neil said 2015-11-11T22:36:44Z

    Thank you for the excellent tutorial--it's a very nice crash course on good practices in backend development!

    I'm a bit confused about the use of "user" in the html templates. At the moment, my "base.html" template has the following block: {% if user %} Hello {{ user.nickname }}! {% else %} no user :( {% endif %} If I'm logged in, on the index page, the title will be "no user :(", although "Hi, Neil!" displays correctly. On the 'user' page, the title will be 'Hello Neil!'. If I change "user" to "g.user" in 'base.html', I will see "Hello Neil!" as the title on all pages. However, if no one is logged in, I see "Hello !". I am curious about when to use g.user (as you do in index.html) vs. user (as in user.html, like in the nickname you pass in {% if posts.has_prev %}<< Newer posts{% else %}<< Newer posts{% endif %} | Where is the "user" object in user.html coming from?

  • #90 Miguel Grinberg said 2015-11-12T08:00:12Z

    The "user" variable used in templates is passed in the render_template call. Any keyword arguments given in render_template() after the template name are variables that are made available within the template. The "g" variable from Flask is automatically available in all templates, so using g.user is a good way to get the logged in user. In the profile page, however, you could be looking at someone else's profile page, so you need the "user" variable.

  • #91 Neil said 2015-11-13T22:44:49Z

    Thank you--especially for replying to a comment on a 3-year-old blog post! I found that soon after posting the comment. Out of curiosity, what is the best practice for checking whether there is NO user logged in--just g.user.is_authenticated? I noticed that {% if g.user %} always evaluates to True, which makes sense given the way Flask handles current_user.

  • #92 Miguel Grinberg said 2015-11-14T06:45:54Z

    @Neil: g.user.is_authenticated is the correct way to check for a logged in user.

  • #93 zelinlee0303 said 2015-12-09T17:41:56Z

    Hi! Miguel,I have some problems with this part. when I run "python run.py" and then sign in the blog. The errors will come out, like below 127.0.0.1 - - [10/Dec/2015 01:30:36] "GET /user/John HTTP/1.1" 500 - Traceback (most recent call last): File "C:\Python27\lib\logging\handlers.py", line 930, in emit smtp = smtplib.SMTP(self.mailhost, port, timeout=self._timeout) File "C:\Python27\lib\smtplib.py", line 256, in init (code, msg) = self.connect(host, port) File "C:\Python27\lib\smtplib.py", line 316, in connect self.sock = self._get_socket(host, port, self.timeout) File "C:\Python27\lib\smtplib.py", line 291, in _get_socket return socket.create_connection((host, port), timeout) File "C:\Python27\lib\socket.py", line 575, in create_connection raise err error: [Errno 10061] Logged from file app.py, line 1423

    hope to have your help:) Thanks so much!

  • #94 Miguel Grinberg said 2015-12-10T18:13:35Z

    @zelinlee0303: the error that you pasted indicates that the system was trying to email you a stack trace for a failure and it could not get a SMTP connection.

  • #95 Matt Bymaster said 2015-12-27T02:32:20Z

    My profile posts under the "Your Profile"(user.html) link are posted from oldest down to newest where I believe the intention would be to have them reversed yielding Newest timestamp to Oldest timestamp. Also the "Older posts >>" link takes me to the newest posts chronologically.

    Here's what I'm getting as I've added a timestamp: Anacrust says @ 2015-12-27 01:09:08.486873: Star Wars was Awesome!!! Test 1 Anacrust says @ 2015-12-27 01:32:29.443756: Star Wars was Awesome!!! Anacrust says @ 2015-12-27 01:44:39.066473: you big dummy

    and after clicking the "Older posts >>" link Anacrust says @ 2015-12-27 01:47:17.443056: yada

    In the app->templates->user.html file, I tried adding a reversed() built-in python function to read the posts.items in reverse but it's throwing an error.

    revised app->templates->user.html file code: ... {% for post in reversed(posts.items) %} {% include 'post.html' %} {% endfor %} ...

    Error: Traceback (most recent call last): File "/usr/lib/python2.7/logging/handlers.py", line 928, in emit smtp = smtplib.SMTP(self.mailhost, port, timeout=self._timeout) File "/usr/lib/python2.7/smtplib.py", line 256, in init (code, msg) = self.connect(host, port) File "/usr/lib/python2.7/smtplib.py", line 316, in connect self.sock = self._get_socket(host, port, self.timeout) File "/usr/lib/python2.7/smtplib.py", line 291, in _get_socket return socket.create_connection((host, port), timeout) File "/usr/lib/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused Logged from file app.py, line 1423 127.0.0.1 - - [26/Dec/2015 21:11:29] "GET /user/Anacrust HTTP/1.1" 500 -

    Thank you for your time!

  • #96 AP said 2015-12-31T16:03:45Z

    I'm noticing that on the index page the order of the posts goes newest->oldest but on the user profile page it goes oldest->newest (making the navigation out of order). What would be causing this for me? Aren't both ultimately displayed the same way? Is it because of how paginate orders the list?

  • #97 Miguel Grinberg said 2016-01-03T22:48:44Z

    @Matt: there are a couple of issues here. The "Connection refused" stack trace is, I think, caused by a bad email configuration. The application errored, and is configured to send a stack trace of the error by email, and that failed. So that stack trace is about the email failure, not the original problem.

    To display the messages in reverse order using reversed(items) is not a good solution, since it will not work for paginated results. The best way to do that is to change the SQLAlchemy query to return the results in the order you want. The "order_by" method in the Post.query object should do that easily.

  • #98 Miguel Grinberg said 2016-01-03T22:52:48Z

    @AP: paginate just returns the items in the order specified in the query. As I said in comment #97 above, you can set up the SQLAlchemy query to return the results in your preferred order.

  • #99 Job Draku said 2016-04-27T12:46:42Z

    hi, Mr.Miguel; am faced with a challenge. i have been following this tutorial but am stuck when i reached the data base section. when i created the database, the python shell complains that "----no module named app--"; i tried to figure it out but i failed. is there any way you can help me? i will be grateful

  • #100 Miguel Grinberg said 2016-04-28T14:23:48Z

    @Joe: Need more information. I recommend that you write all the details on stack overflow, including complete stack traces of your error(s).

Leave a Comment