Standard Library Alternatives To A Database Package for Light Web Applications

I've put over a dozen small Python web applications into service over the past year, either personally handling the development or overseeing the project as a business owner. Many new developers instinctively reach for an established database package as part of the design, usually accompanied with an ORM. But do we really need this complexity?

Consider the following options, all present in your Python Standard library (present at every install)...

1) Many python web frameworks (such as create a service vs. execute a script (like PHP does); this provides you with the ability to maintain information in memory between webpage requests. For a read-only application, have you considered uploading a text-file of your data to the server and loading it into memory as part of the application startup process? Cost: one line of code (a read operation and a list comprehension to unpack the data).

2) Do you need to write data which will rarely be read? That would be an ideal time to consider the logging module. This is ideal for click stream data, since it doesn't incur the overhead of a database operation.

3) Is your application expecting only small volume or a very limited number of concurrent users? This is surprisingly common with many departmental web apps; we may need the site for productivity or security reasons, but only expect 5 - 6 people to use it per day [assume they do something really high value, like managing work orders or project tasks]. This is an ideal application for SQLite (larger list of use cases here). According to the project's documentation, SQLite has been demonstrated to work well with any website that receives <100K hits/day (and supposedly has performed reasonably well for websites that get 10X that traffic); statistically, this pretty much encompasses most websites currently in production. This small database is embedded in the Python standard installation and can be accessed via the sqlite3 module.

So there you have it...three simple solutions to managing the data layer of a small web application, all of which can be implemented using a trivial amount of standard python and modules from the standard library. There are even ORM's that work for SQLite, if you MUST incorporate them...

Do they work for everything? Nope...and sometimes you need the additional features of a larger database such as Postgres or MySQL. But for the typical, "medium value" departmental web application, reference tool, or calculator - you can frequently get away with using one of these lighter alternatives.

Leave comments

  • That is a good insight!
    For 1) how do you maintain information in memory between web requests? Cookies is one way to do it but it is not secure or efficient. Sessions is a a better approach, but there is no a standard library, is there?

    • Gary
  • You're running a process in least for Keep it as a variable, like any other...

    • number_cruncher
  • Interesting. But how do you mange life-cycle of such variable?
    If variable is set for user U1 and then later user U2 connects to the same process, how to make sure he does not end up with U1 user state?

    • Gary
  • Ah - for that I would use sessions. Implementation would vary by framework. Data held in memory would also reset if you ever restart your server process, which can be relatively common.

    The memory trick works best for "read-only" data. If you're building a CRUD application with user login, you want sessions and a database.

    Don't get me wrong...there is a point where you should jump to a full database, in terms of volume and complexity. Write sessions to a DB, etc. But you don't need this for small/simple apps...

    • number_cruncher
  • This is quite an enlightenment. I seldom think beyond mysql (well, I was once a *AMP developer).
    But as for now I create a web application using plain TEXT file, well, it serves it purpose!

    The idea of SQLite is intriguing. I'll give it a thought!
    * although for concurrent write activity, we may have trouble using SQLite, right?

    • eko

Copyright(c) 2017 -
By using this website, you signify your acceptance of Terms and Conditions and Privacy Policy
All rights reserved