This page looks plain and unstyled because you're using a non-standard compliant browser. To see it in its best form, please upgrade to a browser that supports web standards. It's free and painless.

Python WebDev Challenge: The Three Hour App

As my life picks up and I find myself spending more time working on management and PowerPoint rather than code, I occasionally need a detox session. It's good to stay connected with where you came from.

So something I've started doing, just for fun, is trying to develop small but usable applications from scratch in short time window. The idea of a one hour application sounds fun but building a non-trivial project in that time window usually requires a lot of pre-planning. So to keep things honest and not too much like work, I bumped the window out to three hours and set the following three rules:

1 - Use of frameworks and basic templates is allowed, particularly for things such as JavaScript and responsive design (CSS is the bane of my existence (as a perfectionist), that portion of the easily consumes a couple of hours once things get going).

2 - Planning is limited to my lunchtime walk

3 - A second at the project on a subsequent evening is acceptable if you're building completely separate portions of the site

The three hour time limit was dictated by the practical realities of fatherhood. The kids aren't asleep until 8 or 9, I need to head to bed by 11 to midnight. So that gives me a very finite block of time to get stuff done.

One of my first three projects was a quick and dirty hack on a word list generator. This actually turned into three separate projects, two Python and one PHP, but I was able to keep true to principle #3 (each must be separate, each must be done in an evening). The first portion of this loaded a large dictionary into a SQLite database and set up a bottle.py server with some wildcard routes for searching it. For example, to find words beginning with a string of letters, the syntax was: /begin-with/<letters>. For a sample output, I present a list of the words beginning with python (/begin-with/python). But the project was set up so that any string could be fed into the database (if you don't have a problem with getting no results or obscure results).  The front end was stock JQuery Mobile - responsive design right out the box and you can get B+ appearance work by cutting and pasting their demo's - highly recommend looking at this as a first front end framework if you haven't done web design before. Elapsed time: three hours (I was out of town in a hotel room travelling for work that night). I actually wound up recycling most of the code into the starting point for a work-related CRUD app a few months later - JQuery Mobile / SQLite / bottle.py is a very teachable toolset for new developers.

Second challenge, was to extend the pattern matching lookup into a tool to unscramble words (link to main page, which is where this piece of the project got bolted onto the site). This turned out to be even easier than expected. I was able to find a simple way to tweak the database to support the different search - the trick came down to splitting the string of letters into parts and running an efficient search. Itertools to the rescue. Elapsed time - two hours (was able to recycle the database load script). These projects also had my favorite feature of using Bottle.py with 100% standard library modules - the "drag and drop" ftp deploy. Laugh if you will, but being able to dump a project onto a webhost in 3 minutes and just start a process (with Webfaction, it really is that easy; your mileage may vary elsewhere) is a huge advantage.

At this point, the building blocks from a front end and database perspective were getting pretty familiar. So...I had a spare PHP hosting account lying around and an extra domain. Now that I understood the problem, how hard would it be to replicate it using PHP? Turns out.... pretty dang easy... SQLITE3 is an integral module in core PHP so the project came down to writing a twenty line PHP loop to replicate the itertools module functionality we needed to split up the original string and dropping the SQL from our original Python database manager module into a PHP cousin. Boom, a cross-platform solution (demo is here). Completed in about three hours since my PHP was very rusty.

I noticed a couple of other interesting things along the way:

1) Modern web development isn't a one tool skill; note the need for front end (JQuery) and database proficiency, along with Python

2) Once you fully understand a problem in one language, it's not very hard to replicate it in another. This matters when hiring or deciding to re-platform an application.

From a professional / job perspective: a) learn all the pieces and b) when hiring, don't get locked down into one language if you see someone has done non-trivial work in a similar space, and c) the best time to replatform something in a new language or system is when the original architect is still with you, even if they don't know the new technology. Many tools are starting to converge.

Posted by numbercruncher, 07 February 2016 15:25 | Python, word games | Comment (0) | Permalink | Trackbacks (0)

Standard Library Alternatives To A Database Package for Light Web Applications

I've put over a dozen small Python web applications into service over the past year, either personally handling the development or overseeing the project as a business owner. Many new developers instinctively reach for an established database package as part of the design, usually accompanied with an ORM. But do we really need this complexity?

Consider the following options, all present in your Python Standard library (present at every install)...

1) Many python web frameworks (such as bottle.py) create a service vs. execute a script (like PHP does); this provides you with the ability to maintain information in memory between webpage requests. For a read-only application, have you considered uploading a text-file of your data to the server and loading it into memory as part of the application startup process? Cost: one line of code (a read operation and a list comprehension to unpack the data).

2) Do you need to write data which will rarely be read? That would be an ideal time to consider the logging module. This is ideal for click stream data, since it doesn't incur the overhead of a database operation.

3) Is your application expecting only small volume or a very limited number of concurrent users? This is surprisingly common with many departmental web apps; we may need the site for productivity or security reasons, but only expect 5 - 6 people to use it per day [assume they do something really high value, like managing work orders or project tasks]. This is an ideal application for SQLite (larger list of use cases here). According to the project's documentation, SQLite has been demonstrated to work well with any website that receives <100K hits/day (and supposedly has performed reasonably well for websites that get 10X that traffic); statistically, this pretty much encompasses most websites currently in production. This small database is embedded in the Python standard installation and can be accessed via the sqlite3 module.

So there you have it...three simple solutions to managing the data layer of a small web application, all of which can be implemented using a trivial amount of standard python and modules from the standard library. There are even ORM's that work for SQLite, if you MUST incorporate them...

Do they work for everything? Nope...and sometimes you need the additional features of a larger database such as Postgres or MySQL. But for the typical, "medium value" departmental web application, reference tool, or calculator - you can frequently get away with using one of these lighter alternatives.

Posted by numbercruncher, 04 December 2013 22:02 | Python | Comment (5) | Permalink | Trackbacks (0)

IDLE Easter Egg...

Presented without comment, aside from the fact that this made me laugh on an otherwise frustrating day... 

1
import antigravity

Update: I found an article talking about this, with a comment from the BDFL...

Posted by numbercruncher, 12 October 2012 09:57 | Python | Comment (2) | Permalink | Trackbacks (0)

Comprehending List Comprehensions

List comprehensions are often a bit intimidating to developers who are new to Python. They are really much friendlier than that :) It's just a way to repackage that pile of for-loops, if-then statements, and calculation logic which is littering your current code. Here's a simple example to put them in perspective.

Assume I have a list of data - (1,2,3,4,5,6....n).

Now, there are a couple of very common "building block" transformations which we can apply to this dataset. I can create a new list by doing each of the following (alone or in combinator with each other):

Notice something in common here? We're defining a mathematical operation and applying it to each element of a list.

What Python construct do we use to encapsulate mathematical operations? Yes, You in the back. Right, Functions...

What if there were a way to succinctly summarize a mathematical operation as a function and apply it to each element of a Python list. Guess what, you're in luck. This is where Python's functional programming constructs come into play - which include two functions that are designed to take a list and another function (as arguments) and create a new list by applying that function to every element of the intial list. These two functions are:

Entry level list comprehensions are just a one-liner way of expressing this.

Consider the following:

1
2
3
4
initial_list = [1,2,3,4,5] 
 
 
my_func  = lambda x: x * x

We can use for-loops and if-then logic to create a new list using the following in Python or - with minor syntax edits - many other languages, such as Visual Basic and C / C++:

1
2
3
4
5
6
7
8
9
10
new_list = [] 
 
 
for element in initial_list: 
 
 
   if element > 5: 
 
 
       new_list.append(my_func(element))

There's nothing obviously wrong with this - I mean, we get to write lots of "lines of code" so the boss feels good, right? 

If we wanted to take advantage of Python's functional programming tools, we could also write this as:

1
new_list = map(my_func,filter(lambda x:x>5,  initial_list))

Yuck...we've got it down to one line but now I have to think about it when I read it!

Or you could express it as a one line list comprehension:

1
new_list = [my_func(item) for item in initial_list if item > 5]

Ahh...much easier to parse.

The appeal of using a list comprehension is your code is more succinct and easy to read.

This is a simple example. I recommend you take a look at the Python docs on functional programming and list comprehensions for a much richer view of the many things you can do with this tool. But hopefully now you can map this concept back to a familiar technique.

Posted by numbercruncher, 21 September 2012 21:46 | Python | Comment (2) | Permalink | Trackbacks (0)

A Quiet Sense of Accomplishment - First GAE Application Using Bottle.py

Now We Have Working Code On GAE! 

I'm going to bed slightly later than expected tonight, but with a quiet sense of accomplishment.

As of about 30 minutes ago, I just uploaded and started a barebones version of my first Google App Engine app - a lemonade stand revenue calculator (editorial update: was originally website revenue calculator, had a more creative idea this morning). Enter some assumptions about your lemonade stand and it will predict how much money per hour you should be making. I'm using bottle.py as the framework behind the site and the application is a comic twist on my website revenue estimator (the original Non-GAE implementation is here, underlying business research is here).

Why I'm Interested In This...

My day job is in Analytics, where my team and I spend a lot of time coming up with "the formula" behind a business process. Once we have "the formula", we have three challenges:

Historically, analytics has always retreated "to our mountain top" to mine insights from data. Our ideas would then be shared with senior management, who would frequently broadcast them to their subordinates. Net result: everyone knows everything (fail goal #3), we track nothing everything is ad-hoc (fail #2), and we struggled with the first challenge.

The irony is that we're only talking about a formula: something that can be implemented in a handful of lines of Python!

Thus my interest in Python microframeworks. Bottle is a microframework that simplifies the task of setting up a basic web service. In simple terms, you define URL routes (http://mysite.com/path/variable) for a specific request; these are mapped to a python function (passing information from the request) which creates an appropriate response. Many microframeworks are very AJAX friendly (very easy to return data as JSON), opening the door to building simple web applications.

So... if I can code the formula in Python, I can deliver it as a webservice (or simple AJAX application). This protects the formula and will make the formula's implementation easier to use and supervise. One little bonus for bottle and GAE - they have a Google App Engine Adapter. This made the process a snap. Oh...and GAE has a free plan for smaller applications (see them for details). Which makes it attractive to students and hobbyist developers.

I've got a lot of polishing to do (particularly on the web design piece). Interestingly enough, the calculator framework has been relatively easy to edit - I was able to flip the application's entire purpose over lunch (from a website profit estimator into the lemonade stand version). It's just a form and a formula - very easy to edit!

Posted by numbercruncher, 12 September 2012 21:03 | General, bottle.py, Google App Engine, calculators | Comment (3) | Permalink | Trackbacks (0)

Python for Newbies: A Simple Anagram Finder

Once you get past Hello World, there are a several good entry points for a new Python developer to get acquainted with the language.  Three areas which really helped me were simple math problems, the Python challenge, and writing some word game solvers.

The best source of basic math problems for a new python developer is Project Euler. This site is basically an online math test - where you need to write a small program to perform the required calculations. Most of these problems can be solved by a new developer in under an hour (some faster). It will broaden your understanding of mathematics and give you a good grounding in basic Python.

The Python Challenge is another good learning environment. The participant is presented with a web page that contains a riddle, with clues buried somewhere on the page. Your job is to use Python to solve the riddle. This generally requiring you to take a deep dive into one of the modules in the Standard Library. We've used this as a warmup exercise in one of the meetups I attend - I can recall doing modules around regular expressions, urllib (simple web scraping), and image processing.

My deepest dive into the language occured when I started doing some "code doodling" around solving word puzzles. For example, the script below takes a list of words and searches a dictionary to identify their anagrams. For a dictionary, I opted to use Enable (Google hosts a copy of it - look for enable.txt on the Google Code directory behind this link). 

A couple of insights made this script possible:

Since it's five o'clock somewhere, I opted to run this script against a list of beverages.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
testlist = ["wine","beer","mead","liquor","scotch","gin","whiskey","rum"]
 
 
def gethandle(word):
    return ("").join(sorted(word))
 
 
anagrams={}
 
 
wordlist =open("enable1.txt")
for item in wordlist:
     handle = gethandle(item.strip())
     if handle not in anagrams:
         anagrams[handle] = [item.strip()]
     else:
         anagrams[handle].append(item.strip())
 
 
def testforanagrams(word):
    results = anagrams.get(gethandle(word),[])
 
 
    if len(results)>1:
          print word, ":",", ".join([item for item in results if item != word])
    else:
          print word, ":", "None"
 
 
map(testforanagrams,testlist)

producing the following output:

1
2
3
4
5
6
7
8
wine : None
beer : bree
mead : dame, made
liquor : None
scotch : None
gin : None
whiskey : None
rum : None

This is a fairly basic and slightly verbose script and there are a number of ways to improve upon it. I'll suggest a couple of the more obvious ones, the implementation of which is left as an exercise for the student.

First, it could benefit by reorganizing all of the code into proper functions. Several elements of the script execute sequentially with the script vs. being packaged into reusable functions. I'd actually recommend reorganizing the entire script into a class object. This neatly encapsulates the above activities and helps you separate the act of loading the dataset from searching the dataset. From a performance perspective, you spend a lot of IO and CPU time parsing and loading the initial dictionary. If you operated this as some type of "service", you should leverage this setup time across as many requests as possible.

On a personal note, this little script actually got me started on a larger journey - involving two additional steps.

Once I had a basic anagram finder, I started looking at adding other word games into the package. This forced me to do some optimization around the underlying algorithm using several articles I found on Python data structures and processing performance. In the process, I was able to learn a lot about how to benchmark the memory usage and CPU speed for a Python program. While the final product was much more complicated, it was a lean, mean "word searching machine".

While I was building the class object, I was also learning about several python web frameworks for work and wanted to find a safe test project. Once I built the word search class object, I used a web framework to link a URL Route (/solveword) to a class method (solve-word function). This provided me with the server-side chassis for a word game solver website (the first "production grade" solver I wrote is here, a fancier version of the solver is here). I gave a talk on this project at my local Python meetup (overview and slides are here). This series of steps actually prompted me to learn a lot more than Python - HTML, CSS, Jquery, basic devops, etc. 

Don't be afraid of taking little steps when learning a new language. Little steps and silly problems (eg. anagram finding, finding palindromes, many of the other Project Euler exercises) are frequently the gateway to working on more serious applications. Having "code in production" tends to prompt you to create more code - forcing you to learn more than you ever expected about topics you never knew existed until they were suddenly relevant to bringing your baby to life. Silly little steps sometimes can turn into respectable deliverables....

P.S. And don't be afraid to play around with projects that seem hopeless from an adoption perspective. I wrote the website's scrabble solver purely to round out the collection of solvers; while that particular solver received zero public love (too many people already build better solvers and are way too intense about promoting them), the site actually gets traffic from a simpler form of the scrabble solver that was rebranded into tools to unscramble words and serve as a word jumble solver. Look for adjacent spaces and see if your program works there; you may be very surprised at the outcome.

Addendum (three years later): I've apparently found my version of a code kata project. I've actually replicated this project about four times since then. First version was done in two three hour sprints as a challenge (the word unscrambler implements the same algorithm above, except as a database). And replicated it using PHP (version 1 is here and another version with more polished internals is here).

Posted by numbercruncher, 08 September 2012 12:44 | Python, word games | Comment (2) | Permalink | Trackbacks (0)

A Very Silly Introduction to Android SL4A's Text-To-Speech Functionality

Got an Android Phone?

Missing Python when you're separated from your laptop? Need to script any type of routine tasks that your run on your phone?

Then you should check out Google's Android SL4A Python toolkit. Installing this application gives you:

Google's download includes a small editor which you can run from your phone. You can create and edit Python scripts from your phone. The UI is decent - primarily limited by the device itself (hard to type on a phone with a virtual keyboard). If you're going to do any real coding, I recommend developing the modules on a larger machine and porting them over.

I was pleasantly surprised by the power of the device - for example, you've got the ability to run a webserver on your phone (I wrote a quick tutorial about this). I've been able to run several decent sized analytical processing jobs on the device. It's also a decent platform for simple web-scraping and web service calls.

Since you're running a script on a single-screen device, I'd advise reading up on the docs for the multi-processing module. You will need this if you're going to try running a localhost web server from your phone (very useful); you will want to rig your scripts to start the webserver then use the Android API to open a browser.

The installation process comes with a nice library of samples. One of the first programs I wrote for this was a tweak on these samples which used Android's text-to-speech API. Code below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import android
import random
 
 
droid = android.Android()
 
 
spam_lines=["Do you like green eggs and spam?",
              "I do not like them, Sam-I-am.",
              "I do not like green eggs and spam.",
              "Would you like them Here or there?",
              "I would not like them here or there.",
              "I would not like them anywhere"]
 
 
droid = android.Android()
 
 
droid.ttsSpeak(random.choice(spam_lines))

Start the script and you will be treated to a random Dr. Suess line. Actually, this simple Hello World example could be adapted to use anything you wanted to put in the list: management bromides, favorite sayings, marriage proposals, etc...

Posted by numbercruncher, 08 September 2012 11:38 | Python, sl4a | Comment (0) | Permalink | Trackbacks (0)

About Us: Greetings & Introductions

Greetings!

By day I'm an analytics manager, specializing in marketing and pricing. After hours, I like to play with Python, PHP, and Javascript. Welcome to my first attempt at a purely technical blog.

I've been using Python for about five years now and it has been my language of choice for the past two. I got my start using Python for data analysis - I started using it for pretty much anything you could not do using out-of-the-box SQL. Since then, I have started using it for simple web applications (primarily calculators / solvers and CRUD database apps) and to extend my Android phone's functionality.

From a content perspective, this blog is targeted at new Python users - particularly those in the analytics field.

Posted by numbercruncher, 08 September 2012 10:43 | General | Comment (0) | Permalink | Trackbacks (0)

Powered by LT
Valid XHTML 1.0 Strict - Valid CSS