- back-end 2017-03-19T16:58:15+01:00 https://fadeit.dk/blog/tag/back-end.html Firebase advanced data modelling and role based authorization 2016-08-25T16:10:30+02:00 https://fadeit.dk/blog /2016/08/25/firebase-advanced-data-modelling-and-role-based-authentication-authorization <p>It’s been almost 5 years since Firebase was released. It looked cool from the beginning, but I never got to use it in a project. <br /> That’s going to change today, because I am set on using it and there’s no turning back.</p> <p>I’m not experienced with it, so this is just my idea of how to handle things. If you have a better idea, feel free to share so we can all become better at Firebasing!</p> <h2 id="project-requirements">Project requirements</h2> <p>Before diving deep into how I modelled my data, it’s important to understand what are the entities involved, how they relate to each other &amp; what kind of roles I have to deal with.</p> <p>I’ll try not to go into details; they shouldn’t affect my general approach (the stupid devil is <strong>*always*</strong> in the detail).</p> <h4 id="entities">1. Entities</h4> <p>I called the main objects in my real-time database entities: name them as you wish, they are just top-level creatures. The main rule I followed when designing these entities: avoid deep nesting (<a href="https://www.youtube.com/watch?v=9gnvC-xIDgs">related</a>, by <a href="http://twitter.com/chrisesplin">Chris Esplin</a>).</p> <div class="language-javascript highlighter-rouge"><pre class="highlight"><code><span class="nx">Auth</span><span class="err">:</span> <span class="p">{</span> <span class="cm">/* Model designed by Firebase, we can't touch this tanana */</span> <span class="p">}</span> <span class="nl">Users</span><span class="p">:</span> <span class="p">{</span> <span class="cm">/* $uid from Firebase Auth */</span> <span class="nl">$uid</span><span class="p">:</span> <span class="p">{</span> <span class="nl">role</span><span class="p">:</span> <span class="nb">String</span><span class="p">,</span> <span class="c1">// i.e. 'admin', more on that later</span> <span class="p">}</span> <span class="p">}</span> <span class="nl">Projects</span><span class="p">:</span> <span class="p">{</span> <span class="nl">$pid</span><span class="p">:</span> <span class="p">{</span> <span class="cm">/* Metadata */</span> <span class="p">}</span> <span class="p">}</span> <span class="nl">Files</span><span class="p">:</span> <span class="p">{</span> <span class="nl">$pid</span><span class="p">:</span> <span class="p">{</span> <span class="nl">$fid</span><span class="p">:</span> <span class="p">{</span> <span class="cm">/* File */</span><span class="p">}</span> <span class="p">}</span> <span class="p">}</span> <span class="nl">Tasks</span><span class="p">:</span> <span class="p">{</span> <span class="nl">$pid</span><span class="p">:</span> <span class="p">{</span> <span class="nl">$tid</span><span class="p">:</span> <span class="p">{</span> <span class="cm">/* Task */</span><span class="p">}</span> <span class="p">}</span> <span class="p">}</span> <span class="nl">Messages</span><span class="p">:</span> <span class="p">{</span> <span class="nl">$pid</span><span class="p">:</span> <span class="p">{</span> <span class="nl">$mid</span><span class="p">:</span> <span class="p">{</span> <span class="cm">/* Message */</span><span class="p">}</span> <span class="p">}</span> <span class="p">}</span> </code></pre> </div> <p><br /></p> <h4 id="user-roles--relationships">2. User roles &amp; relationships</h4> <p>Once more, I might be omitting some details here and there. Here are the main roles &amp; their permissions:</p> <ul> <li> <p>Admin</p> <ul> <li><strong>Everything</strong>: Create, Read, Update, Delete</li> </ul> </li> <li> <p>Manager</p> <ul> <li><strong>Projects</strong>, <strong>Tasks</strong>, <strong>Files</strong>: Create, Read, Update, Delete</li> <li><strong>Files</strong> Create, Read, Update, Delete</li> <li><strong>Messages</strong>: Create, Read, Update, Delete</li> <li><strong>Tasks</strong>: Create, Read, Update, Delete</li> <li><strong>Users</strong>: Create, Read</li> <li><strong>Self</strong> Read, Update, Delete (users/{$ownUid})</li> </ul> </li> <li> <p>User</p> <ul> <li><strong>Projects</strong>, <strong>Tasks</strong> Read</li> <li><strong>Files</strong> Create, Read, Update, Delete</li> <li><strong>Messages</strong> Read</li> <li><strong>Own Messages</strong> Create, Update, Delete</li> <li><strong>Tasks</strong> Read, Update</li> <li><strong>Self</strong> Read, Update, Delete (users/{$ownUid})</li> </ul> </li> <li> <p>External users / collaborators</p> <ul> <li><strong>Some Projects</strong> Read</li> <li><strong>Some Tasks</strong> Read</li> <li><strong>Files</strong> Create, Read</li> <li><strong>Own Files</strong> Update, Delete</li> <li><strong>Self</strong> Read, Update, Delete (users/{$ownUid})</li> </ul> </li> </ul> <h2 id="general-approach">General approach</h2> <p>As you can see, it’s quite messy. Expressing this with conditionals (i.e. <code class="highlighter-rouge">if(user.admin === true){...}</code>) would be a terrible idea. Here’s why:</p> <ul> <li>hard to document</li> <li>hard to maintain</li> <li>hard to change / scale</li> <li>code coupling</li> <li>a pain to express as Firebase’s JSON rules</li> </ul> <p>I’d like to avoid all this so I ditched the idea quickly. In fact, I can’t see how this approach would work with most tools / systems, not only with Firebase, but that’s another story. <br /> What I do want to go for instead is an activity-based authorization system.</p> <p>Here’s how that’ll work: By default no user has any permission. Then, I create a new entity in my real-time database, called <strong>Permissions</strong> where I can define roles (that give certain permissions). I’ll key by role name and write all permissions for a certain role. Example: <br /></p> <div class="language-javascript highlighter-rouge"><pre class="highlight"><code><span class="nx">Permissions</span><span class="err">:</span> <span class="p">{</span> <span class="nl">user</span><span class="p">:</span> <span class="p">{</span> <span class="c1">// User role</span> <span class="nl">users</span><span class="p">:</span> <span class="p">{</span> <span class="nl">own</span><span class="p">:</span> <span class="p">{</span> <span class="c1">// can read, update, delete own data</span> <span class="nl">r</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span> <span class="nx">u</span><span class="err">:</span> <span class="kc">true</span><span class="p">,</span> <span class="nx">d</span><span class="err">:</span> <span class="kc">true</span> <span class="p">},</span> <span class="nx">other</span><span class="err">:</span> <span class="p">{}</span> <span class="c1">// can't read/write 'other' (not owned) data; we can even omit this field</span> <span class="p">},</span> <span class="cm">/* * ... * skipped some entities because 'messages' are more interesting * ... */</span> <span class="nx">messages</span><span class="err">:</span> <span class="p">{</span> <span class="nl">c</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span> <span class="c1">// can create messages</span> <span class="p">},</span> <span class="nx">message</span><span class="err">:</span> <span class="p">{</span> <span class="nl">own</span><span class="p">:</span> <span class="p">{</span> <span class="c1">// can update &amp; delete own messages</span> <span class="nl">r</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span> <span class="nx">u</span><span class="err">:</span> <span class="kc">true</span><span class="p">,</span> <span class="nx">d</span><span class="err">:</span> <span class="kc">true</span> <span class="p">},</span> <span class="nx">other</span><span class="err">:</span> <span class="p">{</span> <span class="c1">// can read other messages</span> <span class="nl">r</span><span class="p">:</span> <span class="kc">true</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> </code></pre> </div> <p>A small note here, my permissions mirror CRUD, but they can be anything really. I also grouped them in ‘own’ &amp; ‘other’ for readability, but it might as well be <code class="highlighter-rouge">canReadOwn</code> or <code class="highlighter-rouge">canReadOther</code>. For my use case the above is good enough, but you might choose to have more fine-grained control in yours.</p> <p>The way this will work is that upon creation, users are assigned a role. It’s important to note that only admins can add users to this app (there’s no open registration… kind of). Because I don’t want to use a back-end, there’s an immediate problem with this: there’s actually no way to restrict Firebase registration through the console. With some nifty scripts clever people could create an account for themselves, but the joke’s on them. <br /> Firstly, I’ll make sure a <code class="highlighter-rouge">changeRole</code> permission exists for writing to <code class="highlighter-rouge">/users/$uid/role</code>, which only admins have (they’ll have permission to do everything, including this). <br /> Secondly, if there’s no <code class="highlighter-rouge">/users/$uid/role</code> set, I restrict all read &amp; write access. That should keep those pesky clever users from messing with my database.</p> <p>Now that’s fine and dandy, but how to actually implement this?<br /> Don’t worry friend, we got this.</p> <h2 id="bolt-ftw">Bolt FTW</h2> <p>If you didn’t come across it yet, <a href="https://github.com/firebase/bolt">Firebase Bolt</a> is a compiler for Firebase rules (it might be illegal, but could we call it a Firebase rules transpiler?!). It’s supposedly experimental, so make sure to double-check the output after compiling (unit test!).</p> <p>Now, even with Bolt, I’ve noticed I’d be in trouble already. I want to have CRUD-like rules, but Firebase only knows about Read / Write. How to restrict deletion then? <br /> I had to choose between (I believe) two options:</p> <ul> <li><strong>A.</strong> build objects such that they have <code class="highlighter-rouge"><span class="p">{</span><span class="w"> </span><span class="err">public:</span><span class="w"> </span><span class="err">{/*</span><span class="w"> </span><span class="err">what</span><span class="w"> </span><span class="err">I</span><span class="w"> </span><span class="err">have</span><span class="w"> </span><span class="err">so</span><span class="w"> </span><span class="err">far</span><span class="w"> </span><span class="err">*/</span><span class="w"> </span><span class="p">}</span><span class="err">,</span><span class="w"> </span><span class="err">private:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">deleted:</span><span class="w"> </span><span class="err">Boolean</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="err">}</span></code> paths &amp; add a field named ‘deleted’ on <code class="highlighter-rouge">private</code>, for which I add stricter write rules, i.e. only admins will have permissions to write (NB: this way data will never actually be deleted). Then, I’d tweak the rules to prevent access for items with <code class="highlighter-rouge">private.deleted === true</code></li> <li><strong>B.</strong> use <code class="highlighter-rouge">newData.exists()</code> together with my permissions, which will prevent deletion for users that are not authorized to delete</li> </ul> <p>I went with <strong>B.</strong>, but some might find <strong>A.</strong> to be suitable: that way data can be restored later if needed. After I’ll deploy the first version of the app, the only way to change my decision is via (probably) a complicated migration script.</p> <p>I’ll have a similar issue with ‘create’. There’s no create rule in Firebase, but there are ways to make that work too.<br /> Long story short, Bolt supports splitting the <code class="highlighter-rouge">write</code> rule into <code class="highlighter-rouge">create</code>, <code class="highlighter-rouge">update</code>, <code class="highlighter-rouge">delete</code> (<a href="https://github.com/firebase/bolt/blob/master/docs/language.md#write-aliases">see here</a>). It’s a matter of preference in the end, but I’ll stick with it for the sake of simplicity. You can achieve the same in a more explicit manner.</p> <h2 id="bolt-implementation">Bolt implementation</h2> <p>Finally, the fun part. I’m going to sketch out how to implement this in Bolt, but take it with a pinch of salt. Most likely I’ll write up a quick new post on what changes I did to it before actually deploying. For now, here goes:</p> <div class="language-javascript highlighter-rouge"><pre class="highlight"><code><span class="cm">/* * Checks if a given action is allowed. * * @param {object} obj The object to check. * @param {string} entity The entity to check. * @param {string} permission The permission to check for. * @param {boolean} ownResources A flag that disables 'own' data checks if set (if false, users can perform an action if they own the resource) * */</span> <span class="nx">isAllowed</span><span class="p">(</span><span class="nx">obj</span><span class="p">,</span> <span class="nx">entity</span><span class="p">,</span> <span class="nx">permission</span><span class="p">,</span> <span class="nx">ownResources</span><span class="p">)</span> <span class="p">{</span> <span class="nx">auth</span> <span class="o">!=</span> <span class="kc">null</span> <span class="o">&amp;&amp;</span> <span class="nx">auth</span><span class="p">.</span><span class="nx">uid</span> <span class="o">&amp;&amp;</span> <span class="c1">// has to be authenticated</span> <span class="nx">root</span><span class="p">.</span><span class="nx">users</span><span class="p">[</span><span class="nx">auth</span><span class="p">.</span><span class="nx">uid</span><span class="p">].</span><span class="nx">role</span> <span class="o">&amp;&amp;</span> <span class="c1">// has to have a role set</span> <span class="p">(</span> <span class="nx">auth</span><span class="p">.</span><span class="nx">uid</span> <span class="o">!==</span> <span class="nx">obj</span><span class="p">.</span><span class="nx">owner</span> <span class="c1">// user is not owner of the entity</span> <span class="p">?</span> <span class="nx">root</span><span class="p">.</span><span class="nx">permissions</span><span class="p">[</span><span class="nx">root</span><span class="p">.</span><span class="nx">users</span><span class="p">[</span><span class="nx">auth</span><span class="p">.</span><span class="nx">uid</span><span class="p">].</span><span class="nx">role</span><span class="p">][</span><span class="nx">entity</span><span class="p">].</span><span class="nx">all</span> <span class="o">===</span> <span class="kc">true</span> <span class="o">||</span> <span class="c1">// allow everything</span> <span class="cm">/* entity w/o owner: doesn't require extra nesting in the permissions obj (.other) */</span> <span class="nx">root</span><span class="p">.</span><span class="nx">permissions</span><span class="p">[</span><span class="nx">root</span><span class="p">.</span><span class="nx">users</span><span class="p">[</span><span class="nx">auth</span><span class="p">.</span><span class="nx">uid</span><span class="p">].</span><span class="nx">role</span><span class="p">][</span><span class="nx">entity</span><span class="p">][</span><span class="nx">permission</span><span class="p">]</span> <span class="o">===</span> <span class="kc">true</span> <span class="o">||</span> <span class="nx">root</span><span class="p">.</span><span class="nx">permissions</span><span class="p">[</span><span class="nx">root</span><span class="p">.</span><span class="nx">users</span><span class="p">[</span><span class="nx">auth</span><span class="p">.</span><span class="nx">uid</span><span class="p">].</span><span class="nx">role</span><span class="p">][</span><span class="nx">entity</span><span class="p">].</span><span class="nx">other</span><span class="p">[</span><span class="nx">permission</span><span class="p">]</span> <span class="o">===</span> <span class="kc">true</span> <span class="p">:</span> <span class="p">(</span><span class="o">!</span><span class="nx">ownResources</span> <span class="p">?</span> <span class="nx">root</span><span class="p">.</span><span class="nx">permissions</span><span class="p">[</span><span class="nx">root</span><span class="p">.</span><span class="nx">users</span><span class="p">[</span><span class="nx">auth</span><span class="p">.</span><span class="nx">uid</span><span class="p">].</span><span class="nx">role</span><span class="p">][</span><span class="nx">entity</span><span class="p">].</span><span class="nx">own</span><span class="p">[</span><span class="nx">permission</span><span class="p">]</span> <span class="o">===</span> <span class="kc">true</span> <span class="p">:</span> <span class="kc">false</span><span class="p">)</span> <span class="p">)</span> <span class="p">}</span> <span class="cm">/* Nobody can access anything by default */</span> <span class="nx">path</span> <span class="o">/</span> <span class="p">{</span> <span class="nx">read</span><span class="p">()</span> <span class="p">{</span> <span class="kc">false</span> <span class="p">}</span> <span class="nx">write</span><span class="p">()</span> <span class="p">{</span> <span class="kc">false</span> <span class="p">}</span> <span class="p">}</span> <span class="cm">/* Example of permission check: Projects &amp; Project */</span> <span class="nx">type</span> <span class="nx">Projects</span> <span class="p">{</span> <span class="nx">create</span><span class="p">()</span> <span class="p">{</span> <span class="nx">isAllowed</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="s1">'projects'</span><span class="p">,</span> <span class="s1">'c'</span><span class="p">,</span> <span class="kc">false</span><span class="p">)</span> <span class="p">}</span> <span class="p">}</span> <span class="nx">type</span> <span class="nx">Project</span> <span class="p">{</span> <span class="nx">read</span><span class="p">()</span> <span class="p">{</span> <span class="nx">isAllowed</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="s1">'project'</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">,</span> <span class="kc">false</span><span class="p">)</span> <span class="p">}</span> <span class="nx">update</span><span class="p">()</span> <span class="p">{</span> <span class="nx">isAllowed</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="s1">'project'</span><span class="p">,</span> <span class="s1">'u'</span><span class="p">,</span> <span class="kc">false</span><span class="p">)</span> <span class="p">}</span> <span class="k">delete</span><span class="p">()</span> <span class="p">{</span> <span class="nx">isAllowed</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="s1">'project'</span><span class="p">,</span> <span class="s1">'d'</span><span class="p">,</span> <span class="kc">false</span><span class="p">)</span> <span class="p">}</span> <span class="p">}</span> <span class="nx">path</span> <span class="o">/</span><span class="nx">projects</span> <span class="nx">is</span> <span class="nx">Projects</span> <span class="p">{}</span> <span class="nx">path</span> <span class="o">/</span><span class="nx">projects</span><span class="o">/</span><span class="p">{</span><span class="nx">$pid</span><span class="p">}</span> <span class="nx">is</span> <span class="nx">Project</span> <span class="p">{}</span> <span class="nx">path</span> <span class="o">/</span><span class="nx">users</span><span class="o">/</span><span class="p">{</span><span class="nx">$uid</span><span class="p">}</span><span class="sr">/role { /</span><span class="o">*</span> <span class="nx">Only</span> <span class="nx">admins</span> <span class="nx">should</span> <span class="nx">be</span> <span class="nx">able</span> <span class="nx">to</span> <span class="nx">write</span> <span class="nx">the</span> <span class="nx">role</span><span class="p">.</span> <span class="o">*</span><span class="sr">/</span><span class="err"> </span> <span class="nx">read</span><span class="p">()</span> <span class="p">{</span> <span class="nx">auth</span><span class="p">.</span><span class="nx">uid</span> <span class="o">==</span> <span class="nx">$uid</span> <span class="p">}</span> <span class="cm">/* Users can read their own role */</span> <span class="nx">write</span><span class="p">()</span> <span class="p">{</span> <span class="nx">isAllowed</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="s1">'user'</span><span class="p">,</span> <span class="s1">'changeRole'</span><span class="p">,</span> <span class="kc">true</span><span class="p">)</span> <span class="p">}</span> <span class="p">}</span> <span class="nx">path</span> <span class="o">/</span><span class="nx">projects</span><span class="o">/</span><span class="p">{</span><span class="nx">$uid</span><span class="p">}</span><span class="sr">/owner { /</span><span class="o">*</span> <span class="nx">Users</span> <span class="nx">can</span> <span class="nx">only</span> <span class="nx">write</span> <span class="nx">to</span> <span class="nx">their</span> <span class="nx">own</span> <span class="s1">'owner'</span> <span class="nx">fields</span><span class="p">.</span> <span class="o">*</span><span class="sr">/</span><span class="err"> </span> <span class="nx">write</span><span class="p">()</span> <span class="p">{</span> <span class="nx">auth</span><span class="p">.</span><span class="nx">uid</span> <span class="o">==</span> <span class="nx">$uid</span> <span class="p">}</span> <span class="p">}</span> <span class="cm">/* * ... * Other entities omitted; they would look similar. * Each entity with an owner will have a `/owner` rule. * That prohibits anybody from changing it, like we have for `/projects/{$uid}/owner` */</span><span class="sr">/</span><span class="err"> </span></code></pre> </div> <p>That’s about it. It compiles via Bolt, so it should work… maybe with some tweaks here &amp; there. With proper unit tests, this baby’s gonna be able to handle whatever you throw at it.</p> <p>To give one more example, here’s how the admin role would look:</p> <div class="language-javascript highlighter-rouge"><pre class="highlight"><code><span class="nx">Permissions</span><span class="err">:</span> <span class="p">{</span> <span class="nl">admin</span><span class="p">:</span> <span class="p">{</span> <span class="c1">// Admin role: allow everything</span> <span class="nl">users</span><span class="p">:</span> <span class="p">{</span> <span class="nl">all</span><span class="p">:</span> <span class="kc">true</span> <span class="p">},</span> <span class="nx">projects</span><span class="err">:</span> <span class="p">{</span> <span class="nl">all</span><span class="p">:</span> <span class="kc">true</span> <span class="p">},</span> <span class="nx">project</span><span class="err">:</span> <span class="p">{</span> <span class="nl">all</span><span class="p">:</span> <span class="kc">true</span> <span class="p">},</span> <span class="nx">tasks</span><span class="err">:</span> <span class="p">{</span> <span class="nl">all</span><span class="p">:</span> <span class="kc">true</span> <span class="p">},</span> <span class="nx">task</span><span class="err">:</span> <span class="p">{</span> <span class="nl">all</span><span class="p">:</span> <span class="kc">true</span> <span class="p">},</span> <span class="nx">files</span><span class="err">:</span> <span class="p">{</span> <span class="nl">all</span><span class="p">:</span> <span class="kc">true</span> <span class="p">},</span> <span class="nx">file</span><span class="err">:</span> <span class="p">{</span> <span class="nl">all</span><span class="p">:</span> <span class="kc">true</span> <span class="p">},</span> <span class="nx">messages</span><span class="err">:</span> <span class="p">{</span> <span class="nl">all</span><span class="p">:</span> <span class="kc">true</span> <span class="p">},</span> <span class="nx">message</span><span class="err">:</span> <span class="p">{</span> <span class="nl">all</span><span class="p">:</span> <span class="kc">true</span> <span class="p">}</span> <span class="p">}</span> <span class="cm">/* ... */</span> <span class="p">}</span> </code></pre> </div> <h2 id="front-end-implementation">Front-end implementation</h2> <p>Firebase doesn’t require a back-end, but you can use one if you want to (or absolutely need it). For this project, I chose not to use a back-end. That means that I’ll handle the UI rendering 100% in the front-end. In turn, that will require that I know what a certain user is allowed to do upon authentication. This way I can show / hide relevant parts of the UI (i.e. a delete button).</p> <p>No matter if you are spinning up a React, Angular, etc. app or even cowboying your way through with a Vanilla JS app, the implementation shouldn’t differ much. <br /> One way to do it is load permissions when a resource is accessed (in the context of the current auth. user). Similarly to the Bolt implementation, a generic <code class="highlighter-rouge">isAllowed()</code> method should check if certain actions can be performed. Another way is to make use of the real-time database and keep an eye out for permission changes. Of course, that means all permissions will be loaded upon login.</p> <h2 id="caveats">Caveats</h2> <p><small>(yes, I used this word because I don’t like pitfalls. do you?!)</small></p> <ol> <li>Don’t take unit testing lightly, you need to do it properly (Bolt == experimental)</li> <li>It’s a bit hard to see the full picture; there might still be some security issues, but hey, it’s a complex app with many user roles (see 1.).</li> <li>Not being able to restrict registration makes me feel a bit uneasy, so perhaps a back-end would be necessary somewhere down the road.</li> </ol> <p>I want to believe that this strategy will work all the way &amp; I won’t need any server logic. Is that maybe too ambitious?<br /> Would love to get some feedback from experienced Firebasing guru-ninjas.</p> Python logging handler for Mailgun 2015-10-12T13:25:30+02:00 https://fadeit.dk/blog /2015/10/12/mailgun-python-log-handler <p>Yet another reason to love python is the wonderful logging API provided by a standard library. Having a decent logger provided by the language saves developers time from integrating with some third-party library, but most importantly it contributes to a clean system where all modules use the same logger. Out of the box, there is a <a href="https://docs.python.org/3/library/logging.handlers.html">handler</a> for almost any typical use-case, but the best feature is the option to create your own with relative ease. So-far we’ve been using <a href="https://docs.python.org/3/library/logging.handlers.html#smtphandler">SMTPHandler</a>, but since moving to <a href="https://www.mailgun.com/">Mailgun</a>, I decided to experiment with a custom handler to submit logs via their API instead. I’d like to mention that creating a custom logging handler is not a requirement, it is still possible to use their SMTP server with SMTP handler. To get started, we need to extend Handler baseclass and override emit method to make a POST request.</p> <div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">logging</span> <span class="kn">import</span> <span class="nn">requests</span> <span class="k">class</span> <span class="nc">MailgunHandler</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">Handler</span><span class="p">):</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">api_url</span><span class="p">,</span> <span class="n">api_key</span><span class="p">,</span> <span class="n">sender</span><span class="p">,</span> <span class="n">recipients</span><span class="p">,</span> <span class="n">subject</span><span class="p">):</span> <span class="c"># run the regular Handler __init__</span> <span class="n">logging</span><span class="o">.</span><span class="n">Handler</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">api_url</span> <span class="o">=</span> <span class="n">api_url</span> <span class="bp">self</span><span class="o">.</span><span class="n">api_key</span> <span class="o">=</span> <span class="n">api_key</span> <span class="bp">self</span><span class="o">.</span><span class="n">sender</span> <span class="o">=</span> <span class="n">sender</span> <span class="bp">self</span><span class="o">.</span><span class="n">recipients</span> <span class="o">=</span> <span class="n">recipients</span> <span class="bp">self</span><span class="o">.</span><span class="n">subject</span> <span class="o">=</span> <span class="n">subject</span> <span class="k">def</span> <span class="nf">emit</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">record</span><span class="p">):</span> <span class="c"># record.message is the log message</span> <span class="k">for</span> <span class="n">recipient</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">recipients</span><span class="p">:</span> <span class="n">data</span> <span class="o">=</span> <span class="p">{</span> <span class="s">"from"</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">sender</span><span class="p">,</span> <span class="s">"to"</span><span class="p">:</span> <span class="n">recipient</span><span class="p">,</span> <span class="s">"subject"</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">subject</span><span class="p">,</span> <span class="s">"text"</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">record</span><span class="p">)</span> <span class="p">}</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">api_url</span><span class="p">,</span> <span class="n">auth</span><span class="o">=</span><span class="p">(</span><span class="s">"api"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">api_key</span><span class="p">),</span> <span class="n">data</span><span class="o">=</span><span class="n">data</span><span class="p">)</span> </code></pre> </div> <p>I have created a simple flask application that registers the handler and also implemented a route that triggers an error to test it out.</p> <div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">logging</span> <span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span> <span class="kn">from</span> <span class="nn">mailgun_handler</span> <span class="kn">import</span> <span class="n">MailgunHandler</span> <span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span> <span class="c"># Setup logging handler</span> <span class="n">mail_handler</span> <span class="o">=</span> <span class="n">MailgunHandler</span><span class="p">(</span> <span class="n">api_url</span><span class="o">=</span><span class="s">'https://api.mailgun.net/v3/fadeit.dk/messages'</span><span class="p">,</span> <span class="n">api_key</span><span class="o">=</span><span class="s">'secret'</span><span class="p">,</span> <span class="n">sender</span><span class="o">=</span><span class="s">'noreply@fadeit.dk'</span><span class="p">,</span> <span class="n">recipients</span><span class="o">=</span><span class="p">[</span><span class="s">'ss@fadeit.dk'</span><span class="p">,</span> <span class="s">'ja@fadeit.dk'</span><span class="p">],</span> <span class="n">subject</span><span class="o">=</span><span class="s">'Application error!'</span> <span class="p">)</span> <span class="n">mail_handler</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">ERROR</span><span class="p">)</span> <span class="n">mail_handler</span><span class="o">.</span><span class="n">setFormatter</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">Formatter</span><span class="p">(</span><span class="s">''' Message type: </span><span class="si">%(levelname)</span><span class="s">s Location: </span><span class="si">%(pathname)</span><span class="s">s:</span><span class="si">%(lineno)</span><span class="s">d Module: </span><span class="si">%(module)</span><span class="s">s Function: </span><span class="si">%(funcName)</span><span class="s">s Time: </span><span class="si">%(asctime)</span><span class="s">s Message: </span><span class="si">%(message)</span><span class="s">s '''</span><span class="p">))</span> <span class="n">app</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">addHandler</span><span class="p">(</span><span class="n">mail_handler</span><span class="p">)</span> <span class="nd">@app.route</span><span class="p">(</span><span class="s">'/error'</span><span class="p">)</span> <span class="k">def</span> <span class="nf">error</span><span class="p">():</span> <span class="k">raise</span> <span class="nb">Exception</span><span class="p">(</span><span class="s">"Beds are burning!"</span><span class="p">)</span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span> <span class="c">#app.run(debug=True) #Handler isn't executed if app is run in debug mode</span> <span class="n">app</span><span class="o">.</span><span class="n">run</span><span class="p">()</span> </code></pre> </div> Testing Express.js + CouchDB Applications with mock-couch 2015-07-05T22:30:00+02:00 https://fadeit.dk/blog /2015/07/05/testing-express-couchdb-apps-with-mock-couch <h2 id="prerequisites">Prerequisites</h2> <p>The code in this post is based upon <a href="http://coffeescript.org/">CoffeeScript</a>, <a href="http://mochajs.org/">Mocha</a> and <a href="http://gulpjs.com/">Gulp</a> but it should be easily adaptable to other stacks (e.g. <a href="https://www.javascript.com/">JavaScript</a>, <a href="https://jasmine.github.io/">Jasmine</a> and <a href="http://gruntjs.com/">Grunt</a>).</p> <h2 id="intro">Intro</h2> <p>Recently, I worked on a <a href="http://expressjs.com/">Express.js</a> based RESTful API with <a href="https://couchdb.apache.org/">CouchDB</a> storage. For various reasons, I mostly had end-to-end tests (testing HTTP endpoints) and very few unit tests. In the tests, I would rebuild the database before every <code class="highlighter-rouge">describe</code> block. This worked quite well but at one point I noticed that my feedback loops were beginning to get tediously slow (~30 seconds to run the full test suite). Thankfully, I discovered <a href="https://github.com/chris-l/mock-couch">mock-couch</a>: in-memory storage with CouchDB API. The switch was rather easy and, as a result, my test suite would finish in less than 10 seconds (even when reseting test data before each test).</p> <h2 id="setup">Setup</h2> <p>Our sample API will expose functionality to access a database of sofa’s. Let’s set it up.</p> <p>Structure and dependencies</p> <div class="language-bash highlighter-rouge"><pre class="highlight"><code>mkdir -p sofa-api/specs <span class="nb">cd </span>sofa-api npm install mock-couch coffee-script gulp gulp-mocha should supertest express nano q </code></pre> </div> <p>gulpfile.coffee</p> <div class="language-coffeescript highlighter-rouge"><pre class="highlight"><code><span class="nx">require</span> <span class="s">'coffee-script/register'</span> <span class="nx">gulp</span> <span class="o">=</span> <span class="nx">require</span> <span class="s">'gulp'</span> <span class="nx">mocha</span> <span class="o">=</span> <span class="nx">require</span> <span class="s">'gulp-mocha'</span> <span class="nx">sources</span> <span class="o">=</span> <span class="na">tests</span><span class="o">:</span> <span class="s">'./specs/**/*.spec.coffee'</span> <span class="c1"># Task to run tests with mock-couch backend</span> <span class="nx">gulp</span><span class="p">.</span><span class="na">task</span> <span class="s">'test'</span><span class="p">,</span> <span class="o">-&gt;</span> <span class="nx">process</span><span class="p">.</span><span class="na">env</span><span class="p">.</span><span class="na">NODE_ENV</span> <span class="o">=</span> <span class="s">'test'</span> <span class="nx">process</span><span class="p">.</span><span class="na">env</span><span class="p">.</span><span class="na">PORT</span> <span class="o">=</span> <span class="mi">3010</span> <span class="nx">process</span><span class="p">.</span><span class="na">env</span><span class="p">.</span><span class="na">COUCH_PORT</span> <span class="o">=</span> <span class="mi">5987</span> <span class="nx">process</span><span class="p">.</span><span class="na">env</span><span class="p">.</span><span class="na">DATABASE</span> <span class="o">=</span> <span class="s">'sofas_test'</span> <span class="nx">gulp</span><span class="p">.</span><span class="na">src</span><span class="p">(</span><span class="nx">sources</span><span class="p">.</span><span class="na">tests</span><span class="p">)</span> <span class="p">.</span><span class="na">pipe</span><span class="p">(</span><span class="nx">mocha</span><span class="p">())</span> <span class="c1"># Task to run integration tests (real CouchDB backend)</span> <span class="nx">gulp</span><span class="p">.</span><span class="na">task</span> <span class="s">'test-int'</span><span class="p">,</span> <span class="o">-&gt;</span> <span class="nx">process</span><span class="p">.</span><span class="na">env</span><span class="p">.</span><span class="na">NODE_ENV</span> <span class="o">=</span> <span class="s">'test-int'</span> <span class="nx">process</span><span class="p">.</span><span class="na">env</span><span class="p">.</span><span class="na">PORT</span> <span class="o">=</span> <span class="mi">3010</span> <span class="nx">process</span><span class="p">.</span><span class="na">env</span><span class="p">.</span><span class="na">COUCH_PORT</span> <span class="o">=</span> <span class="mi">5984</span> <span class="nx">process</span><span class="p">.</span><span class="na">env</span><span class="p">.</span><span class="na">DATABASE</span> <span class="o">=</span> <span class="s">'sofas_test'</span> <span class="nx">gulp</span><span class="p">.</span><span class="na">src</span><span class="p">(</span><span class="nx">sources</span><span class="p">.</span><span class="na">tests</span><span class="p">)</span> <span class="p">.</span><span class="na">pipe</span><span class="p">(</span><span class="nx">mocha</span><span class="p">())</span> <span class="c1"># Task to run our server</span> <span class="nx">gulp</span><span class="p">.</span><span class="na">task</span> <span class="s">'default'</span><span class="p">,</span> <span class="o">-&gt;</span> <span class="nx">process</span><span class="p">.</span><span class="na">env</span><span class="p">.</span><span class="na">NODE_ENV</span> <span class="o">=</span> <span class="s">'development'</span> <span class="nx">process</span><span class="p">.</span><span class="na">env</span><span class="p">.</span><span class="na">PORT</span> <span class="o">=</span> <span class="mi">3000</span> <span class="nx">process</span><span class="p">.</span><span class="na">env</span><span class="p">.</span><span class="na">COUCH_PORT</span> <span class="o">=</span> <span class="mi">5984</span> <span class="nx">process</span><span class="p">.</span><span class="na">env</span><span class="p">.</span><span class="na">DATABASE</span> <span class="o">=</span> <span class="s">'sofas'</span> <span class="nx">server</span> <span class="o">=</span> <span class="nx">require</span> <span class="s">'./server'</span> <span class="nx">server</span><span class="p">.</span><span class="na">listen</span><span class="p">()</span> </code></pre> </div> <p>server.coffee</p> <div class="language-coffeescript highlighter-rouge"><pre class="highlight"><code><span class="nx">express</span> <span class="o">=</span> <span class="nx">require</span> <span class="s">'express'</span> <span class="nx">app</span> <span class="o">=</span> <span class="nx">express</span><span class="p">()</span> <span class="nx">app</span><span class="p">.</span><span class="na">get</span> <span class="s">'/sofas'</span><span class="p">,</span> <span class="p">(</span><span class="nx">req</span><span class="p">,</span> <span class="nx">res</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span> <span class="s">'not implemented'</span> <span class="nx">app</span><span class="p">.</span><span class="na">get</span> <span class="s">'/sofas/:sofa'</span><span class="p">,</span> <span class="p">(</span><span class="nx">req</span><span class="p">,</span> <span class="nx">res</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span> <span class="s">'not implemented'</span> <span class="nx">exports</span><span class="p">.</span><span class="na">listen</span> <span class="o">=</span> <span class="p">(</span><span class="nx">callback</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nx">port</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="na">env</span><span class="p">.</span><span class="na">PORT</span> <span class="nx">app</span><span class="p">.</span><span class="na">listen</span> <span class="nx">port</span><span class="p">,</span> <span class="nx">callback</span> </code></pre> </div> <p>Let’s create file <em>specs/sofas.spec.coffee</em> which is going to contain our tests.</p> <div class="language-coffeescript highlighter-rouge"><pre class="highlight"><code><span class="nx">should</span> <span class="o">=</span> <span class="nx">require</span> <span class="s">'should'</span> <span class="nx">describe</span> <span class="s">'Sofas API'</span><span class="p">,</span> <span class="o">-&gt;</span> <span class="nx">describe</span> <span class="s">'/sofas'</span><span class="p">,</span> <span class="o">-&gt;</span> <span class="nx">it</span> <span class="s">'should return all sofas with GET'</span><span class="p">,</span> <span class="o">-&gt;</span> <span class="c1"># TODO: write test</span> <span class="p">(</span><span class="no">false</span><span class="p">).</span><span class="na">should</span><span class="p">.</span><span class="na">be</span><span class="p">.</span><span class="na">true</span><span class="p">()</span> <span class="nx">describe</span> <span class="s">'/sofas/:sofa'</span><span class="p">,</span> <span class="o">-&gt;</span> <span class="nx">it</span> <span class="s">'should return a particular sofa with GET'</span><span class="p">,</span> <span class="o">-&gt;</span> <span class="c1"># TODO: write test</span> <span class="p">(</span><span class="no">false</span><span class="p">).</span><span class="na">should</span><span class="p">.</span><span class="na">be</span><span class="p">.</span><span class="na">true</span><span class="p">()</span> </code></pre> </div> <p>If you run the tests right now, you should see similar output:</p> <div class="language-bash highlighter-rouge"><pre class="highlight"><code>gulp <span class="nb">test </span>Sofas API /sofas 1<span class="o">)</span> should <span class="k">return </span>all sofas with GET /sofas/:sofa 2<span class="o">)</span> should <span class="k">return </span>a particular sofa with GET 0 passing <span class="o">(</span>30ms<span class="o">)</span> 2 failing 1<span class="o">)</span> Sofas API /sofas should <span class="k">return </span>all sofas with GET: AssertionError: expected <span class="nb">false </span>to be <span class="nb">true </span>at Context.&lt;anonymous&gt; <span class="o">(</span>specs/sofas.spec.coffee:8:7<span class="o">)</span> 2<span class="o">)</span> Sofas API /sofas/:sofa should <span class="k">return </span>a particular sofa with GET: AssertionError: expected <span class="nb">false </span>to be <span class="nb">true </span>at Context.&lt;anonymous&gt; <span class="o">(</span>specs/sofas.spec.coffee:13:7<span class="o">)</span> Message: 2 tests failed. </code></pre> </div> <h2 id="implementation">Implementation</h2> <p>Sofas will be stored in CouchDB with the following structure:</p> <div class="language-json highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nt">"_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"CouchDB assigned id"</span><span class="p">,</span><span class="w"> </span><span class="nt">"_rev"</span><span class="p">:</span><span class="w"> </span><span class="s2">"CouchDB assigned revision"</span><span class="p">,</span><span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Name of the sofa"</span><span class="p">,</span><span class="w"> </span><span class="nt">"price"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Price of the sofa"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre> </div> <p>With this in mind, let’s build a module for accessing the database.</p> <div class="language-coffeescript highlighter-rouge"><pre class="highlight"><code><span class="nx">nano</span> <span class="o">=</span> <span class="nx">require</span> <span class="s">'nano'</span> <span class="nx">Q</span> <span class="o">=</span> <span class="nx">require</span> <span class="s">'q'</span> <span class="k">class</span> <span class="nx">DAL</span> <span class="na">constructor</span><span class="o">:</span> <span class="p">(</span><span class="nx">port</span><span class="p">,</span> <span class="vi">@</span><span class="na">database</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="vi">@</span><span class="na">conn</span> <span class="o">=</span> <span class="nx">nano</span><span class="p">(</span><span class="s">"http://localhost:</span><span class="si">#{</span><span class="nx">port</span><span class="si">}</span><span class="s">"</span><span class="p">)</span> <span class="c1"># Returns a promise reseting database</span> <span class="na">resetDatabase</span><span class="o">:</span> <span class="p">(</span><span class="nx">withSamples</span><span class="o">=</span><span class="no">false</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nx">docs</span> <span class="o">=</span> <span class="p">[</span><span class="vi">@</span><span class="na">_sofaViews</span><span class="p">]</span> <span class="k">if</span> <span class="nx">withSamples</span> <span class="nx">docs</span> <span class="o">=</span> <span class="nx">docs</span><span class="p">.</span><span class="na">concat</span> <span class="vi">@</span><span class="na">_sofaSamples</span> <span class="vi">@</span><span class="na">_deleteDb</span><span class="p">()</span> <span class="p">.</span><span class="na">then</span> <span class="o">=&gt;</span> <span class="vi">@</span><span class="na">_createDb</span><span class="p">()</span> <span class="p">.</span><span class="na">then</span> <span class="o">=&gt;</span> <span class="nx">Q</span><span class="p">.</span><span class="na">all</span><span class="p">(</span><span class="nx">docs</span><span class="p">.</span><span class="na">map</span> <span class="p">(</span><span class="nx">d</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="vi">@</span><span class="na">_insertDoc</span><span class="p">(</span><span class="nx">d</span><span class="p">))</span> <span class="c1"># Returns a promise of sofas</span> <span class="c1"># If you provide the id- you'll get an array with a particular sofa in it</span> <span class="na">getSofas</span><span class="o">:</span> <span class="p">(</span><span class="nx">id</span><span class="o">=</span><span class="no">null</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nx">params</span> <span class="o">=</span> <span class="p">{}</span> <span class="nx">params</span><span class="p">.</span><span class="na">key</span> <span class="o">=</span> <span class="nx">id</span> <span class="k">if</span> <span class="nx">id</span><span class="o">?</span> <span class="vi">@</span><span class="na">_view</span><span class="p">(</span><span class="s">'sofas'</span><span class="p">,</span> <span class="s">'all'</span><span class="p">,</span> <span class="nx">params</span><span class="p">)</span> <span class="p">.</span><span class="na">then</span> <span class="p">(</span><span class="nx">res</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nx">body</span> <span class="o">=</span> <span class="nx">res</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="nx">sofas</span> <span class="o">=</span> <span class="p">(</span><span class="nx">row</span><span class="p">.</span><span class="na">value</span> <span class="k">for</span> <span class="nx">row</span> <span class="k">in</span> <span class="nx">body</span><span class="p">.</span><span class="na">rows</span><span class="p">)</span> <span class="k">return</span> <span class="nx">sofas</span> <span class="na">_sofaViews</span><span class="o">:</span> <span class="na">_id</span><span class="o">:</span> <span class="s">'_design/sofas'</span> <span class="na">views</span><span class="o">:</span> <span class="na">all</span><span class="o">:</span> <span class="na">map</span><span class="o">:</span> <span class="p">(</span><span class="nx">doc</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="k">if</span> <span class="nx">doc</span><span class="p">.</span><span class="na">type</span> <span class="o">is</span> <span class="s">'Sofa'</span> <span class="nx">emit</span> <span class="nx">doc</span><span class="p">.</span><span class="na">_id</span><span class="p">,</span> <span class="nx">doc</span> <span class="na">_sofaSamples</span><span class="o">:</span> <span class="p">[</span> <span class="p">{</span> <span class="na">_id</span><span class="o">:</span> <span class="s">'sofa1'</span> <span class="na">type</span><span class="o">:</span> <span class="s">'Sofa'</span> <span class="na">name</span><span class="o">:</span> <span class="s">'Sofa1'</span> <span class="na">price</span><span class="o">:</span> <span class="mi">400</span> <span class="p">}</span> <span class="p">{</span> <span class="na">_id</span><span class="o">:</span> <span class="s">'sofa2'</span> <span class="na">type</span><span class="o">:</span> <span class="s">'Sofa'</span> <span class="na">name</span><span class="o">:</span> <span class="s">'Sofa2'</span> <span class="na">price</span><span class="o">:</span> <span class="mi">300</span> <span class="p">}</span> <span class="p">{</span> <span class="na">_id</span><span class="o">:</span> <span class="s">'sofa3'</span> <span class="na">type</span><span class="o">:</span> <span class="s">'Sofa'</span> <span class="na">name</span><span class="o">:</span> <span class="s">'Sofa3'</span> <span class="na">price</span><span class="o">:</span> <span class="mi">500</span> <span class="p">}</span> <span class="p">]</span> <span class="c1"># Returns promise of creating database</span> <span class="na">_createDb</span><span class="o">:</span> <span class="o">-&gt;</span> <span class="nx">d</span> <span class="o">=</span> <span class="nx">Q</span><span class="p">.</span><span class="na">defer</span><span class="p">()</span> <span class="vi">@</span><span class="na">conn</span><span class="p">.</span><span class="na">db</span><span class="p">.</span><span class="na">create</span> <span class="vi">@</span><span class="na">database</span><span class="p">,</span> <span class="nx">d</span><span class="p">.</span><span class="na">makeNodeResolver</span><span class="p">()</span> <span class="nx">d</span><span class="p">.</span><span class="na">promise</span> <span class="c1"># Returns promise of deleting database</span> <span class="na">_deleteDb</span><span class="o">:</span> <span class="p">(</span><span class="nx">ignoreMissing</span><span class="o">=</span><span class="no">true</span><span class="p">)</span><span class="o">-&gt;</span> <span class="nx">d</span> <span class="o">=</span> <span class="nx">Q</span><span class="p">.</span><span class="na">defer</span><span class="p">()</span> <span class="vi">@</span><span class="na">conn</span><span class="p">.</span><span class="na">db</span><span class="p">.</span><span class="na">destroy</span> <span class="vi">@</span><span class="na">database</span><span class="p">,</span> <span class="p">(</span><span class="nx">err</span><span class="p">,</span> <span class="nx">res</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nx">err</span> <span class="o">=</span> <span class="no">null</span> <span class="k">if</span> <span class="nx">err</span><span class="o">?</span> <span class="o">and</span> <span class="nx">err</span><span class="p">.</span><span class="na">statusCode</span> <span class="o">is</span> <span class="mi">404</span> <span class="o">and</span> <span class="nx">ignoreMissing</span> <span class="nx">d</span><span class="p">.</span><span class="na">makeNodeResolver</span><span class="p">()(</span><span class="nx">err</span><span class="p">,</span> <span class="nx">res</span><span class="p">)</span> <span class="nx">d</span><span class="p">.</span><span class="na">promise</span> <span class="c1"># Returns promise of inserting a doc</span> <span class="na">_insertDoc</span><span class="o">:</span> <span class="p">(</span><span class="nx">doc</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nx">d</span> <span class="o">=</span> <span class="nx">Q</span><span class="p">.</span><span class="na">defer</span><span class="p">()</span> <span class="vi">@</span><span class="na">conn</span><span class="p">.</span><span class="na">use</span><span class="p">(</span><span class="vi">@</span><span class="na">database</span><span class="p">).</span><span class="na">insert</span> <span class="nx">doc</span><span class="p">,</span> <span class="nx">d</span><span class="p">.</span><span class="na">makeNodeResolver</span><span class="p">()</span> <span class="nx">d</span><span class="p">.</span><span class="na">promise</span> <span class="c1"># Returns promise of CouchDB view</span> <span class="na">_view</span><span class="o">:</span> <span class="p">(</span><span class="nx">designDoc</span><span class="p">,</span> <span class="nx">view</span><span class="p">,</span> <span class="nx">params</span><span class="o">=</span><span class="p">{})</span> <span class="o">-&gt;</span> <span class="nx">d</span> <span class="o">=</span> <span class="nx">Q</span><span class="p">.</span><span class="na">defer</span><span class="p">()</span> <span class="vi">@</span><span class="na">conn</span><span class="p">.</span><span class="na">use</span><span class="p">(</span><span class="vi">@</span><span class="na">database</span><span class="p">).</span><span class="na">view</span> <span class="nx">designDoc</span><span class="p">,</span> <span class="nx">view</span><span class="p">,</span> <span class="nx">params</span><span class="p">,</span> <span class="nx">d</span><span class="p">.</span><span class="na">makeNodeResolver</span><span class="p">()</span> <span class="nx">d</span><span class="p">.</span><span class="na">promise</span> <span class="nx">module</span><span class="p">.</span><span class="na">exports</span> <span class="o">=</span> <span class="nx">DAL</span> </code></pre> </div> <p>What’s missing is the endpoint and test implementation. Let’s start with the tests.</p> <div class="language-diff highlighter-rouge"><pre class="highlight"><code><span class="gh">diff --git a/specs/sofas.spec.coffee b/specs/sofas.spec.coffee index 8c1eb84..efbff58 100644 </span><span class="gd">--- a/specs/sofas.spec.coffee </span><span class="gi">+++ b/specs/sofas.spec.coffee </span><span class="gu">@@ -1,13 +1,54 @@ </span><span class="gd">-should = require 'should' </span><span class="gi">+should = require 'should' +DAL = require '../dal' +request = require 'supertest' +request = request "http://localhost:#{process.env.PORT}" +server = require '../server' + +app = null </span> describe 'Sofas API', -&gt; <span class="gi">+ beforeEach (done) -&gt; + # Reset database and start express server before each test + dal = new DAL(process.env.COUCH_PORT, process.env.DATABASE) + dal.resetDatabase(true) + .then -&gt; + app = server.listen done + .fail (err) -&gt; done err + + # Close express server after each test + afterEach (done) -&gt; + app.close done + + </span> describe '/sofas', -&gt; <span class="gd">- it 'should return all sofas with GET', -&gt; - # TODO: write test - (false).should.be.true() </span><span class="gi">+ it 'should return all sofas with GET', (done) -&gt; + request + .get '/sofas' + .expect 200 + .expect (res) -&gt; + sofas = res.body + sofas.should.have.length(3) + for s in sofas + s.should.have.property('_id') + s.should.have.property('_rev') + s.should.have.property('name') + s.should.have.property('price') + + .end done + + </span> describe '/sofas/:sofa', -&gt; <span class="gd">- it 'should return a particular sofa with GET', -&gt; - # TODO: write test - (false).should.be.true() </span><span class="gi">+ it 'should return a particular sofa with GET', (done) -&gt; + request + .get '/sofas/sofa1' + .expect 200 + .expect (res) -&gt; + sofa = res.body + sofa.should.have.property('_id') + sofa.should.have.property('_rev') + sofa.should.have.property('name') + sofa.should.have.property('price') + + .end done </span></code></pre> </div> <p>Now the endpoints:</p> <div class="language-diff highlighter-rouge"><pre class="highlight"><code><span class="gh">diff --git a/server.coffee b/server.coffee index 98f1918..a0d30ce 100644 </span><span class="gd">--- a/server.coffee </span><span class="gi">+++ b/server.coffee </span><span class="gu">@@ -1,16 +1,27 @@ </span> express = require 'express' <span class="gi">+DAL = require './dal' </span> <span class="gi">+dal = new DAL(process.env.COUCH_PORT, process.env.DATABASE) </span> app = express() app.get '/sofas', (req, res) -&gt; <span class="gd">- throw new Error 'not implemented' </span><span class="gi">+ dal.getSofas() + .then (sofas) -&gt; + res.json(sofas) + .fail (err) -&gt; res.status(err.statusCode).send() </span> app.get '/sofas/:sofa', (req, res) -&gt; <span class="gd">- throw new Error 'not implemented' </span><span class="gi">+ dal.getSofas(req.params.sofa) + .then (sofas) -&gt; + if sofas.length isnt 1 + res.status(404).send() + else + res.json(sofas[0]) + .fail (err) -&gt; res.status(err.statusCode).send() </span> exports.listen = (callback) -&gt; </code></pre> </div> <p>At this point we can execute <em>gulp test-int</em> command to run tests on real CouchDB:</p> <div class="language-bash highlighter-rouge"><pre class="highlight"><code>gulp <span class="nb">test</span>-int Sofas API /sofas ✓ should <span class="k">return </span>all sofas with GET /sofas/:sofa ✓ should <span class="k">return </span>a particular sofa with GET 2 passing <span class="o">(</span>204ms<span class="o">)</span> </code></pre> </div> <h2 id="mock-couch-tests">mock-couch tests</h2> <p>So how do we use mock-couch for this? Mock-couch is an HTTP server based on <a href="http://mcavage.me/node-restify/">Restify</a>. Note that <a href="https://nodejs.org/">Node.js</a> runtime is single-threaded therefore we won’t be able to run both express and mock-couch at the same time. However, it is possible to run mock-couch on another process.</p> <p>Node provides a couple of ways to create processes out of the box. We’ll use one called <a href="https://nodejs.org/api/child_process.html#child_process_child_process_fork_modulepath_args_options">forking</a>: not only we’ll be able to control the lifecycle of the forked child process but we’ll also have the ability to send and receive messages between parent and the child.</p> <p>OK, less talk and more forking. We’ll do the forking in <em>gulp test</em> task:</p> <div class="language-diff highlighter-rouge"><pre class="highlight"><code><span class="gh">diff --git a/gulpfile.coffee b/gulpfile.coffee index cbd61b5..1200834 100644 </span><span class="gd">--- a/gulpfile.coffee </span><span class="gi">+++ b/gulpfile.coffee </span><span class="gu">@@ -6,14 +6,48 @@ mocha = require 'gulp-mocha' </span> sources = tests: './specs/**/*.spec.coffee' <span class="gi">+ </span> # Task to run tests with mock-couch backend gulp.task 'test', -&gt; process.env.NODE_ENV = 'test' process.env.PORT = 3010 process.env.COUCH_PORT = 5987 process.env.DATABASE = 'sofas_test' <span class="gd">- gulp.src(sources.tests) - .pipe(mocha()) </span><span class="gi">+ + fork = require('child_process').fork + couchProcess = fork('node_modules/gulp/bin/gulp.js', ['mock-couch'], {cwd: __dirname}) + couchProcess.on 'message', (msg) =&gt; + if msg is 'listening' + gulp.src(sources.tests) + .pipe(mocha()) + .once('error', (err) -&gt; + process.exit(1) + ) + .once('end', -&gt; + process.exit() + ) + + process.on 'exit', -&gt; + couchProcess.kill() + +gulp.task 'mock-couch', -&gt; + process.env.NODE_ENV = 'test' + process.env.PORT = 3010 + process.env.COUCH_PORT = 5987 + process.env.DATABASE = 'sofas_test' + + mockCouch = require 'mock-couch' + couchdb = mockCouch.createServer() + DAL = require './dal' + dal = new DAL(process.env.COUCH_PORT, process.env.DATABASE) + docs = [dal._sofaViews] + docs = docs.concat dal._sofaSamples + + couchdb.addDB process.env.DATABASE, docs + + couchdb.listen process.env.COUCH_PORT, -&gt; + process.send 'listening' + </span> # Task to run integration tests (real CouchDB backend) gulp.task 'test-int', -&gt; </code></pre> </div> <p>Also, we need a slight change to the test setup:</p> <div class="language-diff highlighter-rouge"><pre class="highlight"><code><span class="gh">diff --git a/specs/sofas.spec.coffee b/specs/sofas.spec.coffee index efbff58..93e7bd2 100644 </span><span class="gd">--- a/specs/sofas.spec.coffee </span><span class="gi">+++ b/specs/sofas.spec.coffee </span><span class="gu">@@ -9,12 +9,17 @@ app = null </span> describe 'Sofas API', -&gt; beforeEach (done) -&gt; <span class="gd">- # Reset database and start express server before each test - dal = new DAL(process.env.COUCH_PORT, process.env.DATABASE) - dal.resetDatabase(true) - .then -&gt; - app = server.listen done - .fail (err) -&gt; done err </span><span class="gi">+ + if process.env.NODE_ENV is 'test-int' + # Reset database and start express server before each test + dal = new DAL(process.env.COUCH_PORT, process.env.DATABASE) + dal.resetDatabase(true) + .then -&gt; + app = server.listen done + .fail (err) -&gt; done err + else + app = server.listen done + </span> # Close express server after each test afterEach (done) -&gt; </code></pre> </div> <h2 id="moment-of-truth">Moment of Truth</h2> <p>If you try running <code class="highlighter-rouge">gulp test</code> right now, you should see similar output:</p> <div class="language-bash highlighter-rouge"><pre class="highlight"><code>gulp <span class="nb">test </span>Sofas API /sofas ✓ should <span class="k">return </span>all sofas with GET <span class="o">(</span>125ms<span class="o">)</span> /sofas/:sofa ✓ should <span class="k">return </span>a particular sofa with GET <span class="o">(</span>41ms<span class="o">)</span> 2 passing <span class="o">(</span>178ms<span class="o">)</span> </code></pre> </div> <p>As you can see the tests completed slightly faster than with real CouchDB. In a small code base like this, it’s hardly worth switching to mock-couch, if execution speed is your only motivation. However, with bigger codebases, the speed improvement can become very apparent.</p> <p>Besides speed, there’s another benefit to this approach: you don’t actually need CouchDB to test your project. The simplified build process is especially useful with Continuous Integration platforms like <a href="http://travis-ci.org/">Travis CI</a> and <a href="https://codeship.com">Codeship</a>.</p> <h2 id="resources">Resources</h2> <ul> <li><a href="https://github.com/reederz/express-mock-couch-showcase">Code on GitHub</a></li> <li><a href="https://github.com/chris-l/mock-couch">mock-couch</a></li> </ul> Proxy Custom Socket.IO Namespaces with NGINX 2015-05-21T22:30:00+02:00 https://fadeit.dk/blog /2015/05/21/socketio-namespaces-and-nginx <p>A while ago I implemented a chat server based on <a href="http://socket.io/">Socket.IO</a> for <a href="https://hopper.dk">hopper.dk</a>. In addition to the chat server, the system already had 2 HTTP servers: <a href="http://gunicorn.org/">Gunicorn</a> for our RESTful API and <a href="http://nginx.org/">NGINX</a> for serving <a href="https://angularjs.org/">AngularJS</a> frontend. To make SSL management easier I decided to put all 3 servers behind a single NGINX host. It all went pretty smoothly except for 1 thing- my chat server with a custom namespace was unreachable through NGINX.</p> <p>Let’s illustrate this with some code</p> <h2 id="socketio-server">Socket.IO Server</h2> <div class="language-javascript highlighter-rouge"><pre class="highlight"><code><span class="nx">io</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'socket.io'</span><span class="p">)();</span> <span class="nx">io</span><span class="p">.</span><span class="nx">listen</span><span class="p">(</span><span class="mi">3000</span><span class="p">);</span> <span class="kd">var</span> <span class="nx">myNamespace</span> <span class="o">=</span> <span class="nx">io</span><span class="p">.</span><span class="nx">of</span><span class="p">(</span><span class="s1">'/my-namespace'</span><span class="p">);</span> <span class="nx">myNamespace</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">'connection'</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">socket</span><span class="p">){</span> <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'someone connected'</span><span class="p">);</span> <span class="nx">socket</span><span class="p">.</span><span class="nx">emit</span><span class="p">(</span><span class="s1">'news'</span><span class="p">,</span> <span class="s1">'Hi there!!'</span><span class="p">);</span> <span class="nx">myNamespace</span><span class="p">.</span><span class="nx">emit</span><span class="p">(</span><span class="s1">'news'</span><span class="p">,</span> <span class="s1">'Someone joined the party!'</span><span class="p">);</span> <span class="p">});</span> </code></pre> </div> <h2 id="nginx-host">NGINX Host</h2> <div class="language-bash highlighter-rouge"><pre class="highlight"><code>server <span class="o">{</span> listen 8080; server_name localhost; location ~/<span class="o">(</span>my-namespace<span class="o">)</span>.<span class="k">*</span><span class="nv">$ </span><span class="o">{</span> proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Upgrade <span class="nv">$http_upgrade</span>; proxy_set_header Connection <span class="s2">"upgrade"</span>; <span class="o">}</span> <span class="o">}</span> </code></pre> </div> <h2 id="connection-attempts-fail-with-http-404">Connection attempts fail with HTTP 404</h2> <p><img src="/blog/assets/socketio-namespaces-and-nginx/fail.png" alt="Client connection attempts fail" /></p> <p>This setup results in Socket.IO client trying to establish a connection to the server with no success. After a bit of furious <em>DuckDuckGoing</em> and <em>Googling</em>, I gave up and decided to look for answers in <a href="https://github.com/Automattic/socket.io">Socket.IO’s code</a>.</p> <p>The solution is simple: in addition to proxying your custom namespace, you also have to proxy <code class="highlighter-rouge">/socket.io</code> resource to the Socket.IO server:</p> <div class="language-bash highlighter-rouge"><pre class="highlight"><code>server <span class="o">{</span> listen 8080; server_name localhost; location ~/<span class="o">(</span>my-namespace|socket<span class="se">\.</span>io<span class="o">)</span>.<span class="k">*</span><span class="nv">$ </span><span class="o">{</span> proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Upgrade <span class="nv">$http_upgrade</span>; proxy_set_header Connection <span class="s2">"upgrade"</span>; <span class="o">}</span> <span class="o">}</span> </code></pre> </div> <h2 id="success">Success</h2> <p><img src="/blog/assets/socketio-namespaces-and-nginx/success.png" alt="Client connection successfully established" /></p> <p>I hope this helps.</p> Serving generated images from memory 2015-04-30T14:15:00+02:00 https://fadeit.dk/blog /2015/04/30/python3-flask-pil-in-memory-image <p>In this post, we are going to implement a simple image placeholder service like <a href="https://placehold.it/">placehold.it</a>. Use-cases for disposable images extend beyond placeholder services, commonly used to encode text into an image in order to protect websites from being scraped for valuable information.</p> <p>A simple approach could be to generate an image, write it to disc and then serve the static file. One could improve the solution by using temporary files, however the process still remains inefficient. The solution, of course, is to use in-memory files to cut out the process of writing the file to the hard drive.</p> <p>We are using Pillow, a fork of Python’s PIL (Python Imaging Library) that is actively maintained. While there is information out there on how to write an image into memory using StringIO module for Python2, using Python3 version io.StringIO results in an error:</p> <div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span> <span class="n">File</span> <span class="s">"image.py"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">10</span><span class="p">,</span> <span class="ow">in</span> <span class="o">&lt;</span><span class="n">module</span><span class="o">&gt;</span> <span class="n">image</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">string_io</span><span class="p">,</span> <span class="s">'PNG'</span><span class="p">)</span> <span class="n">File</span> <span class="s">"~/.pyenv/versions/3.4.0/lib/python3.4/site-packages/PIL/Image.py"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1685</span><span class="p">,</span> <span class="ow">in</span> <span class="n">save</span> <span class="n">save_handler</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">fp</span><span class="p">,</span> <span class="n">filename</span><span class="p">)</span> <span class="n">File</span> <span class="s">"~/.pyenv/versions/3.4.0/lib/python3.4/site-packages/PIL/PngImagePlugin.py"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">631</span><span class="p">,</span> <span class="ow">in</span> <span class="n">_save</span> <span class="n">fp</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">_MAGIC</span><span class="p">)</span> <span class="nb">TypeError</span><span class="p">:</span> <span class="n">string</span> <span class="n">argument</span> <span class="n">expected</span><span class="p">,</span> <span class="n">got</span> <span class="s">'bytes'</span> </code></pre> </div> <p>As hinted in the error message, the solution is to use io.BytesIO instead. See a minimal working sample below:</p> <div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">io</span> <span class="kn">import</span> <span class="n">BytesIO</span> <span class="kn">from</span> <span class="nn">PIL</span> <span class="kn">import</span> <span class="n">Image</span><span class="p">,</span> <span class="n">ImageDraw</span> <span class="n">image</span> <span class="o">=</span> <span class="n">Image</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="s">"RGB"</span><span class="p">,</span> <span class="p">(</span><span class="mi">300</span><span class="p">,</span> <span class="mi">50</span><span class="p">))</span> <span class="n">draw</span> <span class="o">=</span> <span class="n">ImageDraw</span><span class="o">.</span><span class="n">Draw</span><span class="p">(</span><span class="n">image</span><span class="p">)</span> <span class="n">draw</span><span class="o">.</span><span class="n">text</span><span class="p">((</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">),</span> <span class="s">"This text is drawn on image"</span><span class="p">)</span> <span class="n">byte_io</span> <span class="o">=</span> <span class="n">BytesIO</span><span class="p">()</span> <span class="n">image</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">byte_io</span><span class="p">,</span> <span class="s">'PNG'</span><span class="p">)</span> </code></pre> </div> <p>Putting it all together, let’s implement a simple Flask service that takes one parameter, extracts dimensions, generates an image and writes the passed parameter as a text on the image. Flask is amazing, allowing us to write fully functional web service in just 30 lines of code:</p> <div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">re</span> <span class="kn">from</span> <span class="nn">io</span> <span class="kn">import</span> <span class="n">BytesIO</span> <span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span><span class="p">,</span> <span class="n">abort</span><span class="p">,</span> <span class="n">send_file</span> <span class="kn">from</span> <span class="nn">PIL</span> <span class="kn">import</span> <span class="n">Image</span><span class="p">,</span> <span class="n">ImageDraw</span> <span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span> <span class="nd">@app.route</span><span class="p">(</span><span class="s">'/&lt;dimensions&gt;'</span><span class="p">)</span> <span class="k">def</span> <span class="nf">generate_image</span><span class="p">(</span><span class="n">dimensions</span><span class="p">):</span> <span class="c">#Extract digits from request variable e.g 200x300</span> <span class="n">sizes</span> <span class="o">=</span> <span class="p">[</span><span class="nb">int</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s">r'</span><span class="err">\</span><span class="s">d+'</span><span class="p">,</span> <span class="n">dimensions</span><span class="p">)]</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">sizes</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">2</span><span class="p">:</span> <span class="n">abort</span><span class="p">(</span><span class="mi">400</span><span class="p">)</span> <span class="n">width</span> <span class="o">=</span> <span class="n">sizes</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="n">height</span> <span class="o">=</span> <span class="n">sizes</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="n">image</span> <span class="o">=</span> <span class="n">Image</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="s">"RGB"</span><span class="p">,</span> <span class="p">(</span><span class="n">width</span><span class="p">,</span> <span class="n">height</span><span class="p">))</span> <span class="n">draw</span> <span class="o">=</span> <span class="n">ImageDraw</span><span class="o">.</span><span class="n">Draw</span><span class="p">(</span><span class="n">image</span><span class="p">)</span> <span class="c">#Position text roughly in the center</span> <span class="n">draw</span><span class="o">.</span><span class="n">text</span><span class="p">((</span><span class="n">width</span><span class="o">/</span><span class="mi">2</span> <span class="o">-</span> <span class="mi">25</span><span class="p">,</span> <span class="n">height</span><span class="o">/</span><span class="mi">2</span> <span class="o">-</span> <span class="mi">5</span><span class="p">),</span> <span class="n">dimensions</span><span class="p">)</span> <span class="n">byte_io</span> <span class="o">=</span> <span class="n">BytesIO</span><span class="p">()</span> <span class="n">image</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">byte_io</span><span class="p">,</span> <span class="s">'PNG'</span><span class="p">)</span> <span class="n">byte_io</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="k">return</span> <span class="n">send_file</span><span class="p">(</span><span class="n">byte_io</span><span class="p">,</span> <span class="n">mimetype</span><span class="o">=</span><span class="s">'image/png'</span><span class="p">)</span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span> <span class="n">app</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">debug</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> </code></pre> </div> <p>To run this code, you may need to install few dependencies:</p> <div class="language-bash highlighter-rouge"><pre class="highlight"><code><span class="c">#Install dependencies</span> sudo apt-get install virtualenv python3.4-dev python-pip <span class="c">#Create virtual environment</span> virtualenv -p /usr/bin/python3.4 venv <span class="c">#Activate virtual environment</span> <span class="nb">source </span>venv/bin/activate <span class="c">#Install libraries</span> pip install flask Pillow <span class="c">#Run server</span> python server.py <span class="c">#http://localhost:5000/200x300</span> </code></pre> </div> <p><img src="/blog/assets/python3-flask-pil-in-memory-image/sample.png" alt="Generated image" /></p> SQLAlchemy with PostgreSQL earthdistance extension 2015-03-19T20:30:16+01:00 https://fadeit.dk/blog /2015/03/19/sqlalchemy-postgresql-earthdistance <p>The fact that Earth is round, but not a sphere makes calculating distances difficult. Fortunately PostgreSQL has an extension for just doing that - earthdistance. Doing the calculations in the database is superior as extensions are compiled and it enables us to build an index to reduce the amount of calculations needed to make at query time. I came across <a href="http://johanndutoit.net/searching-in-a-radius-using-postgres/">this</a> article to get me started, however considering I spent more time on making earthdistance work with SQLAlchemy, I decided to share my solution. The example below assumes that you already have earthdistance set up and working with a ORM model class ‘Company’ that has ‘lat’ and ‘lng’ fields present.</p> <div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">sqlalchemy</span> <span class="kn">import</span> <span class="n">func</span> <span class="kn">from</span> <span class="nn">example.model</span> <span class="kn">import</span> <span class="n">Company</span> <span class="kn">from</span> <span class="nn">example.core</span> <span class="kn">import</span> <span class="n">db</span> <span class="k">def</span> <span class="nf">get_companies_from_amsterdam</span><span class="p">():</span> <span class="c">#Calculating each company's distance from Amsterdam.</span> <span class="n">loc_amsterdam</span> <span class="o">=</span> <span class="n">func</span><span class="o">.</span><span class="n">ll_to_earth</span><span class="p">(</span><span class="mf">52.3667</span><span class="p">,</span> <span class="mf">4.9000</span><span class="p">)</span> <span class="n">loc_company</span> <span class="o">=</span> <span class="n">func</span><span class="o">.</span><span class="n">ll_to_earth</span><span class="p">(</span><span class="n">Company</span><span class="o">.</span><span class="n">lat</span><span class="p">,</span> <span class="n">Company</span><span class="o">.</span><span class="n">lng</span><span class="p">)</span> <span class="n">distance_func</span> <span class="o">=</span> <span class="n">func</span><span class="o">.</span><span class="n">earth_distance</span><span class="p">(</span><span class="n">loc_amsterdam</span><span class="p">,</span> <span class="n">loc_company</span><span class="p">)</span> <span class="n">query</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">Company</span><span class="p">,</span> <span class="n">distance_func</span><span class="p">)</span><span class="o">.</span><span class="n">order_by</span><span class="p">(</span><span class="n">distance_func</span><span class="p">)</span> <span class="c">#Resultset is no longer list of Company, but a list of tuples.</span> <span class="n">result</span> <span class="o">=</span> <span class="n">query</span><span class="o">.</span><span class="nb">all</span><span class="p">()</span> <span class="n">mapped</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">result</span><span class="p">:</span> <span class="n">company</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="n">company</span><span class="o">.</span><span class="n">distance</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="n">mapped</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">company</span><span class="p">)</span> <span class="k">return</span> <span class="n">mapped</span> </code></pre> </div> <p>To speed up things, we can pre-compute <em>ll_to_earth(Company.lat, Company.lng)</em> using an index:</p> <div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">sqlalchemy</span> <span class="kn">import</span> <span class="n">func</span><span class="p">,</span> <span class="n">Index</span> <span class="kn">from</span> <span class="nn">example.model</span> <span class="kn">import</span> <span class="n">Company</span> <span class="n">Index</span><span class="p">(</span><span class="s">'company_distance'</span><span class="p">,</span> <span class="n">func</span><span class="o">.</span><span class="n">ll_to_earth</span><span class="p">(</span><span class="n">Company</span><span class="o">.</span><span class="n">lat</span><span class="p">,</span> <span class="n">Company</span><span class="o">.</span><span class="n">lng</span><span class="p">),</span> <span class="n">postgresql_using</span><span class="o">=</span><span class="s">'gist'</span><span class="p">)</span> </code></pre> </div>