- grunt 2017-03-19T16:58:15+01:00 https://fadeit.dk/blog/tag/grunt.html Loading translations from Google Spreadsheets 2015-09-04T11:16:30+02:00 https://fadeit.dk/blog /2015/09/04/managing-translations-with-gulp-or-grunt <h2 id="introduction">Introduction</h2> <p>This is a follow-up post to our <a href="https://fadeit.dk/blog/post/managing-angular-translate-translations">previous</a> translation management article. To recap - in that article we solved the issue of loading ‘fresh’ translation on development server using nginx and proxying translation requests to the api which in turn will fetch new translations from Google Spreadsheets. It can be seen as overkill - this flow complicates nginx configuration, slows down translation loading on every refresh and will make a ton of (unnecessary) requests while developing. While it’s nice to always get the latest translations, truth is that most of the time you don’t need to fetch translations from the cloud. In this post we’ll explore loading translations only during the build process, thus eliminating the need to proxy requests and write api endpoint in the first place.</p> <h2 id="workflow">Workflow</h2> <p>Since writing previous post, I have also changed my mind about lazy-loading translations using partial loader. This is primarily due to the lengths I had to go to cloak translations while they’re loading, especially considering that the size of the translations file is marginal for most applications. By packaging translations in the same minified file as the application I don’t have to worry about <a href="https://fadeit.dk/blog/post/preload-angular-translate-partial-loader">pre-loading</a> translations or flickering as they are being loaded. So we will introduce an extra task to our workflow that will fetch translations from Google Spreadsheets and converts it to a javascript file ready to be used by angular-translate.</p> <h2 id="using-gulp">Using Gulp</h2> <p>Although here at fadeit we started out with using Grunt exclusively, we’ve come to appreciate Gulp more in our recent projects. Primarily since writing custom tasks in good ol’ JavaScript is more intuitive as opposed to Grunt’s configuration approach. Surely piping the result of previous task directly to next one without having to write it to disk is a plus, but not in current context.</p> <div class="language-javascript highlighter-rouge"><pre class="highlight"><code><span class="kd">var</span> <span class="nx">gulp</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'gulp'</span><span class="p">);</span> <span class="kd">var</span> <span class="nx">request</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'request'</span><span class="p">);</span> <span class="kd">var</span> <span class="nx">fs</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'fs'</span><span class="p">);</span> <span class="nx">gulp</span><span class="p">.</span><span class="nx">task</span><span class="p">(</span><span class="s1">'translations'</span><span class="p">,</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">cb</span><span class="p">)</span> <span class="p">{</span> <span class="kd">var</span> <span class="nx">uuid</span> <span class="o">=</span> <span class="s1">'1FsVuRLbtgxMZvWd4mpnKiAhqVYap-ZAx08LBeZ9HFJk'</span><span class="p">;</span> <span class="kd">var</span> <span class="nx">page</span> <span class="o">=</span> <span class="s1">'od6'</span> <span class="kd">var</span> <span class="nx">url</span> <span class="o">=</span> <span class="s1">'https://spreadsheets.google.com/feeds/list/'</span> <span class="o">+</span> <span class="nx">uuid</span> <span class="o">+</span> <span class="s1">'/'</span> <span class="o">+</span> <span class="nx">page</span> <span class="o">+</span> <span class="s1">'/public/values?alt=json'</span><span class="p">;</span> <span class="nx">request</span><span class="p">(</span><span class="nx">url</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span> <span class="nx">res</span><span class="p">,</span> <span class="nx">body</span><span class="p">){</span> <span class="kd">var</span> <span class="nx">json</span> <span class="o">=</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">parse</span><span class="p">(</span><span class="nx">body</span><span class="p">);</span> <span class="kd">var</span> <span class="nx">translations</span> <span class="o">=</span> <span class="p">{</span> <span class="s1">'en'</span><span class="p">:</span> <span class="p">{},</span> <span class="s1">'da'</span><span class="p">:</span> <span class="p">{}</span> <span class="p">};</span> <span class="k">for</span><span class="p">(</span><span class="kd">var</span> <span class="nx">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">i</span> <span class="o">&lt;</span> <span class="nx">json</span><span class="p">.</span><span class="nx">feed</span><span class="p">.</span><span class="nx">entry</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span> <span class="nx">i</span><span class="o">++</span><span class="p">){</span> <span class="kd">var</span> <span class="nx">entry</span> <span class="o">=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">feed</span><span class="p">.</span><span class="nx">entry</span><span class="p">[</span><span class="nx">i</span><span class="p">];</span> <span class="kd">var</span> <span class="nx">key</span> <span class="o">=</span> <span class="nx">entry</span><span class="p">.</span><span class="nx">gsx$key</span><span class="p">.</span><span class="nx">$t</span><span class="p">;</span> <span class="kd">var</span> <span class="nx">enValue</span> <span class="o">=</span> <span class="nx">entry</span><span class="p">.</span><span class="nx">gsx$en</span><span class="p">.</span><span class="nx">$t</span><span class="p">;</span> <span class="kd">var</span> <span class="nx">daValue</span> <span class="o">=</span> <span class="nx">entry</span><span class="p">.</span><span class="nx">gsx$da</span><span class="p">.</span><span class="nx">$t</span><span class="p">;</span> <span class="nx">translations</span><span class="p">.</span><span class="nx">en</span><span class="p">[</span><span class="nx">key</span><span class="p">]</span> <span class="o">=</span> <span class="nx">enValue</span><span class="p">;</span> <span class="nx">translations</span><span class="p">.</span><span class="nx">da</span><span class="p">[</span><span class="nx">key</span><span class="p">]</span> <span class="o">=</span> <span class="nx">daValue</span><span class="p">;</span> <span class="p">}</span> <span class="nx">writeTranslations</span><span class="p">(</span><span class="nx">translations</span><span class="p">,</span> <span class="nx">options</span><span class="p">.</span><span class="nx">src</span> <span class="o">+</span> <span class="s1">'/app/translations.js'</span><span class="p">,</span> <span class="nx">cb</span><span class="p">);</span> <span class="p">});</span> <span class="p">});</span> <span class="kd">function</span> <span class="nx">writeTranslations</span><span class="p">(</span><span class="nx">translations</span><span class="p">,</span> <span class="nx">file</span><span class="p">,</span> <span class="nx">cb</span><span class="p">){</span> <span class="kd">var</span> <span class="nx">contents</span> <span class="o">=</span> <span class="s2">"var appModule = angular.module('fadeit');\n"</span><span class="p">;</span> <span class="nx">contents</span> <span class="o">+=</span> <span class="s2">"appModule.config(function($translateProvider){\n"</span><span class="p">;</span> <span class="k">for</span><span class="p">(</span><span class="kd">var</span> <span class="nx">property</span> <span class="k">in</span> <span class="nx">translations</span><span class="p">){</span> <span class="kd">var</span> <span class="nx">lang</span> <span class="o">=</span> <span class="nx">translations</span><span class="p">[</span><span class="nx">property</span><span class="p">];</span> <span class="nx">contents</span> <span class="o">+=</span> <span class="s2">"$translateProvider.translations('"</span> <span class="o">+</span> <span class="nx">property</span> <span class="o">+</span> <span class="s2">"',\n"</span><span class="p">;</span> <span class="nx">contents</span> <span class="o">+=</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">stringify</span><span class="p">(</span><span class="nx">lang</span><span class="p">);</span> <span class="nx">contents</span> <span class="o">+=</span> <span class="s2">");\n"</span><span class="p">;</span> <span class="p">}</span> <span class="nx">contents</span> <span class="o">+=</span> <span class="s2">"});"</span><span class="p">;</span> <span class="nx">fs</span><span class="p">.</span><span class="nx">writeFile</span><span class="p">(</span><span class="nx">file</span><span class="p">,</span> <span class="nx">contents</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">err</span><span class="p">){</span> <span class="k">if</span><span class="p">(</span><span class="nx">err</span><span class="p">){</span> <span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="s1">'Writing translations failed:'</span><span class="p">);</span> <span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="nx">err</span><span class="p">);</span> <span class="p">}</span> <span class="nx">cb</span><span class="p">();</span> <span class="p">});</span> <span class="p">}</span> </code></pre> </div> <p>As seen from the code, couple modules are used - namely <code class="highlighter-rouge">request</code> to fetch the JSON data and <code class="highlighter-rouge">fs</code> to write a file. The task won’t only write the file, but it will also generate an angular.js module file that will configure translations ready to be used. It can be seen a bit hacky to generate javascript file like that, but I’ve yet to come up with more clean solution. The upside is that this code is not prone to change and I haven’t touched it since writing it. There’s caveat though - if you are using <a href="http://www.browsersync.io/">browsersync</a>, then it may trigger refresh before the translations are actually pulled (because other files changed before). To fix this, we need to make sure that translations are loaded before any other tasks take place. It is easily solved with gulp-sync module, to execute certain tasks synchronously. Same would also apply if the process will concatenate files together - we need to make sure new translations are loaded before that happens. In the sample build process below I’ve split up the build process in two portions where each of them is executed asynchronously. The callback at the end of the gulp task is necessary so that gulp-sync knows when to start processing next batch of tasks.</p> <div class="language-javascript highlighter-rouge"><pre class="highlight"><code><span class="kd">var</span> <span class="nx">gulpsync</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'gulp-sync'</span><span class="p">)(</span><span class="nx">gulp</span><span class="p">);</span> <span class="nx">gulp</span><span class="p">.</span><span class="nx">task</span><span class="p">(</span><span class="s1">'build'</span><span class="p">,</span> <span class="nx">gulpsync</span><span class="p">.</span><span class="nx">sync</span><span class="p">([[</span><span class="s1">'translations'</span><span class="p">,</span> <span class="s1">'clean'</span><span class="p">],</span> <span class="p">[</span><span class="s1">'html'</span><span class="p">,</span> <span class="s1">'scripts'</span><span class="p">]]));</span> </code></pre> </div> <h2 id="using-grunt">Using Grunt</h2> <p>Firstly I must apologise as I did not take time to develop a pure grunt way of doing so. This is because it started out as a python script which I later incorporated in the grunt workflow. The script will serve same purpose as the one above, with a minor difference of calling an external python executeable instead.</p> <div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">json</span> <span class="kn">import</span> <span class="nn">urllib.request</span> <span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">OrderedDict</span> <span class="n">sheet_id</span> <span class="o">=</span> <span class="s">'1FsVuRLbtgxMZvWd4mpnKiAhqVYap-ZAx08LBeZ9HFJk'</span><span class="p">;</span> <span class="n">page_id</span> <span class="o">=</span> <span class="s">'od6'</span> <span class="n">base</span> <span class="o">=</span> <span class="s">'https://spreadsheets.google.com/feeds/list/{0}/{1}/public/values?alt=json'</span> <span class="n">translation_config</span> <span class="o">=</span> <span class="s">'src/app/translations.js'</span> <span class="n">url</span> <span class="o">=</span> <span class="n">base</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">sheet_id</span><span class="p">,</span> <span class="n">page_id</span><span class="p">)</span> <span class="n">json_data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">url</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s">'utf8'</span><span class="p">))</span> <span class="c">#we need to build different data structure</span> <span class="n">en_translation_map</span> <span class="o">=</span> <span class="n">OrderedDict</span><span class="p">()</span> <span class="n">da_translation_map</span> <span class="o">=</span> <span class="n">OrderedDict</span><span class="p">()</span> <span class="k">for</span> <span class="n">entry</span> <span class="ow">in</span> <span class="n">json_data</span><span class="p">[</span><span class="s">'feed'</span><span class="p">][</span><span class="s">'entry'</span><span class="p">]:</span> <span class="n">key</span> <span class="o">=</span> <span class="n">entry</span><span class="p">[</span><span class="s">'gsx$key'</span><span class="p">][</span><span class="s">'$t'</span><span class="p">]</span> <span class="n">en_value</span> <span class="o">=</span> <span class="n">entry</span><span class="p">[</span><span class="s">'gsx$en'</span><span class="p">][</span><span class="s">'$t'</span><span class="p">]</span> <span class="n">da_value</span> <span class="o">=</span> <span class="n">entry</span><span class="p">[</span><span class="s">'gsx$da'</span><span class="p">][</span><span class="s">'$t'</span><span class="p">]</span> <span class="n">en_translation_map</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">en_value</span> <span class="n">da_translation_map</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">da_value</span> <span class="c">#Javascript for translations file we are generating</span> <span class="n">config</span> <span class="o">=</span> <span class="s">""" var appModule = angular.module('fadeit'); appModule.config(function($translateProvider){ """</span> <span class="n">en_translations</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">en_translation_map</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">4</span><span class="p">)</span> <span class="n">da_translations</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">da_translation_map</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">4</span><span class="p">)</span> <span class="n">config</span> <span class="o">=</span> <span class="n">config</span> <span class="o">+</span> <span class="s">"</span><span class="se">\n</span><span class="s">$translateProvider.translations('en',"</span> <span class="o">+</span> <span class="n">en_translations</span> <span class="o">+</span> <span class="s">');'</span> <span class="n">config</span> <span class="o">=</span> <span class="n">config</span> <span class="o">+</span> <span class="s">"</span><span class="se">\n</span><span class="s">$translateProvider.translations('da',"</span> <span class="o">+</span> <span class="n">da_translations</span> <span class="o">+</span> <span class="s">');'</span> <span class="n">config</span> <span class="o">=</span> <span class="n">config</span> <span class="o">+</span> <span class="s">"</span><span class="se">\n</span><span class="s">});"</span> <span class="c">#close function</span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">translation_config</span><span class="p">,</span> <span class="s">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">outfile</span><span class="p">:</span> <span class="n">outfile</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">config</span><span class="p">)</span> </code></pre> </div> <p>Now we can save this file in the project (in our case src/translations.py) and call it using grunt-shell during build time.</p> <div class="language-javascript highlighter-rouge"><pre class="highlight"><code><span class="nx">shell</span><span class="err">:</span> <span class="p">{</span> <span class="nl">translations</span><span class="p">:</span> <span class="p">{</span> <span class="nl">options</span><span class="p">:</span> <span class="p">{</span> <span class="nl">stdout</span><span class="p">:</span> <span class="kc">true</span> <span class="p">},</span> <span class="nx">command</span><span class="err">:</span> <span class="s1">'python3 src/translations.py'</span> <span class="p">}</span> <span class="p">}</span> </code></pre> </div> <p>Since grunt executes tasks synchronously, all there is left to do is call this task in the beginning of the build process and it’ll generate a following file.</p> <div class="language-javascript highlighter-rouge"><pre class="highlight"><code><span class="kd">var</span> <span class="nx">appModule</span> <span class="o">=</span> <span class="nx">angular</span><span class="p">.</span><span class="nx">module</span><span class="p">(</span><span class="s1">'fadeit'</span><span class="p">);</span> <span class="nx">appModule</span><span class="p">.</span><span class="nx">config</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">$translateProvider</span><span class="p">)</span> <span class="p">{</span> <span class="nx">$translateProvider</span><span class="p">.</span><span class="nx">translations</span><span class="p">(</span><span class="s1">'en'</span><span class="p">,</span> <span class="p">{</span> <span class="s2">"HELLO"</span><span class="p">:</span> <span class="s2">"Hello"</span> <span class="p">}</span> <span class="p">);</span> <span class="nx">$translateProvider</span><span class="p">.</span><span class="nx">translations</span><span class="p">(</span><span class="s1">'da'</span><span class="p">,</span> <span class="p">{</span> <span class="s2">"HELLO"</span><span class="p">:</span> <span class="s2">"Hej"</span> <span class="p">}</span> <span class="p">);</span> <span class="p">});</span> </code></pre> </div> <h2 id="closing-words">Closing words</h2> <p>Over-all I’ve come to prefer loading translations during build time instead. While it will add few seconds to the build process, I will make it up by not having to wait for translations being proxied on every reload. If I already have a server running and just wish to reload translations, I can issue the command directly instead of re-building everything. For gulp it would be <code class="highlighter-rouge">gulp translations</code> and for grunt <code class="highlighter-rouge">grunt shell:translations</code>. I’ve also decided to ditch versioning translations and generally will add <code class="highlighter-rouge">src/translations.js</code> to <code class="highlighter-rouge">.gitignore</code>.</p>