Tuesday, 14 March 2017

Building and deploying Mojolicious projects guide

I recently had to write some Mojolicious documentation for my final year project and for $work and I decided to use some of the knowledge I gained to put this guide together. It will hopefully help others who might be confused about how to get started using Mojolicious with Carton and Plenv. By no means I believe it is complete and I know some more experienced Perl developers will want to correct some or point out some mistakes. 

This structure assumes your project is named “ExampleApp”, therefore controllers and other files will use this notation. In real examples, the files may be called something different.

Table of Contents


Project structure


Mojolicious Application Structure Info
lib/ The lib folder is the main folder, contains most logic, controller and models.
ExampleApp/

  • Controller/
  • [Schema/]
    • Result/
    • [ResultSet/]
  • Schema.pm

ExampleApp.pm

Each Controller has .pm file, and potentially sub-controllers which go in folders under the same name. Schema folder holds Model information (DB Schemas, Results and ResultSets)
[local/] CARTON OPTIONAL/NOT PART OF MAIN PROJECT STRUCTURE

This folder may appear on production machines. It contains local installations of dependency modules for this project. Managed & installed through Carton.

[log/] Logging folder, usually has a development.log and production.log file.
[development.log]

[production.log]

public/ All external files go in the public folder (Bootstrap, JS files, fonts, images etc). Can have any sub-structure.
script/ Scripts that can be run using the controllers and their actions. By default, it has the main application start script.
example_appn Main script, it starts the application by loading files from the lib folder.
t/ Tests folder, all tests go in here (if there are any). They all end in the .t extension, but may be grouped in folders.
templates/ Page templates (for the View) which are structured according to the Controllers.
layouts/

example_app/

Layouts are (re)usable views/templates. Files can contain placeholder or perl tags to render elements that are passed to the view down from the controller.
[vendor/] CARTON OPTIONAL/NOT PART OF MAIN PROJECT STRUCTURE

This is a Carton generated folder and may appear in some projects. Contains the local dependencies for this project. Should be

[cpanfile] Dependencies manager file, specifies what perl module/packages this project relies on. Makes installing dependencies easier as the dependency installer can go through the list and their dependencies as well automatically.
[cpanfile.snapshot] CARTON OPTIONAL/NOT PART OF MAIN PROJECT STRUCTURE

File auto-generated and used by Carton.

example_app.conf Main config file, the application reads from this file before it starts, it can hold details or params to be used in the Controllers. Sensitive data should not be added to this conf file.
[example_app.development.conf]

[example_app.production.conf]

[example_app.sample.conf]

Alternatively, because no sensitive data should be committed in the repository, one can include a sample file with the expected config file structure.
For each running mode, Mojolicious can read from the main config file as well as production/development if database connections details are different, for instance.

Square brackets [ ] denote optionality, meaning the folder or files may or may not be present. E.g. One can choose to not enable logging for the application, in which case they don’t need to create a log folder.

What is CPAN/MetaCPAN?

The Comprehensive Perl Archive Network (CPAN) is a repository of over 250,000 software modules and accompanying documentation for 39,000 distributions, written in the Perl programming language by over 12,000 contributors. CPAN can denote either the archive network itself, or the Perl program that acts as an interface to the network and as an automated software installer (somewhat like a package manager). Most software on CPAN is free and open source software.

Links:

CPAN can be installed as a Linux module under system Perl environment with

Or, if a Perl environment manager like plenv or perlbrew is installed, with

Plenv has a shortcut for installing cpanm too:

After installing cpanm, to use it for installing a specific module system wide run:

How does Mojolicious work with CPAN?


Mojolicious applications may use and re-use Perl modules available on CPAN. Some of these are written especially for Mojolicious, but most modules are written to be framework and project agnostic, so they can be “plug-n-play”.

Any Perl module (.pm file) may use other Perl modules by including them at the top of the time or inline by writing this line use Perl::ModuleName. This will work just fine if the user already has that module installed on their machine, but on new machines, the module may not be installed and the application will fail to start. That is why most Perl applications keep track of their dependencies in a cpanfile.

Most Perl applications and projects keep track of their dependencies in a cpanfile file. These can be installed with cpanm and/or Carton.

Mojolicious projects also uses the cpanfile to keep track of their dependency, but it’s up to the user to keep the cpanfile up to date. A user may request the general package or a specific version of it, in order to ensure the project runs correctly and newer versions don’t break functionality.

Cpanfile example


To install all dependencies for a project one can run the following command while in the project directory:

This command will install all dependencies globally. However there are other ways to manage dependencies, using Carton.

By default, all modules are installed under system Perl (the Perl version installed on the machine you are using). If you plan on using Plenv or perlbrew, make sure the modules are installed under the Perl environment and version you need, otherwise your application may not recognise they are installed.

What is Plenv?


https://github.com/tokuhirom/plenv

Use plenv to install, manage and pick a Perl version for your application and guarantee that your development environment matches production. Put plenv to work with Carton for painless Perl upgrades and bulletproof deployments.

When and why would one use Plenv?


Plenv manages multiple Perl environments. For instance, let’s assume the production machine runs Perl 5.20.1 and that cannot be changed because some other applications running on that server rely on it. Your new application requires a newer version of Perl, namely 5.22.0. Plenv can install and manage any Perl version and allow your new application to run using the newer version with little hassle.

It also helps developers test features locally, if their local version of Perl is different than the one in production.

How to install and use Plenv


A full Plenv installation guide is available here:

https://github.com/tokuhirom/plenv#installation

After installing a new Perl version, one can switch between them locally with

Or globally:

After any change or module installation it’s recommended you run:

The version of Perl the application will use can be forced/overridden by specifying the PLENV_VERSION Environment Variable when the application is run. E.g.

 

What is Carton?


https://github.com/perl-carton/carton

Carton is a command line tool to track the Perl module dependencies for your Perl application. Dependencies are declared using cpanfile format, and the managed dependencies are tracked in a cpanfile.snapshot file, which is meant to be version controlled, and the snapshot file allows other developers of your application will have the exact same versions of the modules.

With Carton installed, a developer can download all dependencies locally, and Carton will put them in the local/ directory. Carton will look into the cpanfile file for the dependencies your project needs.

What does Carton exactly do?


It does a couple of things and the developer can choose how they use it, but mainly it manages the dependencies. Here is in a few lines how it can be used and what it can be used for:

  • Carton keeps track of the packages and dependencies’ versions required for your application to run correctly (in the cpanfile.snapshot file)
  • Carton can download and install those dependencies in a local directory, without interfering with other projects that might need other versions of the same dependencies.
  • Carton can create a bundle with those dependencies that you can push as part of your project to make sure no matter what happens to remote source, you always have the versions you tested your app with and know it works.

One can choose to do only 1 and 2, or all three.

Why would I use Carton?


Some might question what are the benefits of using Carton over CPANM.

There are many reasons, but here are some:

  1. If your dependencies suddenly becomes unavailable to download from the internet, you can use the bundle you have with the right versions you know will work (or have tested the application against).
  2. You ensure your project’s independency of other factors (network offline, packages include breaking changes etc).
  3. You make sure everyone in your development team works with the exact same dependencies and their versions, putting an end to “But it works on my machine” arguments.
  4. Sometimes your dependency’s dependencies may become unavailable in which case you might end up with a broken dependency. Carton makes sure you have everything you need.

Carton can be used instead of cpanm and in conjunction with Plenv. Instead of cpanm doesn’t mean you don’t need cpanminus installed, as Carton uses it to fetch the dependencies. It only means you will no longer use cpanm as a command to install and manage a project’s dependencies.

Understanding Mojolicious deployment modes


As mentioned before and as the Mojolicious project structure shows, Mojolicious apps (often referred as Mojo apps) can be run either in development mode or in production mode.

Development is more verbose and allows better debugging, exposes sensitive data and dumps cookies, params and sensitive database connection, which is why it is not recommended for use in production. To run a Mojo app in development mode, use morbo command, which runs in foreground.

In contrast, production mode will fail without exposing sensitive data, but it also makes hot deployment easier. To run a Mojo app in production mode, use the hypnotoad command to spin up a process that will be tracked and run in the background.

To stop it, pass the -s option. To hot deploy it, after making your changes re-run the command above and hypnotoad will make sure you have zero downtime.

Based on the mode you’re running the application, Mojolicious will read the main example_app.conf file and the mode specific configuration file and combine them. You don’t need to have separate config files, but they can prove useful.

How to deploy a Mojolicious project?


Deployment outline:

 

Set up the system with Plenv.


This is not a mandatory step, so consult with the project README file or make sure you’re aware of the ways the machine you’re deploying to is set up.

For how to set up the Plenv environment refer to “What is Plenv?” section.

Pull the repository using git.

Mojolicious – and Perl applications in general – use a cpanfile to determine what Perl dependencies need to be installed. It is the user’s responsibility to keep the cpanfile updated with all the packages they are including in their applications.

Obviously Perl will need to be installed on the server that you will run the application on. On top of that, the application’s dependencies will need to be installed as well.

If using Plenv, make sure you prepend all commands with the plenv version you need (or set it locally/globally) as well as plenv exec.

To install the dependencies one will need to have cpanminus installed. Cpanminus is a dependency installer for Linux based systems. For more information please refer to “What is CPANM/MetaCPAN?” section.

Alternatively, if the developer used Carton to manage the dependencies, use the carton command to install the dependencies. Append with –cached if you want to use the included modules from the project bundle. For more information please refer to “What is Carton?” section

How do I know if the developer used Carton?


Check the project structure and look for hints of Carton (highlighted optional folders and file). Generally a cpanfile.snapshot is a clear evidence you should be using Carton to manage dependencies.

Always use Carton if available instead of cpanm. When using Carton prepend your run command with carton exec to let the application know it should use the Carton fetched and installed packages.

Configuration files


Mojolicious applications can run in two modes – development and production – and one can specify a main config file as well as separate deployment mode config files. Mojolicious will read both the main config file as well as the mode-specific config file and combine those. For more information please refer to “Understanding Mojolicious deployment modes” section.

Because they normally include sensitive data, the config files are normally not committed with the repository. Instead, a sample file is provided. Here is an example of a sample main config file which shows what goes.

The sample mode-specific should give you an idea of what the application’s config file needs to contain in order to run. This may contain database connection information, for instance.

Make sure your machine has access to the database you’re trying to connect to, otherwise you will get an error when loading the pages.

Run the web server


Mojolicious applications can be run as CGI or as web servers. Mojolicious comes with two solutions, morbo which is meant to be used in development and hypnotoad which is a production web server tool which allows really simple server reload and hot deployment, without ever taking it down.

All following commands can be run under different Plenv environments and in conjunction with Carton so you might need to append the right commands.

Development


For development go into the project top level folder and run

This will start the server in development mode, and by default running on port :3000. To stop it, just Ctrl+C it. You can override the port the application listens to by passing on the -l “https://*:3001” option to the command.

Production


For production run the following command:

This will start the server in production mode and keep in in the background. You make make changes to the files and folder, but they will only take effect when you reload/hot deploy the server again by re-running the same command. Hypnotoad will spin up another instance before it shuts down the previous one ensuring the server has zero downtime.

To stop the server just run:

If used with Plenv and Carton you’d probably need to run something like:

Where is the projects’ logs information?


By default, Mojo applications don’t save the log in production mode, whereas in development mode it just dumps it to the console. To make sure you save the log, create a log folder in the projects and the application will save all logs in either development.log or production.log file.

Please refer to the Project Structure section for more information on log folder.

For more information consult the Mojolicious Documentation.

Saturday, 25 February 2017

Check Uber estimated times and prices from command line

Lately I’ve found myself using Uber exclusively for traveling around when I need to. But I’ve also found myself waiting just a little bit longer to see if that surge price goes down. It just seems that there is always surge pricing, and it’s so annoying trying to refresh from the application because you never know if it’s a new price or the old one. But I’ve come to accept it because I trust it more than I trust regular cabs and it’s also a lot more convenient. So if I am in front of my laptop now I can use this, because I iz developer, right?

That being said, as a lazy person it pains me everytime open my phone, open the Uber app, type my destination, and see the estimated price, only for my inner, responsible, cost-cutting, fiduciary-self to end up taking the bus all the way home. Jae Bradley

You’re not the only one…

Check estimated time and prices for your next ride from the command line using Uber CLI →

Quite fun and also pretty good. Also check out these other nifty Uber projects to play around with that I discovered on ProductHunt:

However, ever since this news broke out, I am seriously considering ceasing the use of the app and joining the #DeleteUber movement. Just waiting on the independent review outcome.

Friday, 24 February 2017

List of sites possibly affected by Cloudflare’s Traffic Leak

Great news this morning from Cloudfare →

[…] in some unusual circumstances […] our edge servers were running past the end of a buffer and returning memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data. And some of that data had been cached by search engines.

List of Sites possibly affected by Cloudflare’s #Cloudbleed HTTPS Traffic Leak →

Between 2016-09-22 – 2017-02-18 passwords, private messages, API keys, and other sensitive data were leaked by Cloudflare to random requesters. Data was cached by search engines, and may have been collected by random adversaries over the past few months.

Requests to sites with the HTML rewrite features enabled triggered a pointer math bug. Once the bug was trigerred the response would include data from ANY other cloudfare proxy customer that happened to be in memory at the time. Meaning a request for a page with one of those features could include data from Uber or one of the many other customers that didn’t use those features. So the potential impact is every single one of the sites using CloudFare’s proxy services (including HTTP & HTTPS proxy).

Emphasis:

This list contains all domains that use cloudflare DNS, not just the cloudflare proxy (the affected service that leaked data). It’s a broad sweeping list that includes everything. Just because a domain is on the list does not mean the site is compromised, and sites may be compromised that do not appear on this list.

Cloudfare has been working on it before it made the announcement.

The infosec team worked to identify URIs in search engine caches that had leaked memory and get them purged. With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory. Those 770 unique URIs covered 161 unique domains. The leaked memory has been purged with the help of the search engines.

But just to be sure, change your passwords & clear your cookiesGitHub users kamaljoshy and avian2 have written some scripts to check domains of saved logins for Chrome and Firefox.

https://gist.github.com/kamaljoshi/2cce5f6d35cd28de8f6dbb27d586f064
https://gist.github.com/avian2/30db0d579732287d758c21ba8ded9393

Update:

Monzo’s Response to Cloudbleed →

Good to see a transparent bank. Anyone willing to try it out, I have a Golden Ticket.

Sunday, 19 February 2017

How to enable Hey Siri on Mac with your voice

When Siri for Mac was first announced at WWDC 2016 the first question I asked myself whether it would be available as a voice command. That proved not to be the case, at least not by default, so here is how you can create a dictation trigger to summon Siri on your Mac using the “Hey Siri” command. You can watch the whole video on YouTube.

How to enable Siri using “Hey Siri” command on your Mac

  1. Open System Preferences
    The System Preferences icon can normally be found in the dock, or you can use Spotlight to search for it.

  2. Select Siri and choose “Press Fn (Function) Space” as the Keyboard Shortcut for triggering Siri. Alternatively you may customise the shortcut. I chose Option+S. Note down your shortcut as you will need to perform it later.
  3. Go back to the main window in System Preferences and make sure you have Dictation enabled under Keyboard -> Dictation.
  4. Go back to the main window in System Preferences and choose the Accessibility option. Scroll down in the right panel and select Dictation under Interacting.
  5. Check Enable the dictation keyword phrase
  6. Replace the word Computer with Hey

  7. Click Dictation Commands… at the top of the window.
  8. Enable Advanced commands and press the “+” button just above the check to create a new custom command.
  9. Call it Siri if you want your trigger to be “Hey Siri” just like on iPhone, iPad or Apple Watch.
  10. Perform the keyboard shortcut that matches Siri’s Keyboard Shortcut in step 2. This could be the Fn (Function) Space combo or a custom combination. I chose Option + S.
  11. Click Done and test it out.

Good luck triggering all your other devices with the command! One suggestion would be to change the trigger word – “Hey”- to something else.

Wednesday, 15 February 2017