Latest Posts

Functional Promises in JavaScript Crash Course

Functional Promises in JavaScript Crash Course

See the companion project to this article: escape-from-callback-mountain Build Status

Refactoring NodeJS/JavaScript - a 2017 Guide

I am a big fan of Functional Programming and Modular JavaScript. This project’s goal is to demonstrate better coding habits by guiding you through a refactor of some real-world NodeJS JavaScript code.

The overall technique I demonstrate is what I call the ‘Functional River’ pattern. Where your input/parameter/data is the water, and the code forms the riverbed. More or less.

To the Haskell pros out there, before you flame me for not defining ‘monad’, this is meant to be a more welcoming place. So forgive me if I skip the overblown theory & jargon.

For this project I happen to use Bluebird Promises. Apologies to Promise Resistance Leader Brian Leroux.

‘Functional River’ Goals/Benefits:

Note: Relies on ideas from Lisp to SmallTalk - adapted to a JavaScript world.

Have feedback, fixes or questions? Please create issues or PRs. Or just complain at me on https://twitter.com/justsml

If you feel this subject has been thoroughly explored, please see my post Beating a dead horse?


Sample Task:

Authentication Method

Here’s a rough visualization of our function: image

See both Before and After examples below.

Before

Node-style Callbacks w/ Nesting

Note: This is intentionally reasonable callback code. Even if nested. Not trying a straw-man attack.

callback-mountain-before

After

‘Functional River’ Pattern

callback-mountain-after

Key Steps

  1. Step 1: Break Up The Big Functions - see the PR #2: Flatten Functions
  2. Step 2: DRYer Code - see the PR #3: DRYer Code
  3. Step 3: Cleanup Code - see the PR #5: Post Cleanup

Pros & Cons

Pros

  • Less ad hoc code results in:
    • More uniform code between different teams & developers,
    • Performance tooling & refactoring is an appreciably better experience,
    • More certainty about code correctness,
    • Higher code reuse.
  • 100% Unit Testability
    • Unit tests uniquely prove you found, understand, AND resolved a given bug,
    • Faster bug resolution process,
  • Flatter code hierarchy == less filler to remember

Cons

  • Performance. I’ve run some micro-benchmarks - it’s not awesome. However, 3 important things:
    1. It’s not meaningfully slower in real world applications.
    2. If it is necessary, performance analysis & tuning is a much improved experience. Smaller functions make it easier to see where slow code lurks - especially if you profile unit tests.
    3. As more people adopt these patterns, things will improve. V8/Chrome has been impressively fast at optimizing for emerging patterns.
  • Debugging can be more difficult. Though I have updated my dev tricks to debug this style of code, even without the confort of Bluebird’s error handling. I’ll add more sample scripts for this later.
  • Something new to learn. Deal with it, you’re a developer.
  • If you have an existing project with lots of code, the unfortunate reality is: Refactors Suck.
  • EventEmitter- & Stream-based code is not improved much, if at all, using this technique. Look into RxJS
    • Ongoing experiments include simple closures, extend Promise with EventEmitter, or using Bluebird’s .bind to inject variable state into the Promise chain. (I know, “ugh side-effects, gross.” PRs welcome.)

Any consensus around best practices is still far off, Functional JS patterns still vary.

Concerns

Some really smart people out there have reservations about over-modularization.

image

While true of other coding patterns, an overly-done flat & modular JS Project can get more disorganized over time. Project and code discipline is just as important as it’s always been. Also, we’re still developing consensus around Functional JS patterns.

Another solution I’ve found is to add a Code Style Guide preferably with naming conventions - see my thoughts on that subject. This becomes much more important as team size grows.

When done right, one of Functional River’s greatest strengths is the ability to relocate & rearrange modules with low risk. If this still feels risky, your modules are still too entangled.


Ultimately my goal is to better understand & advance Modular + Functional JS patterns. Hopefully I can sway some of the skeptics along the way :crossed_fingers:


Please read my more detailed article demonstrating 4 Functional JavaScript Techniques (with Examples)

Credits & Inspiration

Linux Server Benchmarking Scripts

Goal is a simple set of bash scripts which allow rapid assessment of any given hardware.

Currently only CPU & HDD tests are wired up. Other tests are a work-in-progress, added as needed.

CREATE BENCHMARK SHORTCUT ALIASES

# COPY + PASTE THE FOLLOWING TO CREATE FOLDER & MAIN SCRIPT(S)
# Create folder for results & scripts
export BENCH_DIR=$HOME/benchmarks
mkdir -p $BENCH_DIR/results

cat << 'EOT' >> $HOME/benchmarks/bench-library.sh
#!/bin/bash
set -e

# Install some deps
if [ "$(which sysbench)" == "" -o "$(which inxi)" == "" -o "$(which tcpdump)" == "" ]; then
  apt-get update && apt-get install -y sysbench inxi htop iotop tcpdump hddtemp
fi
# Variables
export DATE_TAG=`date +%F` #YYYY-MM-DD
export CPU_CORES="$([ -e /proc/cpuinfo ] && grep -sc ^processor /proc/cpuinfo || sysctl -n hw.ncpu)"
export BENCH_DIR=$HOME/benchmarks/

mkdir -p $BENCH_DIR

function benchCpu() {
  thread_limit=${1:$CPU_CORES}
  prime_limit=${2:-20000}

  if [ $CPU_CORES -lt `expr 1 + $thread_limit` ]; then
    printf "\n\n${yellow}ALERT: Skipping tests limited by \"${thread_limit} thread test\"\n${cyan}Not enough CPU Cores ($CPU_CORES)  ${reset}\n\n"
  else
    printf "\n\n${yellow}ALERT: Skipping tests limited by \"${thread_limit} thread test\"\n${reset}"

    sysbench --test=cpu \
      --cpu-max-prime=${prime_limit} \
      --num-threads=${CPU_CORES} \
      run | tee -a $BENCH_DIR/results/cpu-test.log
  fi
}

# benchSingleDisk seqrd 120G 8K 300
function benchSingleDisk () {
  sysbench --test=fileio --init-rng=on  --file-test-mode=${1:-seqrd} --file-block-size=${3:-64K} \
    --num-threads=${CPU_CORES} --max-time=${4:-180} --file-total-size=${2:-60G} \
    --max-requests=0 run | tee -a $BENCH_DIR/results/sysbench-fileio.log
}


# benchDisk - tests random read & write, and sequential r, and sequential write, before final cleanup.
function benchDisk() {
  #   Generates test files - up to 75% of your free space - in local dir, then runs the 3 tests (up to 20 minutes each)
  freeSpace=`df -k . | tail -1 | awk '{print $4}'`
  freeSpace="${freeSpace//G|T/}"
  testSize=$(awk "BEGIN {print ($freeSpace / 1024 / 1024) * 0.75; exit}")
  testSize=${testSize}G
  printf "####>>> \nWriting $testSize test data to ${PWD}...\n"

  benchSingleDisk seqrd ${testSize} 8K 300
  benchSingleDisk seqwr ${testSize} 8K 300
  benchSingleDisk seqrw ${testSize} 8K 300
  benchSingleDisk rndrd ${testSize} 8K 300
  benchSingleDisk rndwr ${testSize} 8K 300
  benchSingleDisk rndrw ${testSize} 8K 300

  benchSingleDisk seqrd ${testSize} 64K 300
  benchSingleDisk seqwr ${testSize} 64K 300
  benchSingleDisk seqrw ${testSize} 64K 300
  benchSingleDisk rndrd ${testSize} 64K 300
  benchSingleDisk rndwr ${testSize} 64K 300
  benchSingleDisk rndrw ${testSize} 64K 300

  printf "\n\n####>>> \nCOMPLETED TESTS! Great Success!!! \n\n\n"
}

EOT

chmod +x $BENCH_DIR/*.sh
source $HOME/benchmarks/bench-library.sh

Step 2: CREATE RUNNER SCRIPT

cat << 'EOT' >> tee $HOME/benchmarks/run-bench.sh
#!/bin/bash
set -e

source ./bench-library.sh

# Benchmark HDD Speed (in Current Directory)
###########
benchDisk

# Benchmark CPU - trying different thread counts (and work sizes)
# It'll automatically skip test if we don't have enough cores (to have an impact)
# NB: results comparable between different hardware - up to their same CPU CORE #.
###########
benchCpu 1
benchCpu 4
benchCpu 8  50000
benchCpu 12 100000
benchCpu 16 100000
benchCpu 32 250000
benchCpu 48 500000
benchCpu 64 2000000

EOT

chmod +x $BENCH_DIR/*.sh





Usage Examples:

Make sure to source ~/benchmarks/bench-library.sh before running the following commands manually.

benchCpu 8   250000
benchCpu 16  250000
benchDisk
Cloud Tuning: NVMe SSDs are fast path to Radical Speed Boost

Boost Cloud Performance up to 70%

General notes & sections for select hosting providers below.

Amazon Web Services / EC2 / EBS / S3

TLDR; AWS features restrictive hardware & pricing tiers. The i3 hardware series is the only type I would consider from a Price/performance (and absolute performance) perspective.



image

Note the i3.*xlarge is the only hardware to feature competitively priced NVMe storage (ultra-fast +1GB/s speeds). The major limiting factor I found was the real network speed. Servers with advertised speeds “up to 10/Gb/s” struggled to get close to 1/Gb/s (60-80MB/s).

Network tests used up to 9 additional instances in the same availability zone. Any erroneous data points I replaced with a 0. Additional tests are needed as only 1-2 samples were collected.

Head to Head

Credits

Guide 2017: Write Better JavaScript, Faster

A 4-part Crash Course in Composable JavaScript

Preface

Disclaimer: includes Dan Levy’s heretical musings thought-leadering.

If you want more complete examples, see my project escape-from-callback-mountain. Build Status


Part 1

Code Nirvana through 4 Key Rules

One of the most important ideas in software development is: keep it simple.

The funny thing is, every developer believes: they write simple & beautiful code, the real problem is those other developers.

Don’t worry, I’m not wading into the tarpit defining “Simple Code.”

Instead I’ll list 4 general rules, each which have helped me write more readable, testable & adaptable code.

  1. Restrict functions to single-purpose. See Single Responsibility Principle.
  2. Restrict functions to single-argument (or 2 when using NodeJS/Express callback (err, value) style). So, your 3 parameter function (a, b, c) => {} should instead accept ({a, b, c}) => {}. Helps with Dependency inversion, Interface segregation.
  3. Never use prototype if possible. (Confession: I only used once, by necessity, in escape-from-callback-mountain errors module.) Composition is a pure embrace of the Open/closed principle.
  4. Build functions without nesting/variable hoisting. Mitigate with ImmutableJS, else you can pass needed values as arguments. (Factory patterns are ok if state sharing is avoided or centralized. Examples in callback-mountain project.)

Let me emphasize: Single parameter functions are not restricting - simply use an Array or Object for your function argument.

Now, you may be thinking “hold on, is there a real difference between multiple arguments and wrapping everything in a single-arg object/array?” Great critical thinking (and thanks to Jani Hartikainen for bringing this up). The answer is yes, it’s a huge difference.

Essentially, parameter limiting can make your code compatible with many pre-existing patterns & tools: Promises, Array.[map,find,forEach,filter], RxJS, etc. This feels like an advantage to me.

If you are wondering how single-purpose functions ever amount to anything except code sprawl, well, let me introduce you to my friend, Higher Order Components, or HOCs. Which is really a fancy way of saying Function, Controller, Class, etc. The goal is to have your code feel like using LEGO™ building blocks.

Let’s look at some code…


Part 2

Four JavaScript Composition Examples

Let’s wire up a few functions to do some simple math.

Composable JS Example Includes:

  1. Pure ES6
  2. Lodash
  3. Array.reduce (built-ins)
  4. Promises
/*
@author: Dan Levy <Dan@DanLevy.net>
Twittering: @justsml
*/
const test = require('tape')

// Demo of 4 techniques to 'glue' functions together (Higher Order Components)
// we'll do some simple math:  add5(5)==10,  half(10)==5,  square(5)==25.0

// Example/Util Math Methods:
// Pure-ish functions (https://en.wikipedia.org/wiki/Pure_function)
const add5 = n => {
  n = parseFloat(n) + 5  
  return n // required
}
const half = n => {
  n = (parseFloat(n) * 0.5).toFixed(2)
  return n
}
const square = n => {
  n = parseFloat(n * n).toFixed(2)
  return n
}

test('Method #1: Pure JS/ES6: math functions', t => {
  // `compose([functions])` accepts a list of functions, to be executed in order, starting with the value passed in
  const compose = (...fns) => x => fns.reduce((v, f) => f(v), x)

  const add5HalfSquare = compose(add5, half, square)
  t.equals(add5HalfSquare(5), '25.00', 'I can caz maths?')
  t.end()
})

test('Method #2: Lodash: math functions', t => {
  const {flow} = require('lodash');
  const add5HalfSquare = flow(add5, half, square);
  t.equals(add5HalfSquare(5), '25.00', 'I can caz maths?');
  t.end();
})

test('Method #3: Promises Chain: math functions', t => {
  const add5HalfSquare = n => Promise
    .resolve(n)
    .then(add5)
    .then(half)
    .then(square);

  add5HalfSquare(5).then(answer => {
    t.equals(answer, '25.00', 'I can promise maths?');
    t.end();
  });
})

test('Method #4: Array.reduce: math functions', t => {
  const add5HalfSquare = n => [add5, half, square]
    .reduce((val, fn) => fn && fn(val), n);
  t.equals(add5HalfSquare(5), '25.00', 'I can reduce maths?');
  t.end();
})

With this technique, you no longer have to fight the Framework Wars. I don’t have time for it.

Hip to Lodash?

Great, use _.flow, or .chain().

Using Promises?

Cool, Promises are just another pluggable pattern.

There are so many choices for gluing your functions together! Just make sure to insulate yourself by sticking to my 4 rules (in Part 1).

Let’s look at a more real-world example…

Part 3

Example Login Pattern

Let’s say we’re given requirements:

  1. User clicks [login] button.
  2. Modal opens, prompts for fields user and pass.
    • User submits form.
    • User clicks ‘forgot password’.
  3. Upon success, load app+msg data for logged-in user.
    • Cache ajax data for offline use.
    • Update user status to ‘online’
  4. Upon failure, plunge into fire pit.
  5. Show appropriate UI messaging

Using bluebird’s unique take on “Promises,” I’ll show the code in almost as many lines as the (terse) requirements above.

login = () => openLoginModal()
  .then({user, pass}) => ajaxLogin({user, pass}))
  .then(user => {
      setUserStatus('online') // < non blocking, as result is irrelevant.
      return [getContacts(user), getRooms(user), getMessages(user)]
  })
  .then(([contacts, rooms, messages]) => {
      alertOnNew({contacts, messages})
      return this.cache({contacts, rooms, messages})
  })
  .catch(UserCancel, hideModal)
  .catch(ForgotPassword, () => showUserMessage({message: 'think harder'}))
  .catch(LoginError, compose(hideModal, initFirePit, destroyApp))
  .catch(err => showUserMessage({message: 'Something truly unexpected happened, congratulations.'}))

That looks like some solid Functional JavaScript, however it still has some subtle weaknesses.

Namely - coupled code - see those anonymous arrow functions in .then(fn)? Are 2 distinct things happening here? This needs more attention, the call to setUserStatus doesn’t seem to relate to the returned new array structure. Let’s see if single responsibility can help us fix this up a little.


Let’s look at a refactored example…

Part 4

Summary with Further Optimized Code

Here’s how I applied rules #1 and #2 to refactor:

  • Avoid nested logic - it’s a sign you are mixing 2+ different things. It’s also a sign of untestable/high dimensionality.
  • More than 2 lines per function? Stop and think, or ask yourself/nearest dev: “Is it really related? Why is this together? Is there a better order to chain methods?”

Ok, I’m being a bit of a troll about an ‘absolute 2 line limit.’

Here is what I’m getting at: separate your code into distinct logical functions.

chatApp.getUserData = user => {
    return [getContacts(user), getRooms(user), getMessages(user)]
}
chatApp.login = () => openLoginModal()
    .then({user, pass}) => ajaxLogin({user, pass}))
    .tap(() => setUserStatus('online'))
    .then(chatApp.getUserData)
    .tap(([contacts, rooms, messages]) => alertOnNew({contacts, messages}))
    .then(this.cache)
    .catch(UserCancel, hideModal)
    .catch(ForgotPassword, () => showUserMessage({message: 'think harder'}))
    .catch(LoginError, compose(hideModal, initFirePit, destroyApp))
    .catch(err => showUserMessage({message: 'Something truly unexpected happened, congratulations.'}))

Here’s why it’s better: it’s flatter & therefore more (unit) testable.

chatApp.getUserData is testable because it’s not hidden inside the login() and tied to the status update. Previously, the nested .then’s which called ad-hoc background functions are now explicitly not affecting the control flow of the chain (using tap). And as an added bonus, the decision tree of a function is easier to understand when using custom errors and filtering by Error type with Bluebirds’ .catch(<type>, <error>) interface. See my example pattern of a ‘Finite State Machine’ using the Error handling in Bluebird Promises

Ultimately my goal is code which reads like a story.

Alternative Theories

While I differ in approach & reasoning, I highly recommend reading Best practices for JavaScript function parameters on codeutopia.net.


If I forgot to give credit, please send a PR, or let me know twitter.com/justsml

Naming things real good

Let’s tackle something deceptively simple & subtle: Naming

I want to avoid the super-fancy-tech-lingo for this article; and hopefully I can illustrate the issue in a more useful fashion.

While covered in exhausting detail before, the subject matter often gets too technical for the novice programmer to draw any practical understanding. You probably don’t need to read this if the following makes sense Boyce Codd Normal Forms

The Problem - by Example

Have you ever designed a data model (in code, Sql, or excel worksheets)? Does the following look familiar?

*** anti-pattern - don't copy-paste ***
* User
  - id
  - avatarUrl
  - email
  - passwordHash

* Agent
  - id
  - primaryPhoto
  - agentName
  - agentEmail
  - agentPhoneMain
  - agentEmailPrimary
  - agentPhonePrimary
  - agentAddressLine1
  - agentCompanyName
  - agentCompanyAddress
  - *userEmail* - 'Pointer' to User table ^^^

If this is familiar to you, I’ll bet you:

  1. Feel any change to your app will necessitate hours of arduous debugging.
  2. Fear ANY Changing Requirements

schema refactor

The Cost of Bad (Naming) Habits

Why is naming a field User.agentEmailPrimary the worst?

For starters, you are not creating an entirely new object unto the universe. I hope you meant something like agent.emails[0] or user.displayEmail

Over-specificity & entangled code has some consequences:

  1. Strong hint there’s some bad schemas, modular design or seperation of logical objects.
  2. When you are commited to highly specific name, it means way less reusable code, with annoyingly recurring bugs like:
    • Mismatched names can occur frequently in templates and code, leading to a frustrating team experience.
    • Awkward 3-part names are a strong red flag that fields may have been arbitrarily added to tables/schemas.
    • Validation rules/logic somehow manage to be used both repetitivly and inconsitently.
    • Fragility piles up with every single new file, as an extremely high attention to detail is required for trivial changes.
  3. agentEmailPrimary could mean a few different things. Avoid ambiguity with shorter names.
    • Watch out for silly excess wording. Primary? Just leads to more questions: Is there a Secondary? Is it for their Primary Next-of-kin?
    • Use a new object agent, so something like agent.emails[0]. Or give more meaning in the context of a user with user.displayEmail

Hang in there, we’re almost to a suggested solution…

A Solution

// Dan's Recommended Schema Consolidation:

User
  - id
  - role: ['agent', 'lead', 'admin']
  - name
  - phone
  - address
  - email
  - passwordHash
  - company
    - name
    - address

I removed the Agent table, as it didn’t contain fields which were uniquely related to Agents.

All changes were made with these general ideas in mind:

  1. Eliminate unessesary tables. If you have a few dozen tables, this step is mandatory.
  2. Try merge related tables. Important if you are coming from a SQL background to No-SQL
  3. Delete redundant data collection (e.g. remove ActivityLogs table if replaced by Google Analytics)
  4. Try keeping all field names to a single word/noun/pro-noun.
  5. There is no such thing as Agent.agentEmail or Agent.agentPhonePrimary. Period.
  6. By using Highly Specific Names, you cast-in-stone a specific level of code-reusability and durability, well, specifically ZERO %.
  7. Don’t think you are doing yourself any favors with crap like this User.profileSummaryEmail (where ‘profile’ could include contact details for a personal ads site) . This is probably a good point to create a new table, say Profiles which includes Profiles.email.

And if you think I’ve just got crazy ideas, hopefully this will help:

fuck this

Recommended reading includes:

  1. Book: Code Complete
  2. http://phlonx.com/resources/nf3/
  3. https://en.wikipedia.org/wiki/Database_normalization

Tag Cloud

Jekyll::Drops::SiteDrop