Pythonizing JavaScript

Python has some amazing utility functions like range, enumerate, zip, etc. built on top of the iterable and iterator protocols. Together with generator functions, these protocols have been available to us in JavaScript in all evergreen browsers and Node.js since circa 2016, and in my opinion are criminally underused. In this post I am going to implement some of these helpers in TypeScript to try and change that.

Iterators, iterables and generator functions

The iterator protocol

The iterator protocol is a standard way of producing sequences of values. For an object to become an iterator, it has to adhere to the iterator protocol by implementing a next method, for example:

const iterator = {
  i: 0,
  next() {
    return {done: false, value: this.i++};
  }
}

We can then repeatedly call the next method to obtain the values:

console.log(iterator.next().value); // → 0
console.log(iterator.next().value); // → 1
console.log(iterator.next().value); // → 2
console.log(iterator.next().value); // → 3
console.log(iterator.next().value); // → 4

The next method should return an object with a value prop containing the actual value and a done prop specifying if the iterator has been exhausted, i.e. no more values can be produced. According to MDN neither of these properties are strictly necessary, and if both are missing, the return value is treated as {done: false, value: undefined}.

The iterable protocol

The iterable protocol allows an object to define its own iteration behavior. To adhere to the iterable protocol, the object has to define a method using the Symbol.iterator key which returns an iterator. Many built-ins like Array, TypedArray, Set and Map implement this protocol and can therefore be iterated over using for...of loops.

For arrays, for example, the values method is specified as the array‘s Symbol.iterator method:

console.log(Array.prototype.values === Array.prototype[Symbol.iterator]); // → true

We can combine the iterator and iterable protocols to create an iterable iterator like so:

const iterable = {
  i: 0,
  [Symbol.iterator]() {
    const iterable = this;
    return {
      next() {
        return {done: false, value: iterable.i++}
      }
    }
  }
}

The names of the two protocols unfortunately very similar and still manage to confuse me to this day.

As you might have guessed our iterator and iterable examples are infinite, which means they can produce values forever. This is a very powerful feature but can also easily become a footgun. So for example if we were to use the iterable in a for...of loop, the loop would go on forever, or used as the argument in Array.from, JS will eventually throw a RangeError because the array will become too large:

// Will loop forever:
for (const value of iterable) {
  console.log(value);
}

// Will throw RangeError
const arr = Array.from(iterable);

The reason iterators and iterables can even be infinite is that they are lazily evaluated, i.e. no values are being produced until they are being used.

Generator functions

While iterators and iterables are an invaluable tool, they are a bit cumbersome to write. As an alternative, generator functions have been introduced.

Generator functions are specified using function* (or function *, the asterisk can be anywhere between the function keyword and the name of the function) and allow us to interrupt the execution of the function, return values using the yield keyword and at a later point pick back up where it was interrupted, all while maintaining its internal state:

function* sequence() {
  let i = 0;
  while (true) {
    yield i++;
  }
}

const seq = sequence();
console.log(seq.next().value); // → 0;
console.log(seq.next().value); // → 1;
console.log(seq.next().value); // → 2;

// Will loop infinitely, starting with 3
for (const value of seq) {
  console.log(value);
}

Python utilites

As mentioned in the introduction, Python has some very useful built-in utilities that build upon the aforementioned protocols. JavaScript recently gained some helper methods for iterators as well, like .drop() and .filter(), but doesn’t (maybe yet?) have some of the more interesting utilities from Python.

Let’s get our hands dirty!

Now that the theory is out of the way, let’s get to implementing some of Python’s functions!

Note: None of these implementations shown here should be used as is in a production environment. They lack error handling and boundary condition checks.

enumerate(iterable [,start])

enumerate in Python returns a sequence of tuples for every item in the input sequence or iterable, containing the count at the first position and the item at the second position:

for index, value in enumerate(['a', 'b', 'c']):
    print(index, value)

# Outputs:
# 0 a
# 1 b
# 2 c

enumerate also accepts an optional start parameter indicating, where the counter should start:

for index, value in enumerate(['a', 'b', 'c'], start=100):
    print(index, value)

# Outputs:
# 100 a
# 101 b
# 102 c

Let’s implement it in TypeScript using a generator function. As a guide we can use the implementation outlined in the python docs

function* enumerate<T>(iterable: Iterable<T>, start = 0) {
  let index = start;

  for (const item of iterable) {
    yield [index++, item] satisfies [number, T];
  }
}

Since strings in JavaScript implement the iterable protocol, we can simply pass a string to our enumerate function and call it like so:

for (const [index, value] of enumerate('Hello, world!', 10)) {
  console.log(index, value);
}

/*
Outputs:
10 H
11 e
12 l
…
20 l
21 d
22 !
*/

repeat(elem [,n])

repeat is part of the built-in itertools library and repeats the given input elem n times or infinitely if n is not specified. And again we can take the implementation from the python docs as a starting point.

function* repeat<T>(elem: T, n?: number) {
  if (!n) {
    while (true) {
      yield elem;
    }
  } else {
    for (let i = 0; i < n; i++) {
      yield elem;
    }
  }
}

cycle(iterable)

cycle is also part of the built-in itertools library and repeats every element in the input iterable indefinitely.

for element in cycle('ABC'):
  print(element);

# Outputs:
# A
# B
# C
# A
# B
# …

Let’s again take the Python sample implementation as a starting point and implement cycle in TypeScript:

function* cycle<T>(iterable: Iterable<T>) {
  const saved = [];

  for (const element of iterable) {
    saved.push(element);
    yield element;
  }

  while (saved.length) {
    for (const element of saved) {
      yield element;
    }
  }
}

Since there is no way to rewind a iterator in either Python or JavaScript, as opposed to e.g. PHP, we have to manually keep track of the already yielded elements. This means that for infinite iterators, our saved array will also grow infinitely (or at least until the JavaScript engine throws).

range([start,] stop [,step])

Probably one of the most used built-in functions in Python is range. It allows us to create a lazy range of numbers. One thing that has always intrigued me about this function is that if you only pass one argument, the argument becomes the stop argument, instead of the start argument, which allows for really concise calls. Let’s implement range in TypeScript:

function range(stop: number): Generator<number>;
function range(start: number, stop: number): Generator<number>;
function range(start: number, stop: number, step: number): Generator<number>;

function* range(...args: number[]) {
    let start = 0, stop, step = 1;

    if (args.length == 1) {
        [stop] = args;
    } else if (args.length == 2) {
        [start, stop] = args;
    } else {
        [start, stop, step] = args;
    }

    for (let i = start; i < stop; i += step) {
      yield i;
    }
}

At the top, we have defined overloads to range, to allow for a one-, two- and three-argument form of the function. Make sure to throw a TypeError or similar in the case there are no arguments, when using this function in production.

Conclusion

This is my first blog post, so I hope you found it interesting and maybe you will use iterators, iterables and generators in your future projects. If you have questions or need clarification, please leave a comment and I’ll be more than happy to provide further information.

It should be noted though, that compared to a raw for loop with a counter, the performance doesn’t come close. This should probably not matter in a lot of situations, but definitely matters in high-performance scenarios. I found out when drawing PCM data to a canvas and dropping frames left and right when using iterators and generators. In hindsight it probably is obvious but wasn’t to me at the time :D

Cheers!