Aggregate Multiple API Requests with Promise.all()

Hi,

I promise you'll get a summary of this post at the end.

Asynchronous operations are at the backbone of implementing interactivity in modern JavaScript applications. These are used when making API calls, network requests, or even via a basic delay function.

Asynchronous operations utilize promises, callback functions, or async/await. Commonly, these operations are singular and do not require aggregation of multiple async operations into one.

Recently, I started working to build an aggregation service that utilizes multiple 3rd party APIs and aggregates the resulting data. In this post, we'll learn how we make concurrent async requests using Promise.all() in JavaScript. Also, we'll learn how to limit these requests to run in certain groups/portions at a time.

Using Promise.all()

An async function to fetch data from an API typically looks like:

async function fetchData() {
  const res = await axios.get("./names.json");
  console.log(res.data);
}

Here we utilize Axios, a promise-based HTTP client, to make an HTTP request to retrieve data in a local json file. An alternative to using async/await is to use the .then() method of a promise.

With Promise.all(), we handle multiple similar requests concurrently and return a single aggregated response. Promise.all() takes an iterable (an array) of promises. It returns an array containing each promise resolution in the same order.

If any of the promises in Promise.all() is rejected, the promise aggregation is rejected. Here's an example below:

const fetchNames = async () => {
    try {
      const res = await Promise.all([
        axios.get("./names.json"),
        axios.get("./names-mid.json"),
        axios.get("./names-old.json")
      ]);
      const data = res.map((res) => res.data);
      console.log(data.flat());
    } catch {
      throw Error("Promise failed");
    }
  };

This code sample is more elaborate and in a try/catch block to catch any failure in the promise resolution.

Promise.all() doesn't resolve the promises and only aggregates the promises into an array with a single resolution. I'll cut the crap; this means you need to use Promise.all() with an await or .then() to resolve it. 😁

.flat() is a useful array method that flattens the array. Previously, this would be done with a forloop or reduce function.

An alternative with the fetch API looks like this:

const fetchNames = async () => {
      try {
        const res = await Promise.all([
          fetch("./names.json"),
          fetch("./names-mid.json"),
          fetch("./names-old.json")
        ]);
        const data = await Promise.all(res.map(r => r.json()))
        console.log(data.flat());
      } catch {
        throw Error("Promise failed");
      }
};

After using fetch(), .json() is required to parse the response and it also returns a promise! Multiple promises, this is becoming a telenovela!

Another promise.all is required to aggregate the response.

To better understand the effect of Promise.all, we'll create a timing function that resolves a promise after a certain period.

Observing with timing functions

We'll create three promises with:

  const promise1 = new Promise((resolve, reject) => {
    setTimeout(() => {
      const newValue = Math.floor(Math.random() * 20);
      resolve(newValue);
    }, 5000);
  });

  const promise2 = new Promise((resolve, reject) => {
    setTimeout(() => {
      const newValue = Math.floor(Math.random() * 20);
      resolve(newValue);
    }, 8000);
  });

  const promise3 = new Promise((resolve, reject) => {
    setTimeout(() => {
      const newValue = Math.floor(Math.random() * 20);
      resolve(newValue);
    }, 2000);
  });

Each promise resolves at different times of five, eight, and two seconds respectively.

Calling each function separately in an async/await will return the resolved value of each after the set period. Aggregating the result will require further JavaScript operation to build an array.

Calling all in Promise.all() will resolve them all at the same time. In this case, in the time the function requiring the most time executes - 8 seconds.

Using Promise.all(), we have:

const fetchAsyncData = async () => {
    const res = await Promise.all([promise1, promise2, promise3]);
    console.log(res);
  };

It's an efficient and cleaner code for me. 😃

Limiting concurrency

Still, on efficiency, you may want to make large numbers of concurrent requests. Rather than make them all at once, it would be efficient to chunk them.

A useful npm package I found to do this is p-limit.

You can add it to your project using npm or yarn with:

npm install p-limit

yarn add p-limit

Create a limit and specify concurrency count with:

import pLimit from 'p-limit'

const limit = pLimit(2)

Use this limit in the promise with:

const res = await Promise.all([
      limit(() => promise1),
      limit(() => promise2),
      limit(() => promise3)
]);

This block runs two promises at a time.

Here's a CodeSandbox link with all the code blocks running in a React app and logging data to the console.

Summary

Just like promises in JavaScript, you knew this summary was coming. I told you at the beginning. This is just like promises in JavaScript. In this post, we saw how to make aggregated promise resolutions using Promise.all() and limit the concurrency where necessary using p-limit.

Other promise methods to check out include:

• Promise.allSettled()
• Promise.any()
• Promise.race()

Here's to becoming better. 🥂

William