DEV Community

Cover image for #Increasing the Loading Speed of a CLI Web Scraper to Improve the User Experience...
Pamela Torres-Rocca
Pamela Torres-Rocca

Posted on • Edited on

#Increasing the Loading Speed of a CLI Web Scraper to Improve the User Experience...

Slow application load times can cause frustration on behalf of the user. This can lead to users abandoning the use of the application.

In the terminal, you can use the Linux time command to retrieve the time it takes to run a command. The command I used was time ./bin/Urgentcare. This command starts the application and loads a menu for the user.

real    0m12.122s
user    0m4.472s
sys 0m2.130s

Enter fullscreen mode Exit fullscreen mode

The real time listed is the total start to end time of the call. The above time is the time for the scraper application to load the initial menu after the command "time ./bin/Urgentcare". The menu is displayed to the user after the application is started. This requires that the website is scraped and office objects are created with the data.

According to google think, the user bounce rate greatly increases once the load time goes from 1 to 3 seconds. The initial 12 second load time for the menu far exceeds that expectation. This could be due to the website itself taking a long time to load and/or the code is taking a long time to run.

To help resolve this issue you can provide feedback to the user to let them know that the application is successfully running that the data they are looking for is loading. I added a simple method in the CLI component called loading_message that prints a message to the screen that the data is loading. This way the user knows that the application is working and that their input was valid.

Welcome to the Urgent Care CLI




Retrieving data....
Loading......
Enter fullscreen mode Exit fullscreen mode

In order to reduce how long it took the main menu to load, I changed one of the iterators from .each_with_index to .each. The office_url method that used it did not really require the addition of an index as a different method call was already adding it. This helped to reduce the load time.

However, the biggest factor in increasing the speed of the application was refactoring the code so that I was not waiting for a call to each individual office page to retrieve the current appointment times before loading the initial user menu. Instead, I now only call the office page for the appointment time upon user request from the initial menu. This produced a new time of

real    0m4.145s
user    0m1.451s
sys 0m0.861s
Enter fullscreen mode Exit fullscreen mode

to run the time ./bin/Urgentcare command. This is now one third the time it took to load the menu prior to the refactor.

Top comments (0)