Logfish - a Twilio Hackathon Project! (SUBMISSION)

What I built

Category Submission: Exciting X-Factors

While working as a React developer for Bluehost, I found myself continuously running front end unit tests that could have quite a lot of output. For example, there was output regarding if a test passed or failed, or warnings about older React code that developers long before my time wrote.

This quickly became a problem when I began writing my own unit tests that wouldn't work and I needed to figure out why. The easiest solution was to start pumping out console.log statements, but Jest clutters up the console with its own output when a test fails. I would just want to see the value of a variable at a certain point in the test, and every time I would have to do a cmd+F in iTerm to find what I was looking for.

Wouldn't it be easier, I thought, to be able to redirect only the output I wanted to a single location? I could write it to a file, I suppose, but that would be a bit of a hassle to set up any time I wanted to do this. Plus I would end up with annoying files whose output I really only needed while I was debugging the test / code I was testing.

And that's how the idea for Logfish was born! Logfish is a simple solution to real time logging in your browser. The first part of the project is a TS Express backend which uses socket.io to manage client connections. When a new client connects, the server gives it a temporary API key. It also exposes a POST endpoint. When someone POSTS to the /log endpoint with the given API key as a header, it will send the POSTed data to the client with the given API key. I created a React client that will connect to this server and a logfish npm package that will abstract away the details of POSTing to the server's endpoint.

Now I have a solution to my problem! I just need to open the React client and then use the npm package to log what I need to see. Then I will see everything I log in the browser! I also added a feature to send logging output to your phone, using Twilio's API.

Link to Code

Logfish server: https://github.com/bandrewfisher/logfish_server

Logfish client: https://github.com/bandrewfisher/logfish_client

Logfish npm package: https://github.com/bandrewfisher/logfish

How I built it

Setting up the client and server

I started out by creating a Typescript Express application using this template from Github (created by yours truly 😁). Then I set up socket.io so that whenever a new client connects to the server, the server sends the client a temporary API key. The server stores the given API key and the associated socket ID in a little data structure I made called ApiKeyDictionary. It's just an abstraction around a JS object.

import express from 'express';
import http from 'http';
import socket from 'socket.io';
import { v4 } from 'uuid';
import ApiKeyDictionary from './ApiKeyDictionary';

export const app = express();
const server = http.createServer(app);
export const io = socket(server);

io.on('connection', (sock) => {
    const key = v4();
    ApiKeyDictionary.addConnection(key, sock);
    sock.emit('CONNECT', key);
});

server.listen(8081);

I then created a TS + React + Tailwind CSS project using another handy template. In my App.tsx, I connect to the server and store the API key:

import React, { useState, useEffect } from 'react';
import io from 'socket.io-client';

const socket = io.connect('http://localhost:8081')

const App = () => {
  const [apiKey, setApiKey] = useState<string>('');

  useEffect(() => {
    socket.on('CONNECT', (key: string) => {
      setApiKey(key);
    });
  }, []);

  return (
    <div className="w-full flex p-24">
      <h2 className="text-2xl font-semibold text-gray-700 mb-3">
        Your temporary API key (click to copy):
      </h2>
      <div className="rounded bg-green-200 p-3 border border-green-400 mb-8 cursor-pointer">
        {apiKey}
      </div>
    </div>
  )
}

export default App;

When the server emits a 'CONNECT' event, the client stores the given key in the apiKey variable. You can see that this is the green box in the picture at the beginning of the article where the API key was displayed.

Implementing logging

Ok, now we've got a connection between the client and server. But there's still no way to log. I created a POST endpoint on the server that would look for the API key in the request headers and a JSON request body with a data attribute that would contain what we want to log.

const router = express.Router();
const getApiKey = (req: Request): string => (req.headers['logfish-key'] as string);
const getLogData = ({ body: { data } }: Request): any => data;
const getSocket = (apiKey: string): Socket => ApiKeyDictionary.getSocket(apiKey);

router.route('/log').post(verifyRequest, async (req: Request, res: Response) => {
  const apiKey = getApiKey(req);
  const data = getLogData(req);
  const sock = getSocket(apiKey);

  let message;
  if (typeof data === 'string') {
    message = data;
  } else {
    message = JSON.stringify(data);
  }

  try {
    sock.emit('DATA', message);
    res.status(201).end();
  } catch (err) {
    res.status(500).json({ message: 'Socket error' });
  }
});

app.use(router);

Since I associate sockets with their API key in ApiKeyDictionary, I just used that to retrieve the socket using the provided API key in the header. Then I can use that socket to emit the received data. Now I just need to add a listener for the "DATA" event in the React client. I also created a logs variable to hold everything the server sent and a readonly textarea to display those logs.

Here's my updated App.tsx:

import React, { useState, useEffect } from 'react';
import io from 'socket.io-client';

const socket = io.connect('http://localhost:8081')

const App = () => {
  const [apiKey, setApiKey] = useState<string>('');
  const [logs, setLogs] = useState<string[]>([]);

  useEffect(() => {
    socket.on('CONNECT', (key: string) => {
      setApiKey(key);
    });
    socket.on('DATA', (data: any) => {
      setLogs((l) => [data, ...l]);
    });
  }, []);

  return (
    <div className="w-full flex p-24">
      <h2 className="text-2xl font-semibold text-gray-700 mb-3">
        Your temporary API key (click to copy):
      </h2>
      <div className="rounded bg-green-200 p-3 border border-green-400 mb-8 cursor-pointer">
        {apiKey}
      </div>
      <textarea
        className="w-full bg-gray-200 rounded h-full mt-6"
        readOnly
        value={logs.join('\n')}
      />
    </div>
  )
}

export default App;

I tested out the POST route with Postman, and the data I posted showed up! I was as excited as a schoolgirl.

Mobile logging

Now there's no way I could submit this project for the Twilio Hackathon without using Twilio. So I figured - if I can log to my browser, why not send logs to my phone too? When I refresh my browser, all those logs will be lost since they are just stored in a React variable. But if I get logs on my phone, then I can keep them for as long as I want.

I created a button that would open a modal allowing the user to enter her phone number if she wanted to enable mobile logging. After entering her phone number, the server would create a secure 6 digit verification code, store the phone number and code in an object, and then send a verfication text to the user's phone (originally I was storing the codes in a Dynamo DB table, but decided that would make it too complicated to set up locally). I decided a verification code was necessary so that users couldn't enter the phone number of someone they dislike and then spam them with a bunch of annoying log messages!

The user would then be prompted to enter the verification code. When the server receives that, it will check to see if the code is valid. If the user sent the correct verification code, the user's phone number will be stored with the socket in my ApiKeyDictionary object. Then whenever the POST endpoint is hit, the log message will be texted to the given phone number!

I won't go into the code details for the modal prompting the user for her phone since it's pretty verbose. You can look at the code on Github if you're interested. But here's what the modal looks like:

The client sends an 'ADD-PHONE' event to the server. The server responds by generating the code and storing the phone and code in a JS object.

sock.on('ADD-PHONE', async (clientNumber: string) => {
  try {
    await addPhone(clientNumber);
    sock.emit('ADD-PHONE-SUCCESS');
  } catch (err) {
      sock.emit('ADD-PHONE-ERROR', err.message);
  }
});

addPhone just adds the phone to the object. Again, check out the code on Github if you're interested in the details.

When the client receives the 'ADD-PHONE-SUCCESS' event, it prompts the user for the verification code:

When the user enters that, the server checks the code and registers the phone number in ApiKeyDictionary if the code is correct. Then on subsequent POST requests, a text with the log data is sent to the client. All the server has to do is call a sendMessage function:

import twilio from 'twilio';

export const createTwilioClient = () => {
  const accountSid = process.env.TWILIO_ACCOUNT_SID;
  const authToken = process.env.TWILIO_AUTH_TOKEN;
  const twilioClient = twilio(accountSid, authToken);
  return twilioClient;
};

export const sendMessage = async (clientNumber: string, messageBody: string) => {
  const twilioClient = createTwilioClient();
  const twilioPhoneNum = process.env.TWILIO_PHONE_NUM;
  const twilioResult = await twilioClient.messages.create({
    to: clientNumber,
    from: twilioPhoneNum,
    body: messageBody,
  });
  return twilioResult;
};

Simplifying logging

Everything works fine so far, but I wanted logging to be as simple as typing console.log in the program I wanted to debug. I didn't want to have to POST to my server with the API key header and construct a JSON object with the data I wanted to log. To solve this, I decided to create a logfish npm package that would abstract away all those details. I would just need to pass in the API key and URL of my server and I would be good to go.

The package is just a wrapper around an axios object performing a POST request. The source code is very lean as you can imagine. And here's the documentation for using the package: https://www.npmjs.com/package/logfish.

Now if I'm running a program I need to debug, I download logfish and do this:

import Logfish from "logfish";

const lf = new Logfish("API_KEY", "http://localhost:8081");
....
const myVar = 123;
lf.info(`Value of my variable: ${myVar}`);
lf.warn('Something could go wrong here!');

Then I would see those logged to the client, or sent to my phone if I had configured mobile logging.

Obstacles

One of the first obstacles I ran into was writing unit tests for my client and server. I hear the TDD mantra all the time - write your tests first! - but I still didn't really follow it which I believe was the reason I had difficulty writing unit tests.

The hardest part about unit testing this application was mocking socket.io. My unit tests would be brittle if I actually needed my client and server running at the same time so that the socket methods would work properly, so I knew I needed to mock the sockets and test them that way.

I tried following the Jest documentation for mocking Node modules, but for some reason the real socket.io package was still being used and I had no idea why. So I ended up just writing my own mock for the socket object. It was a bit of a pain getting it working with Typescript but this is what I came up with:

interface MockSocket {
  emit: jest.Mock;
  on: jest.Mock;
}
const mockSocket: MockSocket = {
  emit: jest.fn(),
  on: jest.fn(),
};

jest.mock('socket.io', () => jest.fn().mockReturnValue({
  on: jest.fn((_event: string, cb: (sock: MockSocket) => void) => {
    cb(mockSocket);
  }),
}));

Socket objects have a on function that is given an event and a callback. This mock just calls the callback immediately which isn't ideal, but this is just a little project for a hackathon so it was ok for my purposes. I ended up getting lazy with my testing as I was finishing up my project, but it was at least good unit testing practice. In the future, I'll write my tests first and only write as much production code as it takes to make that test pass. I see the value in TDD and want to master it.

My other big obstacle was trying to get this project working in the cloud. Originally, I deployed my server to Elastic Beanstalk where it was working beautifully. I had my client pointed to it as well as my npm package. No problems there....yet.

I then deployed my React client to Netlify and bought the logfish.dev domain from Google domains. Again, things worked great. The original plan was for users to just have to download the npm package and then they could log to logfish.dev.

Since Netlify provides free SSL certificates, I got one for logfish.dev, and that's where things started going wrong. Since Elastic Beanstalk sites are HTTP, my frontend refused to use it since it was HTTPS. Ok, I thought, I'll just need to setup SSL for my Elastic Beanstalk endpoint. Turns out, things weren't so simple.

This was the only article that seemed to make sense - I wanted to create a server.logfish.dev subdomain and have it point to my server. My SSL certficate encrypts *.logfish.dev domains, so I should have been covered with HTTPS. I followed the instructions in the article, setting up a hosted zone in the AWS Route 53 and an NS record in Google domains. However, no luck. I still don't really understand how domain records work, but it would be really cool if I could get this working in the cloud. If only I could have set up HTTPS with Elastic Beanstalk, then no one would have to run the client and server locally! (Please message me if you know how to do this!)

Conclusion

Overall, this was an incredibly fun project that I could see myself using to make debugging my code easier. I learned a lot about Twilio and socket.io and am excited to use my new knowledge of these technologies for future projects. Happy coding everyone!