And so we end another chapter in the development of Telescope. Telescope version 2.8 arrived 4 days ago, and with it, a couple of interesting features arrived.
Probably the most important one is the setting up of Telescope's private Docker registry, to manage Telescope's Docker images. I am still not sure the consequences of this, except being able to upload and versioning our Docker images on the "cloud".
As for my end, there were 4 specific PRs that I worked on.
Redis corruption
One of the PRs that I worked on was related to fixing a Redis file corruption issue during a power outage.
My first stab at the issue was trying to run a command that would repair the corrupted file before starting out redis. Of course, there were various degrees of success with this approach.
So, what exact file was being corrupted? You see, redis, although being an in-memory database, introduces a few persistence mechanisms to prevent loss of data during a power outage. One of these mechanisms is writing the data being stored to an AOF (append-only file) to disk.
The main problem was properly replicating this issue. When a power outage occurs, redis would end the AOF file abruptly. Because of this malformed file, if you were to startup redis again, then redis would report the file as corrupted and terminate immediately. One way to somewhat replicate this issue is by truncating the file to a certain length.
The problem with my first approach is that, depending on how much you truncate, the program that would fix the AOF (redis-check-aof) would fail on certain situations. For example, if you removed the first couple of kilobytes, redis-check-aof would fix the file with no issue. However, remove over 8 megabytes, and now redis-check-aof cannot fix anything. This may be to how the format of the AOF is, and certain limitations of the redis-check-aof. If the program finds that the file was truncated at a offset that it does not know how to resolve, then it would just fail to fix it.
So, after thinking about it for a while and considering other factors, I went with another approach: use the rdb file that redis generates as a backup. The reason for the "AOF" corruption was because of the constant writing to the file. The operating system can buffer all of these writes to efficiently write everything at once (instead of writing a file for every small string written to disk). However, if the OS hasn't flushed the buffer and there is no energy at all on an unexpected situation, even there the OS cannot do much.
This rdb file is very similar to the AOF one, but it is written when a configured interval of time has passed (for example, every 30 minutes). There is less chance of corruption, for the simple fact that the file is written less often. Of course, this means that the redis database will lose the data that was written in the last 30 minutes, which is not a big deal because our main persistent store is PostgreSQL, and redis is more used for caching common responses.
Documentation
The next PR is related to documentation. It is a very small PR, where I document the API of the feed-discovery service. Not much to point out, except that the style I chose for writing the documentation is very similar to the GitHub API.
ESLint configuration
Another small PR was configuring the dependency-discovery to prepare the linting process for the service. There's nothing much to describe here.
NPM package information in dependency-discovery
Finally, we have the dependency-discovery related PR. Right now, we added a way to get the NPM package information by providing the package name. A small highlight is that the service caches everything, since the NPM information does not change that much. So, for the next time you request the same project, the response should be quite fast! This is the first step to actually provide the GitHub issue information, which hopefully will be brought to light on release 2.9.
Top comments (0)