Recently I released a small package for Go.
akshaybharambe14
/
go-jsonc
go-jsonc provides a way to work with commented json by converting it to plain json.
While building this, I came across various problems and best practices. So I just wanted to share my learnings.
Make it compatible with the standard library
This is my first priority when I build something for public. This makes your library widely compatible and reusable.
In this library, I made the decoder compatible with io.Reader. Now, I don't have to care about the source. It can be an HTTP request, a file, etc. Anything that implements io.Reader, is supported. I recommend you to read this excellent article from Mat Ryer
Reduce your API surface
In simple words, avoid using external libraries for some simple tasks. Try to use the standard library as much as you can.
In the context of this library, I intentionally avoided some functionality, like validating resulting JSON, so that the importing library can reuse its existing dependencies. Stick to the original problem instead of providing some fancy functionality.
Unit Tests are must
Writing unit tests is fun and I would say it makes your program more robust and easy to maintain. Whenever you change some logic, tests make sure your program is working as intended or not. If tests fail, then you know what to do!
This library provides 100% test coverage and this helped me allot.
You can test your program with the following command.
go test -cover
Benchmark your program
Don't just get to the conclusion without benchmarking your programs. Go tooling makes it super easy to benchmark your programs. You get the following information from benchmarks for a single function.
- Number of operations performed
- Time required for each operation in nanoseconds
- Data processed per second
- Memory required per operation
- Allocations made per operation
You can benchmark your program with the following command.
go test -bench=. -benchmem
Have a look at benchmarks from this package. As you can see, using io.Reader, memory is being reused resulting in almost 500% performance increase in data processing than existing available library jsonc
goos: windows
goarch: amd64
pkg: github.com/akshaybharambe14/go-jsonc/benchmarks
BenchmarkOwnSmallJSONBytes-4 256599 4952 ns/op 353.00 MB/s 0 B/op 0 allocs/op
BenchmarkOwnSmallJSONBytesReader-4 206823 5832 ns/op 299.70 MB/s 6224 B/op 5 allocs/op
BenchmarkJSONCSmallJSONBytes-4 171474 6925 ns/op 252.41 MB/s 1792 B/op 1 allocs/op
BenchmarkOwnBigJSONBytes-4 33517 35921 ns/op 462.26 MB/s 0 B/op 0 allocs/op
BenchmarkOwnBigJSONBytesReader-4 105244 11292 ns/op 1470.45 MB/s 6224 B/op 5 allocs/op
BenchmarkJSONCBigJSONBytes-4 19599 61422 ns/op 270.34 MB/s 18432 B/op 1 allocs/op
PASS
ok github.com/akshaybharambe14/go-jsonc/benchmarks 26.250s
CPU profiling and memory profiling
If you think your program is not fast enough, then you can profile your programs. This really helps in finding the culprits.
In my case, I identified that my decode function was spending almost 40% time in just checking the byte can be added to resulting JSON or not. Results, I ended up gaining a 40% performance increase.
You don't need any separate tools to profile your programs. Benchmarks can be used for this also. You can generate profiles and read them with pprof.
go test -bench=. -benchmem -memprofile memprofile.out -cpuprofile profile.out
You can't avoid all allocations
I was happy with the above tweaks, but still, I was trying to avoid those 5 allocations per operations in the case of io.Reader as input (See benchmarks). But seriously stop being greedy and check it in the memory profile if those allocations are avoidable or not.
In my case, internal functions were making allocations to the heap and I can't control them. So I stopped there.
Work on feedbacks
Don't stop here and work on the feedback provided by your colleagues, friends, and community. File issues and work on them in your free time.
That's it for now. It was a nice experience and I am looking forward to contributing to some opensource go projects.
akshaybharambe14
/
go-jsonc
go-jsonc provides a way to work with commented json by converting it to plain json.
Give it a try. Your feedback is highly appreciated. I will surely work on your inputs, that's what I just learned.
Happy coding!
Top comments (2)
I've used golang in two or three college projects and so far so good. Though, I felt overwhelmed sometimes while looking at the stdlib because I felt like there was too many interfaces and such and many ways to do one thing. What do you recommend looking at to get a good grasp on the stdlib? Thanks!
I felt like there was too many interfaces and such and many ways to do one thing. This is a little contradictory statement as go encourages us to have only one possible way to do something.The interfaces in the standard libraries are essential for compatibility. I recommend you go through the documentation of the respective interface. Go has well-written documentation for the standard library. Let me if I missed something.