DEV Community

Cover image for Binoculearn.ai β€” π˜“π˜¦π˜’π˜³π˜―π˜ͺ𝘯𝘨 π˜ͺ𝘯 𝘭𝘰𝘸-𝘣𝘒𝘯π˜₯𝘸π˜ͺπ˜₯𝘡𝘩 π˜ͺ𝘯𝘡𝘦𝘳𝘯𝘦𝘡 π˜™π˜¦π˜₯𝘦𝘧π˜ͺ𝘯𝘦π˜₯⚑
Pratyay Banerjee
Pratyay Banerjee

Posted on • Updated on

Binoculearn.ai β€” π˜“π˜¦π˜’π˜³π˜―π˜ͺ𝘯𝘨 π˜ͺ𝘯 𝘭𝘰𝘸-𝘣𝘒𝘯π˜₯𝘸π˜ͺπ˜₯𝘡𝘩 π˜ͺ𝘯𝘡𝘦𝘳𝘯𝘦𝘡 π˜™π˜¦π˜₯𝘦𝘧π˜ͺ𝘯𝘦π˜₯⚑

Category Submission:

  • Choose Your Own Adventure βš›
  • Google Cloud Superstar ☁️

What we built πŸ€”

Binoculearn is a bleeding-edge smart P2P educational video conferencing web application aimed to deliver a reliable frame rate & is backed by low-latency support along with low jitter (smooth and consistent), as well as high audio quality. We do this by converting the video stream into ASCII characters on the client side and send it via WebRTC using Twilio’s video conferencing service deployed on Google Cloud App Run ☁️ & is fuelled by MongoDB Atlas! πŸƒ

image

πŸ’‘ Implementing video conferencing using this technique saves bandwidth bidirectionally & especially on the receiver end. This method is both vertically & horizontally scalable as we can feed more users as they enter the conference.

App Tryout Link πŸ”—

Binoculearn.Ai πŸ‘‰ https://binoculearn-5jjb23a4oq-ue.a.run.app [Deployed on Google Cloud App Run ☁️]
image

Features 🎠

  • P2P lagfree video conferencing app with ultra-low bandwidth support!
  • Bleeding-edge Image Compression Algorithm!
  • Vertically & Horizontally Scaleble [Currently capped at 4, because of Twilio Credits]
  • Twilio Live Transcription [Stored in MongoDB Atlas]
  • P2P Messaging with Sentiment Analysis via Natural Language API!
  • Generate Summary & Transcript of the meeting!
  • File Sharing (blob) via MongoDB Atlas!
  • User Dashboard with Previous Activity Tracker!
  • Minimalist UI/UX powered by ReactJS & Tailwind CSS
  • High Quality Multiplexed Audio!
  • Overall Meeting Emotion Tracker
  • MongoDB Atlas as Non-SQL DB
  • Deployed on Google Cloud App Run
  • Saves πŸ’° + Internet Data!
  • Secure O-Auth via Firebase by Google!
  • 100% GDPR compliant & SEO friendly inteface!

Video ▢️

Privacy & Security πŸ”

Binoculearn deals with a wide range of sensitive information. In the wrong hands, this data could dramatically harm individuals. We took special efforts and considerations to ensure that our platform protects the privacy and sensitive information of all of our users making it 100% GDPR compliant!

We also made sure that all data is sent securely over the network. Binoculearn leverages the security benefits of TLS for encryption. We also encoded all of our data using Base64 encoding. Ideally, in a future iteration, we would like to encrypt all data using a more secure method.

Background πŸ“œ

As ubiquitous and fast as the internet seems in developed countries, developing countries still struggle with reliable internet connections. The impact of poor internet connectivity exasperate the education inequality between children from prosperous countries and children from developing countries, because the latter cannot benefit from remote learning via video conferencing. πŸ˜”

MSU News

Last year, it was found that millions of students in the state of Odisha in India are stuck at home with no access to either internet or online education. My friend (teammate) Subham Sahu, an Odisha native, has had first-hand experience of interruptions during his undergraduate studies.

For even those who have access to the internet, the price is premium and the bandwidth is limited. For instance, while talking to his parents in India, Subham found that they frequently run out of their allocated 1 GB far before the allowance period, after which the bandwidth gets throttled: stalled frames, choppy audio, painful delays, and eventual disconnections, and subsequent retries are a normal occurrence, but still arguably much better than normal telephone conversations because he gets to β€œsee” them.

At the heart of the problem may lack tele-infrastructure for the implementation of education on virtual platforms.

Why RGB Frames are heavy? Why of What?
image image

Videos are array of images that consists of large bytes of data which packets transfer from one place to another. Using our smart algorithm, we are compressing the image and then converting it into grayscale bit images which are then transfused by ASCII characters while transmitting. πŸͺ„

To address this problem, we propose a new approach based on the insight that if we are willing to give up some realism or realistic rendering of faces and screens, then there is a whole new world of face and screen representations that can be derived for ultra-low bandwidth, with an acceptable quality of experience! πŸ‘ͺ

The proposed solution can be primarily implemented as software needing no change in the underlying infrastructure! This would in turn be cheaper, and allow internet access to people that are currently being marginalized based on their affordability.

We aim to prioritize a reliable frame rate with low latency and low jitter (smooth and consistent), as well as high audio quality serving the purpose of online education for the better! ✨

Group-1686550957.gif

Our goal with this platform is to connect students in poorly connected areas with highly qualified teachers in metropolitan areas and abroad to facilitate Cost-Effective Stable Remote Collaboration.


Screenshots πŸ–ΌοΈ

Home Page [Before O-Auth] Home Page [After O-Auth]
image image
Host a Meeting [Old / 16:9] Join a Meeting [Old / 16:9]
image image
Host a Meeting [New] Join a Meeting [New]
image image
User Dashboard [New] Dashboard Insight [New]
image image

ezgif-2-2129dd4bf5

☝️ Final Demo done by Pratyay [Click Above]

Description πŸ¦„

Experience a Superfast, low-latency P2P videochat even on ultra-low Bandwidth networks. Redefining the communication gap, Binoculearn is a MIT licensed open-sourced project made for students, by the students & will be Free Forever! ⚑

On top of the bandwidth-saving functionality, we also offer educational and content-moderation tools like Sentiment Analysis via Google Cloud's Natural Language API for session chat QnA and our custom Trained ML model deployed on GCP for Meeting Summarization.

image

These features allow both the educators and students to maintain decorum in the meeting and also have follow-up material to retain information about the meeting!

Link to Source Code πŸ‘¨β€πŸ’»

GitHub logo Neilblaze / Binoculearn.AI

π˜“π˜¦π˜’π˜³π˜―π˜ͺ𝘯𝘨 π˜ͺ𝘯 𝘭𝘰𝘸-𝘣𝘒𝘯π˜₯𝘸π˜ͺπ˜₯𝘡𝘩 π˜ͺ𝘯𝘡𝘦𝘳𝘯𝘦𝘡 π˜™π˜¦π˜₯𝘦𝘧π˜ͺ𝘯𝘦π˜₯⚑ β€” Project Submission for MongoDB Atlas Hackathon'22

thumbnail-gh

Binoculearn.ai

π˜“π˜¦π˜’π˜³π˜―π˜ͺ𝘯𝘨 π˜ͺ𝘯 𝘭𝘰𝘸-𝘣𝘒𝘯π˜₯𝘸π˜ͺπ˜₯𝘡𝘩 π˜ͺ𝘯𝘡𝘦𝘳𝘯𝘦𝘡 π˜™π˜¦π˜₯𝘦𝘧π˜ͺ𝘯𝘦π˜₯⚑ β€” Project Submission for MongoDB Atlas Hackathon'22 πŸƒ

Binoculearn is a bleeding-edge smart P2P educational video conferencing web application aimed to deliver a reliable frame rate & is backed by low-latency support along with low jitter (smooth and consistent), as well as high audio quality. We do this by converting the video stream into ASCII characters on the client side and send it via WebRTC using Twilio’s video conferencing service & is fuelled by MongoDB Atlas! πŸƒ

image

πŸ’‘ Implementing video conferencing using this technique saves bandwidth bidirectionally & especially on the receiver end. This method is horizontally scalable as we can feed more users as they enter the conference.


Installing / Getting started

There are two folders, where my-app is for the front-end & server is for the backend.

Setting up Dev [Make sure .env is loaded with your own credentials]

git clone
…
Enter fullscreen mode Exit fullscreen mode

Permissive License βš–οΈ

MIT

How we built it βš™οΈ

First and foremost, it is Crafted with πŸ’™. The whole process can be broken into the following points :-

  • ➀ React.JS, Redux + Tailwind CSS on the frontend
  • ➀ Express.js, Node.js, Sockets, WebRTC, Twilio Live on the backend

image

  • ➀ Prisma for connecting the Frontend to the MongoDB Atlas to store user Data + Logs
  • ➀ External services like Twilio, GCP Natural Language API
  • ➀ GitHub as CI/CD and Google's App Engine for Deployment [Duplicate Deployment done on Vercel]

image

QnA model Architecture BERT: Summarizer Architecture
image-172.png image-173.png

➀ Prisma Schema for MongoDB Atlas:
image

➀ Database Deployments for MongoDB Atlas [Click Below .Gif ⬇️]:
image

➀ Data Services β€” MongoDB Atlas [Click Below .Gif ⬇️]:
Recording

➀ Collections β€” MongoDB Atlas [Binoculearn]:
Recording2

➀ Session Timeline-Chart MongoDB Atlas [Binoculearn]:
image

➀ Session Sentiment β€” Timeline MongoDB Atlas [Binoculearn]:
image

Why MongoDB Atlas?

Databases are always a tricky thing, as they're stateful in nature. Although our app is not serving millions of customers, but as software enthusiast, we strive to build an infinitely scalable application. And here MongoDB Atlas helped a lot.

image

We didn't have to manage the database, no need to work on complex networking between shards, etc. (And thanks to Google Cloud Run, our server can scale too. Hopefully this app gets viral and we're able to cater a million real users! 😊)

We used MongoDB as our primary database to store user sessions, meeting details, user socket IDs (which powers our real-time websockets engine), etc.

We're also generating summary of the meeting, and instead of generating in an synchronous flow, we're using MongoDB realm to have a serverless function hosted to generate the summary. In this way, we're able to keep the app more cohesive! πŸ™‚

We learnt a lot of things including most of the MongoDB Realm, and many of the features of MongoDB Atlas. Data modelling in MongoDB was bit tricky, as both Neel and I come from transitional relational database space. We together studied few of the video lectures from MongoDB University to understand the anti patterns and in-depth data modelling in MongoDB.

Apart from MongoDB we learnt about Google App Engine, Cloud run, and Cloud build. 🌟


Design 🎨

We were heavily inspired by the revised version of Double Diamond design process, a model popularized by the British Design Council, which not only includes visual design, but a full-fledged research cycle in which you must discover and define your problem before tackling your solution & then finally deploy it.

UPDATETHIS

  1. Discover: a deep dive into the problem we are trying to solve.
  2. Define: synthesizing the information from the discovery phase into a problem definition.
  3. Develop: think up solutions to the problem.
  4. Deliver: pick the best solution and build that.

Moreover, we utilized design tools like Figma, Photoshop & Illustrator to prototype our designs before doing any coding. Through this, we are able to get iterative feedback so that we spend less time re-writing code.

image-178.png

Research πŸ“š

Research is the key to empathizing with users: we found our specific user group early and that paves the way for our whole project. Here are a few of the resources that were helpful to us β€”

CREDITS

  • Design Resources : Freepik, Behance
  • Icons : Icons8, fontawesome
  • Font : Urbanist / Roboto / Raleway

Challenges we ran into 😀

This project was initially built in under 24 hours, 19th Nov β€” 20th Nov, 2022 & I actually got to know about the MongoDB Atlas hackathon from Subham on 20th of November, 2022 (IST time) when we were eagerly waiting for MetroHack's result. Personally, I'm a Hackathon freak. I love hackathons because it helps to generate specific ideas in a distinct domain within a small span of time & as a result, it not only increases the creativity but also enhances the curiosity while tackling corner cases while building the project/product. Since then, we kept committing towards this project. We [I, Subham & Gaurang] did face some challenges during the hackathon, many of which ironically related to working remotely. One of the major challenges was the time difference. All of us participated from different time zones, which created communication challenges.

One of the biggest challenge we faced was with deployment. Since our application has a web socket endpoint, we had very few deployment options. We initially thought that App Engine standard environment would serve the purpose, but in the end realised that it doesn't support Websockets. Finally we had to migrate our deployment stack to Google Cloud Run, which provided us the flexibility of environment and also allowed to use web sockets.

image

We are proud of finishing the project on time which seemed like a tough task as we started working on it quite late due to other commitments. We were also able to add most of the features that we envisioned for the app during ideation. And as always, working overnight was pretty fun! :)

What's next? πŸš€

We believe that our App has great potential. We just really want this project to have a positive impact on people's lives! We would love to make it more scalable & cross-platform so that the user interaction increases to a great extent! Also, it's noted that we do have a bunch of ideas in our bucket-list which we are looking forward to transform into reality!

Conclusion 🐣

It has been all fun, & I would love to thank my buddies @subhamx & Gaurang for helping me, & Special thanks goes to @stanimiravlaeva, @mlynn & @joel__lord πŸ™Œ. And as always, thank you #DEV #DEVCommunity & #MongoDB for hosting this hackathon! πŸ’š

Update [13/12/2022] ⚠️ β€” Few days ago, we ran out of credits, hence API credentials had been revoked! You can run the same on your local, hence, use your own credentials. But recently, we re-deployed it via Google Cloud Run, so hopefully you can explore our live-app at β€” https://binoculearn-5jjb23a4oq-ue.a.run.app

Update [15/12/2022] ⚠️ β€” Replaced Old Demo Video with New one!

breaker.png

Top comments (4)

Collapse
 
imamdev_ profile image
Imamuzzaki Abu Salam • Edited

What tools do you use to create the flowchart? It's so πŸ’– beatufiul and easy to understand.

Collapse
 
neilblaze profile image
Pratyay Banerjee

Thanks a lot @imbios πŸ˜„
I made it using Figma.

Collapse
 
stytchastory profile image
Stytch • Edited

Stytch specializes in 2D Animation, 3D Animation and video production. Every brand has a story, a personality, and characteristics, our job is to understand your story and to express it in the best way possible. The team at Stytch offers full-service video production services including ideation, script writing, storyboarding, video shooting, 3D modeling, rigging, sound design, color correction, video editing, pre and post production. For any query or more information, visit the official website.

ADDRESS

AAC-006 2nd Floor Shipra Krishna Azure, Kaushambi, Ghaziabad, Uttar Pradesh 201010

+91 99104 09680 | info@stytchastory.com | stytchastory.com

SOCIAL

Instagram | LinkedIn | YouTube

TAGS

2D Animation Company | 3D Animation Company | Video Production Company | 2D Animation Services | 3D Animation Services | Video Production Services | 2D Animation Studio | 3D Animation Studio | Video Production Studio | 2D Animation Agency | 3D Animation Agency | Video Production Agency | anamorphic animation | real estate walkthrough video | documentary video format | ai character rigging | professional animators | motion capture ai | procedural animation | rotoscoping vs mocap | saas explainer videos | cgi vs vfx

Collapse
 
jishnujd profile image
Jishnu • Edited

Best Digital marketing institute in Jayanagar, Bangalore

(digitalacademy360.com/digital-mark...)