My Experiments with Showcast - A Thread - Part I:

In a galaxy, far, far away, there was an idea. By @saxenasaheb. It started as a fun exercise of connecting frens with FC accounts randomly on a video call (ala Omegle).

Assessed the requirements, went through a couple of github repos around randomised calling. Easy, peasy.

Infra to setup video calling - Check
Farcaster based Auth - Check
People eager for it - Check

This started way back in February. I quickly cobbled together a prototype using Huddle’s Video calling sdk. The Farcaster auth kit helped set up the auth pathway.

So far, so gooood! Flying out of the blocks. An ugly but working prototype when connecting 2-3 people. WGMI!

But then, it is a dev’s life. Our happiness makes the universe conspire to take it away from us!

Challenge 1: Make Huddle Auth work with Farcaster Auth (custom credential provider built over SIWE)

Mistake: I thought I need to make them work together. Looked into custom JWT (fetching & wrapping Huddle room token into another token which would also serve as the auth for Showcase app on top of FC auth). Way too complicated an approach. 

Got deeper and deeper into auth & JWT mechanisms, loved the learning aspect of it and forgot the original purpose. 

(And THAT is why we devs need great product managers! And a problem that I am consciously working on!). I spent about a week down this rabbit hole. Went nowhere but learnt a ton of stuff.

Meanwhile, #wenshowcast grumblings were just starting to appear.

Correction: Separated out both the auth mechanisms. 

Showcast would work on and accept FC Auth and the session generated would serve as the basis for all interactions within the app. Huddle API & room token auth would work as another layer nested inside and only for interactions with the video room.

So obvious in hindsight!

Challenge 2: Randomisation in room allocation

The core question was: "How do you allocate a room at random to a user signing in?”

Mistake: Convoluting a logic based on a custom random allocation logic by storing the room states in my own DB while also trying to use the Host-Peer joining & participants mechanisms provided by Huddle API simultaneously

Overengineering dev brain at work again!

Correction: Isolated all Huddle interactions. Forced myself to ignore their data apis aournd details of a room and number of participants in them. Setup own DB with rooms table having three main columns: huddleRoom | hostJoined (bool) | peerJoined (bool)

The two columns could also have been setup as roomStatus as available and occupied. I could visualise the binary flags in terms of the four combinations better (TT, TF, FF, FT) (T and F being True and False). Since, I would treat both users equivalently, TF and FT states were equivalent. 

So, this was the basic outline for the algorithm:

Whenever a user signed in, the app would look up in the DB and find if the rooms with state as TF  (available with one user sitting).

If yes, pick a random room from this list and send the user to that room and mark the room as TT.
If no, it would look for rooms with state as FF (vacant). Pick a random room & send the user there and mark it as TF.
If no again (no vacant rooms), create a new room and send the user there.

The reverse process would happen when you left a room - TT => TF | TF => FF

This was all happening just after the user had signed in. Logic was sound and testing with a few users, things seemed to be working. 

There were CSS issues around matching it with the design and responsiveness. But I had folks helping me (thanks to 0xbhaisahab and Idrees)

I knew it was slow! But, confirmation bias led me to believe it was decent enough to launch.

Dev nirvana achieved. This was sometime in March. We had launched a website and domain. But that was the extent of it.

Also identified another problem which I will discuss below. Was hoping though it would not be an issue till we had achieved a decent number of users.

Can you guess what that was?

A few other projects came along and gradually I lost interest in taking this forward.

The murmurs of #wenshowcast would occasionally appear in the Telegram channels, often led by Saxenasaheb but was too occupied with other stuff. And so, April went by. 

Everything - live but dormant with a few css niggles.

In my mind, I was “over it” so to say. I buried it in the graveyard of dormant products and paid my respectful obituary with my final commit message. 

"I thank thee for the learnings. Thy watch is now over. Rest in peace till forked”

And, git push!

And then, FBI Fellowship happens. 

Saxenasaheb calls me up, asks me to launch Showcast at the Fellowship and leads a revolution of #wenshowcast everywhere.

And just like that, Showcast was undead!

Fixed the css issues & a couple of days back, pushed the final code into production (followed by premature celebrations). 

It built successfully on Vercel. Did a very soft test with 4 people. It worked as expected - joined the room at random, left it to be reassigned another random room and so on. 

I thought it was ready for launch (again). Then worked on the other project @basedbuilders for the rest of the day. Oh, how happy I was!

Challenge 3:  Race conditions strike back! Now that’s what I am talking about!

Mistake: Yesterday, we went for a soft launch with 8-10 people at the fellowship and all hell broke loose.

Slow internet + 5-10 second latency in joining and leaving rooms + similar latency in updating the database flags for available rooms’ state = recipe for disaster. 

People were being assigned the same rooms before it could get updated in the DB and be removed from the pool of available rooms.

I had never seen a live demo of race conditions working to perfection in your life.

It was hilarious and tragic at the same time.

My Experiments with Showcast - A Thread - Part II:

Before I get into the ongoing effort to ship Showcast,  a massive massive thank you to @proxystudio.eth for motivating me to do this & share this story with everyone!

Correction: So where were we! Yeah! Race conditions!

Once everything broke, was forced to go back and review everything.
I knew deep in my heart that it was a case of race conditions but still wanted to be sure before moving ahead with revamping the whole thing.

Kudos to @vrajdesai from Huddle for sitting together on such a short notice and reviewing the code and algorithm.

He came to the same conclusion - race conditions due to latency.

No way out. Have to revamp. Dev shortcut path cut short! 

We came up with two potential solutions.

Solution 1: A transaction at DB level which could lock the DB for each user, run the randomisation algo, allocate the room & update the room state (TT, TF) - an implicit sequential processing for each user.

Pros: Resolves it at the source

Cons: Required SQL may be too convoluted for maintaining it

Solution 2: A queue (Kafka) - 

User signs in & their id gets added to a queue.

A consumer receives the user id & picks a room.

A new table to maintain (userId | allocatedRoomId) was added.

Meanwhile, the user waits in a lobby & keeps polling to find out the roomId assigned to them.

Sounds simple, doesn’t it?

My Experiments with Showcast - A Thread - Part I:

In a galaxy, far, far away, there was an idea. By @saxenasaheb. It started as a fun exercise of connecting frens with FC accounts randomly on a video call (ala Omegle).

Assessed the requirements, went through a couple of github repos around randomised calling. Easy, peasy.

Infra to setup video calling - Check
Farcaster based Auth - Check
People eager for it - Check

This started way back in February. I quickly cobbled together a prototype using Huddle’s Video calling sdk. The Farcaster auth kit helped set up the auth pathway.

So far, so gooood! Flying out of the blocks. An ugly but working prototype when connecting 2-3 people. WGMI!

But then, it is a dev’s life. Our happiness makes the universe conspire to take it away from us!

Challenge 1: Make Huddle Auth work with Farcaster Auth (custom credential provider built over SIWE)

Mistake: I thought I need to make them work together. Looked into custom JWT (fetching & wrapping Huddle room token into another token which would also serve as the auth for Showcase app on top of FC auth). Way too complicated an approach. 

Got deeper and deeper into auth & JWT mechanisms, loved the learning aspect of it and forgot the original purpose. 

(And THAT is why we devs need great product managers! And a problem that I am consciously working on!). I spent about a week down this rabbit hole. Went nowhere but learnt a ton of stuff.

Meanwhile, #wenshowcast grumblings were just starting to appear.

Correction: Separated out both the auth mechanisms. 

Showcast would work on and accept FC Auth and the session generated would serve as the basis for all interactions within the app. Huddle API & room token auth would work as another layer nested inside and only for interactions with the video room.

So obvious in hindsight!

Challenge 2: Randomisation in room allocation

The core question was: "How do you allocate a room at random to a user signing in?”

Mistake: Convoluting a logic based on a custom random allocation logic by storing the room states in my own DB while also trying to use the Host-Peer joining & participants mechanisms provided by Huddle API simultaneously

Overengineering dev brain at work again!

Correction: Isolated all Huddle interactions. Forced myself to ignore their data apis aournd details of a room and number of participants in them. Setup own DB with rooms table having three main columns: huddleRoom | hostJoined (bool) | peerJoined (bool)

The two columns could also have been setup as roomStatus as available and occupied. I could visualise the binary flags in terms of the four combinations better (TT, TF, FF, FT) (T and F being True and False). Since, I would treat both users equivalently, TF and FT states were equivalent. 

So, this was the basic outline for the algorithm:

Whenever a user signed in, the app would look up in the DB and find if the rooms with state as TF  (available with one user sitting).

If yes, pick a random room from this list and send the user to that room and mark the room as TT.
If no, it would look for rooms with state as FF (vacant). Pick a random room & send the user there and mark it as TF.
If no again (no vacant rooms), create a new room and send the user there.

The reverse process would happen when you left a room - TT => TF | TF => FF

This was all happening just after the user had signed in. Logic was sound and testing with a few users, things seemed to be working. 

There were CSS issues around matching it with the design and responsiveness. But I had folks helping me (thanks to 0xbhaisahab and Idrees)

I knew it was slow! But, confirmation bias led me to believe it was decent enough to launch.

Dev nirvana achieved. This was sometime in March. We had launched a website and domain. But that was the extent of it.

Also identified another problem which I will discuss below. Was hoping though it would not be an issue till we had achieved a decent number of users.

Can you guess what that was?

A few other projects came along and gradually I lost interest in taking this forward.

The murmurs of #wenshowcast would occasionally appear in the Telegram channels, often led by Saxenasaheb but was too occupied with other stuff. And so, April went by. 

Everything - live but dormant with a few css niggles.

In my mind, I was “over it” so to say. I buried it in the graveyard of dormant products and paid my respectful obituary with my final commit message. 

"I thank thee for the learnings. Thy watch is now over. Rest in peace till forked”

And, git push!

And then, FBI Fellowship happens. 

Saxenasaheb calls me up, asks me to launch Showcast at the Fellowship and leads a revolution of #wenshowcast everywhere.

And just like that, Showcast was undead!

Fixed the css issues & a couple of days back, pushed the final code into production (followed by premature celebrations). 

It built successfully on Vercel. Did a very soft test with 4 people. It worked as expected - joined the room at random, left it to be reassigned another random room and so on. 

I thought it was ready for launch (again). Then worked on the other project @basedbuilders for the rest of the day. Oh, how happy I was!

Challenge 3:  Race conditions strike back! Now that’s what I am talking about!

Mistake: Yesterday, we went for a soft launch with 8-10 people at the fellowship and all hell broke loose.

Slow internet + 5-10 second latency in joining and leaving rooms + similar latency in updating the database flags for available rooms’ state = recipe for disaster. 

People were being assigned the same rooms before it could get updated in the DB and be removed from the pool of available rooms.

I had never seen a live demo of race conditions working to perfection in your life.

It was hilarious and tragic at the same time.

My Experiments with Showcast - A Thread - Part II:

Before I get into the ongoing effort to ship Showcast,  a massive massive thank you to @proxystudio.eth for motivating me to do this & share this story with everyone!

Correction: So where were we! Yeah! Race conditions!

Once everything broke, was forced to go back and review everything.
I knew deep in my heart that it was a case of race conditions but still wanted to be sure before moving ahead with revamping the whole thing.

Kudos to @vrajdesai from Huddle for sitting together on such a short notice and reviewing the code and algorithm.

He came to the same conclusion - race conditions due to latency.

No way out. Have to revamp. Dev shortcut path cut short! 

We came up with two potential solutions.

Solution 1: A transaction at DB level which could lock the DB for each user, run the randomisation algo, allocate the room & update the room state (TT, TF) - an implicit sequential processing for each user.

Pros: Resolves it at the source

Cons: Required SQL may be too convoluted for maintaining it

Solution 2: A queue (Kafka) - 

User signs in & their id gets added to a queue.

A consumer receives the user id & picks a room.

A new table to maintain (userId | allocatedRoomId) was added.

Meanwhile, the user waits in a lobby & keeps polling to find out the roomId assigned to them.

Sounds simple, doesn’t it?