So there were two fundamental problems with this architecture that we needed to solve very quickly

And we had to do this every single day in order to deliver fresh and accurate matches to our customers, especially one of those new matches that we deliver to you may be the love of your life

The first problem was related to the ability to perform high volume, bi-directional searches. And the second problem was the ability to persist a billion plus of potential matches at scale.

So finally, the last issue was related to since we are running on Postgres, we start using a lot of several advanced indexing techniques with a complicated table structure that was very Postgres-specific in order to optimize our query for much, much faster output

So here was our v2 architecture of the CMP application. We wanted to scale the high volume, bi-directional searches, so that we could reduce the load on the central database. So we start creating a bunch of very high-end powerful machines to host the relational Postgres database. Each one of the CMP applications was co-located with a local Postgres database server that stored a complete searchable data, so that it could perform queries locally, hence reducing the load on the central database.

So the solution worked pretty well for a couple years, but with the rapid growth of eHarmony user base, the data size became bigger, and the data model became more complex. This architecture also became problematic. So we had five different issues as part of this architecture.

So one of the biggest challenges for us was the throughput, obviously, right? It was taking us about more than two weeks to reprocess everyone in our entire matching system. More than two weeks. We don’t want to miss that. So of course, this was not an acceptable solution to our business, but also, more importantly, to our customer. So the second issue was, we’re doing massive court operation, 3 billion plus per day on the primary database to persist a billion plus of matches. And these current operations are killing the central database. And at this point in time, with this current architecture, we only used the Postgres relational database server for bi-directional, multi-attribute queries, but not for storing. So the massive court operation to store the matching data was not only killing our central database, but also creating a lot of excessive locking on some of our data models, because the same database was being shared by multiple downstream systems.

And the fourth issue was the challenge of adding a new attribute to the schema or data model. Every single time we make any schema changes, such as adding a new attribute to the data model, it was a complete night. We have spent several hours first extracting the data dump from Postgres, massaging the data, copy it to multiple servers and multiple machines, reloading the data back to Postgres, and that translated to a lot of high operational cost to maintain this solution. And it was a lot worse if that particular attribute needed to be part of an index.

So finally, any time we make any schema changes, it requires downtime for our CMP application. And it’s affecting our client application SLA. So the application design became a lot more Postgres-dependent, and that was not an acceptable or maintainable solution for us.

So at this point, the direction was very simple. We had to fix this, and we needed to fix it now. So my entire engineering team started to do a lot of brainstorming about from application architecture to the underlying data store, and we realized that most of the bottlenecks are related to the underlying data store, whether it’s related to querying the data, multi-attribute queries, or it’s related to storing the data at scale. So we started to define the new data store requirements that we’re going to select. And it had to be centralized.