One of the biggest differences between folks on the business side and the engineering side is the way they view certain problems. While having tons of data is seen as a serious issue that requires tons of management by developers, business folks tend to see it as a boon, as a way to build a business. Since problems like this only show up after bit of success, these can easily be dubbed luxury problems. A lot of the time, these luxury problems manifest themselves as scaling issues, mostly in the database layer.
As the Behance Network has grown and traffic increased, more and more challenges appeared. We’ve taken these problems as a sign that things are growing. Our MySQL databases are a great example. No matter how poorly the database was designed, when it was brand new and empty, it performed great. Especially in the incubation stage, when the Behance Network saw very few visitors, and only had a little bit of data to contend with.
Once we put some stress on the server, things fell apart quickly. Looking back, this was a great problem to have, considering the alternative was to never have problems because no one ever puts data into your app. It was also an opportunity for our team to take a look at what happens under load, and find ways to fix it. As we’ve found out, this isn’t just a simple opportunity, but a never ending cycle of struggling to keep up with demand. I hear this is what some business folks call a “viable business model.”
Without problems stemming from growth, there are few opportunities for a team to learn the best way to solve tough problems. These moments should be looked forward to, not dreaded. Notes, knowledge, processes and data the comes from resolving tough problems should be regarded as the most valuable property a company has.
Small, obscure optimizations sometimes have the potential to make the greatest impact. For example, every time a connection is made, MySQL will do a DNS lookup of the host that is trying to connect. If MySQL is handling many connections, the overhead of an extra DNS lookup can be hefty, simply because of the number of extra operations that have to be performed before MySQL can actually start doing actual work
Thankfully, there is an option in recent versions (4.1+) of MySQL that will instruct MySQL to skip the extra DNS lookup. It’s a fairly obscure option called skip-name-resolve. The only caveat to using this option is that the users defined the GRANT tables can only use IP addresses as hostnames. For most MySQL users, this shouldn’t be an issue