One of the hottest topics right now is “big data”. While the buzz of big data has penetrated nearly all verticals and markets from executives to technology architects, what seems to be getting lost is “medium data”. Big data is not for everybody – it is a radical, fundamental shift in how information is managed. Organizations with data issues may think Hadoop or sharded NoSQL clusters are their only way out. However, the 5 tips below can help you to get the most of the “medium data” platform you already know and love.
- Be informed. Know what tools are available within your current technology stack. If you are using a mature RDS database like Oracle or SQL Server, become an expert in how those products are designed to scale. If your issue is large tables, then know how to materialize views and partition tables to improve database engine processing. If your issue is complex and large queries, then know how to properly analyze execution plans, leverage statistics and procedures, and optimize lookups with well-designed indexes to verify your changes make the designed impact.
- Be proactive. Monitor your most expensive queries by memory usage or IO cycles. Track your largest tables by size and rows. By knowing the most likely points of failure as your system capacity diminishes, you can strategically invest in architectural improvements before they become high priority bug fixes.
- Be curious. Explore creative approaches when solving your performance bottlenecks. Maybe the solution to the bottleneck is another pipeline altogether. Not every slow query can improve with just an index or cache, but scalability is not achieved with just hardware.
- Be cautious. Do not try to embrace the latest and greatest technology too quickly. It pays to let new products mature and take their lumps before investing your own time and energy that could be better spent leveraging your existing platform.
- Be deliberate. You may realize that your only solution is a big data solution, but don’t rush in too quickly. Without proper planning and ensuring the right technical and operational skillsets are in place, your organization may sacrifice the reliability, scalability, and flexibility these systems are designed to bring.
Hopefully, you will find these tips useful as your teams and products grow. Then, perhaps you’ll find the jump from medium data to big data easier and life changing.