drupal

Solr for Drupal Developers, Part 1: Intro to Apache Solr

It's common knowledge in the Drupal community that Apache Solr (and other text-optimized search engines like Elasticsearch) blow database-backed search out of the water in terms of speed, relevance, and functionality. But most developers don't really know why, or just how much an engine like Solr can help them.

I'm going to be writing a series of blog posts on Apache Solr and Drupal, and while some parts of the series will be very Drupal-centric, I hope I'll be able to illuminate why Solr itself (and other search engines like it) are so effective, and why you should be using them instead of simple database-backed search (like Drupal core's Search module uses by default), even for small sites where search isn't a primary feature.

As an aside, I am writing this series of blog posts from the perspective of a Drupal developer who has worked with large-scale, highly customized Solr search for Mercy (example), and with a variety of small-to-medium sites who are using Hosted Apache Solr, a service I've been running as part of Midwestern Mac since early 2011.

Why not Database?

Apache Solr's wiki leads off it's Why Use Solr page with the following:

If your use case requires a person to type words into a search box, you want a text search engine like Solr.

At a basic level, databases are optimized for storing and retrieiving bits of data, usually either a record at a time, or in batches. And relational databases like MySQL, MariaDB, PostgreSQL, and SQLite are set up in such a way that data is stored in various tables and fields, rather than in one large bucket per record.

In Drupal, a typical node entity will have a title in the node table, a body in the field_data_body table, maybe an image with a description in another table, an author whose name is in the users table, etc. Usually, you want to allow users of your site to enter a keyword in a search box and search through all the data stored across all those fields.

Drupal's Search module avoids making ugly and slow search queries by building an index of all the search terms on the site, and storing that index inside a separate database table, which is then used to map keywords to entities that match those keywords. Drupal's venerable Views module will even enable you to bypass the search indexing and search directly in multiple tables for a certain keyword. So what's the downside?

Mainly, performance. Databases are built to be efficient query engines—provide a specific set of parameters, and the database returns a specific set of data. Most databases are not optimized for arbitrary string-based search. Queries where you use LIKE '%keyword%' are not that well optimized, and will be slow—especially if the query is being used across multiple JOINed tables! And even if you use the Search module or some other method of pre-indexing all the keyword data, relational databases will still be less efficient (and require much more work on a developer's part) for arbitrary text searches.

If you're simply building lists of data based on very specific parameters (especially where the conditions for your query all utilize speedy indexes in the database), a relational database like MySQL will be highly effective. But usually, for search, you don't just have a couple options and maybe a custom sort—you have a keyword field (primarily), and end users have high expectations that they'll find what they're looking for by simply entering a few keywords and clicking 'Search'.

Moving on to Acquia

I wanted to post this here, since this is more of my sounding board for the Drupal community, but the details are on my personal blog: starting October 6, I will be working for Acquia as a Technical Architect in their Professional Services group!

What does this mean for this site/blog, Hosted Apache Solr, and Server Check.in? Not much, actually—they will continue on, likely at the same pace of development they've been for the past year or so (I'll work on them when I get an odd hour or two...). I am still working on completing Ansible for DevOps, and will actually be accelerating my writing schedule prior to starting the new job, since I'll have a little wedge of free time (a.k.a. unemployment!) between Mercy (my current full-time employer) and Acquia.

I'm excited to start working for Acquia, and am also excited to be able to continue doing what I love—working on Drupal, working on Solr, working on Ansible/infrastructure, and working in their respective communities.

Multi-value spatial search with Solr 4.x and Drupal 7

For some time, Solr 3.x and Drupal 7 have been able to do geospatial search (using the location module, geofield, or other modules that stored latitude and longitude coordinates in Drupal that could be indexed by Apache Solr). Life was good—as long as you only had one location per node!

Sometimes, you may have a node (say a product, or a personality) affiliated with multiple locations. Perhaps you have a hammer that's available in three of your company's stores, or a speaker who is available to speak in two locations. When solr 3.x and Drupal 7 encountered this situation, you would either use a single location value in the index (so the second, third, etc. fields weren't indexed or searched), or if you put multiple values into solr's search index using the LatLonType, solr could throw out unexpected results (sometimes combining the closest latitude and closest longitude to a given point, meaning you get strange search results).

With Solr 4.x, especially Solr 4.3 and 4.5, there were some great improvements to spatial search, mostly enabled by switching from the old and trusty single-value LatLonType to the new (but slightly slower) SpatialRecursivePrefixTreeFieldType, or RPT for short.

DevOps for Humans - Ansible presentation at DrupalCon Austin

I'm still recovering from an intense week of Drupal here in Austin, TX. I kicked things off by walking around the downtown area, then taking the intensive Acquia Drupal Developer Certification exam. Once the conference started, I attended a few sessions, met a few awesome Drupalists, and learned a lot. On the last day of the 'Con (the last session, in fact), I presented DevOps for Humans: Ansible for Drupal Deployment Victory!.

I think the presentation went well, and I heard some great questions at the end which really contributed to the discussion of Ansible and Drupal deployments in general. It was a great way to finish up the official DrupalCon sessions, though it meant I was revising slides for the hundredth time during the rest of the week, instead of relaxing and enjoying DrupalCon!

Before I post a video and slides from the session, I wanted to highlight some resources for anyone who attended (or didn't attend) DrupalCon Austin:

Below is the video and slides from the DevOps for Humans presentation. Please let me know what you think!

Thoughts on the Acquia Drupal Developer Certification Exam

tl;dr: I passed, the exam is better than I was expecting, but I still have mixed feelings about Acquia's Drupal Developer Certification program.

Acquia Certified Developer 2014

When I first heard about Acquia's Drupal Certification Program, I had mixed feelings. For most programming jobs, especially the majority of web-related jobs, a certification doesn't hold a lot of weight. Certifications are often like final exams for a university course—they show that you know a particular set of material, but they don't indicate whether you can actually use that knowledge effectively.* Further, tech-based certifications are less meaningful over time, as technology progresses and the tested knowledge required to gain a specific certification becomes less relevant.

According to a few people who have already taken the Acquia certification exam, it tested real-world Drupal knowlege rather than rote memorization. I was intrigued, but couldn't put together 1.5 hours of uninterrupted time to take the test online at home or elsewhere. Lucikly, Acquia offered the exam during DrupalCon, where I could sneak out for a while to take it.

DrupalCon Austin 2014 - Notes on my First Timer's Guide

I wrote A First Timer's Guide to DrupalCon two years ago, and 99% of the advice in that post is still relevant. I've been preparing my presentation DevOps for Humans: Ansible for Drupal Deployment Victory! for a few months now, and since DrupalCon is less than a week away, I thought I'd take a few minutes to supplement the earlier post:

Pages

Subscribe to RSS - drupal