Software Engineer – Data Infrastructure

San Francisco, CA · Full Time - Posted by Jacob Perkins on November 7, 2018

“The goal is to turn data into information, and information into insight.” – Carly Fiorina

At Insight Engines, we want to end data hoarding. Too much data is overwhelming, but not enough data means missing valuable insights. Our goal is to get the right data into the right structure to turn it into information, then process that information into something accessible to humans, so they can gain insights that weren’t possible before.

But enough about us, let’s talk about you. Do you enjoy optimizing data pipelines and database indexes? How about building scalable data systems with robust monitoring? Have you seen the problems of data overload, and want to help fix them? As an integral member of our technology team, you will engineer, operate, and optimize machine learning models, ETL pipelines, text search engines, OLAP datastores, and everything in between. Your work will enable our groundbreaking natural language platform and help us develop, scale, and deploy our applications in a variety of contexts. You’ll wear many hats, touch many parts of our system, and have a significant impact on our products.

The kinds of problems you’ll be working on include:

  • Indexing and summarizing large data-sets to enable high performance analytics
  • Optimizing database queries for efficient real-time processing
  • Scaling and maintaining cloud databases and data processing pipelines
  • Developing data driven APIs for machine learning applications
  • Crafting data normalization models and rules
  • Leveraging existing open source technologies like Kafka, Hadoop, Druid, Spark, PostgreSQL and other tools

When applying, please tell us about your real world data engineering experience. Women, people of color, minorities, and LGBTQ candidates are encouraged to apply.


  • BS, MS, PhD in Computer Science, Engineering, or related discipline, or 3+ years equivalent technology experience
  • 2+ years of software development (Go, Python, Java, or equivalent)
  • Familiar with complex database management, replication, and backup
  • Operational experience with OLAP datastores, text search engines, key-value stores, or distributed databases.
  • Expertise with writing efficient, complex database queries
  • Secure cloud development experience on AWS, GCP, or equivalent
  • Use engineering best practices – deliver high code quality, automated testing, and build reusable components
  • Authorized to work in the United States

Company benefits

  • Open vacation policy
  • Health care insurance
  • Dental & vision insurance
  • Life insurance
  • Short-term & long-term disability insurance
  • Health care FSA
  • Transit & parking FSA
  • Free lunch at SF office
  • Flexible work hours
  • Holiday time off