When I joined Flatfile in 2021, the company was about a year into building their V3 product.
Flatfile’s most popular product, Portal V2, is an embeddable csv data importer featuring a wizard like user interface that allowed users to upload data, map their data to a configured target schema, followed by a review table where they could check everything looked ok before hitting “submit.” A simple yes-code product that supports a single use case, this particular product was so popular that it created the “data onboarding” industry and allowed Flatfile to raise a total of $100MM over several rounds of funding.
That last round of funding hinged on promises to go after the enterprise customer via Flatfile V3. V3’s Workspaces product was positioned as an enterprise-grade product that served more complex usecases than V2, and allowed for collaborative data onboarding projects. Instead of being yes-code, V3 featured a GUI interface for a PM-like persona, that allowed you to create and update target schemas and write “data hooks” ie “short functions to re-format, correct, validate, and enrich data automatically during a data import. They can run on specific fields and records (rows), or they can run across an entire batch of records.”
This product was an absolute disaster. Turns out:
- The data experience for the sub-1000 records that we were used to seeing for Portal V2 was a very poor UX for larger datasets . It was a simple data table experience that had a great error highlighting UI but couldn’t support filtering, search, or any sort of user triggered bulk transformations just doesn’t work for anything but the smallest datasets. Users would load 100,000s of records into the Flatfile UI and then give up after they found no tooling to actually fix the high volume of errors they were seeing.
- Each enterprise seemed to want different things, often, a feature that made one enterprise happy would piss off another one.
- Low code UIs are extremely expensive to build, maintain and ship new features to. Customers kept asking for updates to functionality for the GUI interface to define schemas and data hooks, and this required a ton of engineering effort.
The first felt very solvable, the product needed a better data experience and that’s what I was there to do.
The second seemed like a failure of build strategy. We had failed to come up with a solution that fit the constraints of the problem space we wanted to address.
The third felt like a failure of Flatfile’s attempt at offering a no-code offering. No one was buying Flatfile for a great configuration experience, they were buying a great data experience backed by a reliable data pipeline. Which means that all the engineering effort going into building new configuration features was not a revenue driver, but it was a blocker – we’d moved away from code based configuration. Flatfile is a young company with a small engineering team, every low-code feature we shipped represented data experience features we didn’t ship.
I started with the most solvable problem first – the data experience. We shipped a one-click “record status” filter that allowed users to see either “all records,” “valid records” or “invalid records,” which was very well received. Followed that up with a “Filter by error” feature – a dropdown that allowed users to filter by errors on a specific column, for eg. “Filter by records that have invalid entries on email address.”
I had a Figma full of designs for further improvements.
Custom bulk row actions – a first attempt at offering customizability
Every enterprise seemed to want to do some niche but bespoke thing to their data loaded in Flatfile. Sometimes it was to address some niche error pattern that we only saw in that customer’s data loads. Sometimes it was to facilitate some specific workflow that was unique to their organization.
What if we added a checkbox select mode to the datatable and had a spot above the table for button-triggers and allowed our customers to define what they wanted their users to see for their implementation of Flatfile?
I shopped this idea around to our enterprise customers with the help of our success team – they loved it. But it wasn’t enough.
Too disruptive, not enough value
Flatfile’s V3 Workspaces were being sold as a collaborative product where our customers could invite their customers to upload and onboard their data to ready it before it landed in a target system. Even though our customers seemed willing to take their data from their customers, dump it in their Flatfile Workspace to clean it up, they weren’t actually willing to invite their customers to do so.
The product didn’t fit their existing ways of doing things, retraining teams that are used to working a certain way is expensive. After all, from the customers’ perspective, whatever they were doing worked, even if not optimally efficient.
If we couldn’t mimic their existing processes while clearly and definitively accelerating them, they weren’t interested.
Platform, not a product
What if everything was customizable, not just what you could do to a set of records? What if Flatfile was a series of opinionated data experiences optimized for mapping and error resolution, that a developer could configure any sort of workflow around?
After months of ideating on how we could improve V3, the CEO and I concluded that we…couldn’t. V3’s fatal flaw was that it was a product, a complete and opinionated solution, built to meet a specific set of needs for a specific set of users. And the problem space of data exchange had too long a tail of usecases and workflows for any single product to serve a large enough segment to scale beyond a certain level.
A platform is a series of capabilities that you can use to build products. Not all capabilities need necessarily be connected, even. Building a platform would allow us to meet the needs of that very long tail of customer needs.
V3 needed to go, indeed, Flatfile is on schedule to deprecate all instances of V3 by Jan 2025. And instead we needed to build a yes-code platform that allowed our customers to use our building blocks to create a data exchange experience that was exactly what they needed.
Yes code!
The amount of customizability you can offer with a no or low code GUI is limited by how much engineering power you can throw at each problem. Airtable, another company selling a data experience with a layer of customization “Airtable Apps”, has 100s of engineers, not 10s. And while they offer a variety of interfaces to customize apps built on their product, it is still marketed as a product that requires at least “hobbyist” developer skills.
Any UI powerful enough to offer the kind of workflow customization and optimization required will be complex to use. They are often only powerful tools for expert users, a difficult cohort to create for a new, limited-use tool.
Why build a shitty UI when instead we could build a beautiful framework supported by excellent developer docs? (H/T to Ashley Mulligan, who was the Head of Developer Experience at Flatfile and responsible for most of the work done here, as of mid-2024.)
Custom actions, not just for rows
Flatfile, rebuilt from scratch as a Platform, features powerful data experience tooling augmented by an almost infinitely flexible orchestration framework.
Bulk row actions, which I mentioned before, are custom code you can run on one or more records in a sheet. Imagine you have a dataset of Smart Toilets – 100,000s of connected toilets that can be controlled via API. 40 of those toilets are in Austin, TX, for use by SXSW guests. You want to flush the SXSW toilets before the conference starts. You could load your entire database of toilets into Flatfile, and configure a custom action on your Smart Toilets sheet, “Flush Toilet” that flushes any toilet (record) on which this action is deployed. This is an example of an Action you can configure on records in a sheet.
What if you have a Workbook, with multiple sheets? In addition to the “Smart Toilets” sheet, you also have a sheet of “Toilet Cleaners” and another for “Events That Need Smart Toilets.” Using Flatfile’s orchestration framework, you could also configure custom actions on Workbooks. For example, when you click “Deploy Toilet Workforce” action on a Workbook, that could automatically schedule every Toilet Cleaner to go manually clean one or more assigned Smart Toilets before the date of the associated schedule Event – some custom workflow that pulls data across multiple sheets in a workbook and acts on all of it together.
We abstracted out all parts of the UI – Workbooks and Sheets among them – as “resources,” some of which have an hierarchical relationship with each other. This allows a developer building with Flatfile to build complex and specific apps by leveraging the Actions framework to tap into the Platform’s underlying event-listener system to create complex and specific workflows.
To reiterate, an action is a user experience affordance that one can configure on to any resource. It is augmented by Jobs, an affordance that allows a developer to customize the feedback loop around any deployed action. For example, once a user deploys the Flush Toilets action on those 40 toilets, and the 40 toilets are successfully flushed, the developer could configure a Job Outcome toast that says “40 toilets were successfully flushed! Good Job!”
Jobs are also how we allow developers to tap into state management affordances. For example, if a user were to transform 1 million records in their sheet, that might take a while – during which it would be unwise for the user to further interact with the data. That particular action could be configured to trigger a Job that would block the sheet from edits until completion.
One could also configure fairly complex actions that accept job inputs via an input UI, as well as chain together a series of jobs that occur simultaneously or sequentially.
Developers can even skip the UI affordances entirely and just tap into Flatfile’s Mapping API to automatically generate rules for moving data between any two schemas to augment their ETL processes.
We made it easy to get started by offering customers a set of pre-built example apps, based on common usecases, built on the Flatfile platform. It’s often easier to modify something that exists than to have to start from scratch.
We also built out some of our most commonly requested features into open-source plugins. We’ve had a few contributions from customers to the plugin library already, and the hope is that expands to many more.
Did it work?
V3 appeared quite successful at first, though this was more a success of an absolutely 🔥sales effort combined with a gaping need for more powerful data exchange tooling. However, when large enterprise customers who’d signed 6 figure deals with Flatfile weren’t seeing the value and we experienced a ton of churn throughout 2022 and 2023.
In early 2024, about 8 months after we launched the Platform, Flatfile signed its first 7 figure deal with a large enterprise, with several other deals of this size in the pipeline. Will it stick? I expect if Flatfile continues to execute along this path and at high quality, it is very likely.
Lisa Mahapatra 