[ad_1]
Gigabytes of data for a bag of groceries. This is what you get when you make robot deliveries.This is a lot of data-especially if you repeat it More than a million times Just like us.
But the rabbit hole is deeper. The data is also very diverse: robot sensors and image data, user interaction with our application, order transaction data, and so on. Use cases are equally diverse, from training deep neural networks to creating beautiful visualizations for our merchant partners, and everything in between.
So far, we have been able to handle all these complexities through our centralized data team. So far, continuous exponential growth has prompted us to seek new ways of working to keep up.
We found that the data grid paradigm is the best way forward. I will describe Starship’s view of the data grid below, but first, let’s briefly introduce the method and why we decided to adopt it.
What is a data grid?
The data grid framework is First describe Author: Zamak Dehani’s paradigm is based on the following points Core idea: Data products, data fields, data platforms, data governance.
The main purpose of the data grid framework is to help large organizations eliminate data engineering bottlenecks and deal with complexity. Therefore, it addresses many details related to the corporate environment, from data quality, architecture and security to governance and organizational structure.For now, only Several companies It has publicly declared compliance with the data grid paradigm-all large enterprises worth billions of dollars. Nevertheless, we think it can also be successfully applied to smaller companies.
Data grid in Starship
Does the data work near the person producing or consuming the information
In order to operate the hyper-local robot delivery market on a global scale, we need to transform all kinds of data into valuable products. Data comes from robots (such as telemetry, routing decisions, ETA), merchants and customers (and their applications, orders, products, etc.), and all operational aspects of the business (from brief remote operator tasks to global spare logistics parts and robots).
The diversity of use cases is the key reason that attracts us to adopt the data grid approach-we want to carry out data work very close to the people who produce or consume the information. By following the principles of the data grid, we hope to meet the diverse data needs of our team while maintaining reasonable central oversight.
Since Starship has not yet reached enterprise scale, all aspects of our data grid implementation are impractical. Instead, we have identified a simplified method that makes sense to us now and puts us on the right path for the future.
Data product
Define what your data product is-each product has an owner, interface, and user
Applying product thinking to our data is the foundation of the entire approach. We treat anything that exposes data to other users or processes as a data product. It can expose its data in any form: as BI dashboards, Kafka topics, data warehouse views, responses from predictive microservices, etc.
A simple example of a data product in Starship might be a BI dashboard for potential site customers to track their site’s business volume. A more complex example is a self-service pipeline for robotic software engineers to send any type of driving information from the robot to our data lake.
In any case, we will not treat our data warehouse (actually a Databricks lake library) as a single product, but as a platform that supports multiple interconnected products. Such fine-grained products are usually owned by data scientists/engineers who build and maintain them, rather than dedicated product managers.
Product owners should know who their users are and what needs they solve with the product-and on this basis define and realize the quality expectations for the product. Perhaps because of this, we have begun to pay more attention to the interface, that is, components that are critical to usability but difficult to modify.
Most importantly, understanding the users and the value each product creates for them makes it easier to prioritize ideas. This is essential in a startup environment where you need to act fast and don’t have time to make everything perfect.
Data field
Group your data products into domains that reflect the organizational structure of your company
Before understanding the data grid model, we have successfully used Lightly embedded data scientist Stayed in Starship for a while. In fact, some key teams have a data team member working with them part-time-no matter what this means in any particular team.
We continue to define data fields based on our organizational structure, this time carefully covering every part of the company. After mapping data products to domains, we assigned a data team member to manage each domain. This person is responsible for managing the entire set of data products in the domain-some of them are owned by the same person, some are owned by other engineers in the domain team, and some are owned by other data team members (for example, for resource reasons).
We like many aspects of our domain settings. First of all, now every area of the company has a dedicated person responsible for its data architecture. Considering the subtleties inherent in each field, it’s just because we have divided the work.
Creating structures in our data products and interfaces also helps us better understand our data world. For example, with more domains than data team members (currently 19 vs. 7), we are now doing better to ensure that each of us is working on a set of interrelated topics. We now understand that in order to alleviate the pain of growth, we should minimize the number of interfaces used across domain boundaries.
Finally, a more subtle benefit of using data fields is that we now feel that we have a way to deal with new situations. Whenever a new plan appears, everyone knows where it belongs and who should run it.
There are also some open questions. Some fields naturally tend to primarily open source data, while others tend to use and transform it, and some fields do both. When they become too large, should we separate them? Or should we set up subdomains in a larger domain? We need to make these decisions in the future.
Data platform
Provide support for people who build data products through standardization rather than centralization
The goal of the data platform in Starship is simple: to allow a single data person (usually a data scientist) to handle the domain end-to-end, that is, to keep the central data platform team away from daily affairs-today’s work. This requires good tools and standard data product building blocks for domain engineers and data scientists.
Does this mean that you need a complete data platform team to use the data grid approach? Not really. Our data platform team consists of a data platform engineer who embeds half of the time in a domain at the same time. The main reason why we can be so lean in data platform engineering is to choose Spark+Databricks as the core of our data platform. Due to the diversity of our data domains, our previous more traditional data warehouse architecture has brought us a lot of data engineering overhead.
We found it useful to make a clear distinction in the data stack between components that are part of the platform and all other components. Some examples we provide to domain teams as part of our data platform:
- Databricks+Spark as a working environment and multi-functional computing platform;
- Single-row functions for data ingestion, for example from Mongo collections or Kafka topics;
- An Airflow instance used to schedule the data pipeline;
- Used to build and deploy predictive models as a template for microservices;
- Cost tracking of data products;
- BI and visualization tools.
As a general approach, our goal is to standardize as much as possible in our current context-even for parts that we know will not remain standardized forever. As long as it now helps increase productivity and does not focus on any part of the process, we are happy. Of course, some elements are completely missing from the platform at present. For example, tools for data quality assurance, data discovery, and data lineage are what we leave for the future.
Data governance
Strong personal ownership supported by a feedback loop
In some aspects of governance, having fewer people and teams is actually an asset, such as easier decision-making. On the other hand, our key governance issues are also a direct result of our scale. If there is only one data person per domain, they cannot be expected to be experts in every potential technical aspect. However, they are the only ones who have detailed knowledge of their field. How can we maximize their chances of making the right choice in their field?
Our answer is: through the culture of ownership, discussion and feedback within the team.We learn a lot from management concepts On Netflix And cultivated the following:
- Personal responsibility for the result (one’s product and field);
- Seek different opinions before making a decision, especially those that affect other areas;
- Soliciting feedback and code reviews are both a quality mechanism and an opportunity for personal growth.
We also reached some specific agreements on how we deal with quality, write down our best practices (including naming conventions), etc. But we believe that a good feedback loop is a key factor in turning the guidelines into reality.
These principles also apply outside the “building” work of our data team-this is the focus of this blog post. obviously, there are more How our data scientists create value in the company instead of providing data products.
One final thought on governance-we will continue to iterate the way we work. There will never be a single “best” way of doing things, and we know we need to adapt over time.
Final words
This is it! These are the 4 core data grid concepts applied in Starship. As you can see, we have found a data grid approach that is suitable for us as an agile growth company. If this sounds attractive in your context, I hope reading our experience will help you.
If you want to participate in our work, please refer to Our career page Get a list of vacant positions.Or check out Our YouTube channel Learn more about our world’s leading robotic delivery service.
If you have any questions or ideas, please contact me and let us learn from each other!
[ad_2]
Source link
