How Girls Lead Knowledge Engineering at Slack

The Knowledge Engineering staff is accountable for Slack’s knowledge lake, analytics dashboards, and different knowledge companies. The staff’s mission is to empower customers to leverage knowledge to make choices rapidly, precisely, and simply. Slack’s knowledge lake grew in measurement from sub-petabyte to over 100 petabytes in recent times and it now spans thousands and thousands of tables. Because the complexity of managing this knowledge grew, so did a various staff of Slack engineers devoted to supporting the ecosystem. 

We’ve robust feminine illustration amongst engineers in Knowledge at Slack. Our Knowledge Engineering tradition celebrates variety in views and experiences. As knowledge complexity intensifies, having a mosaic of inventive drawback solvers from completely different backgrounds is the important thing to navigating intricate challenges with agility and perception.

Let’s dive into the private tales of ladies who’re redefining Slack’s knowledge panorama.

• Hearken to Jessica, a current rent, as she navigates the complexities of Pinot and Tableau expertise.
• Observe Nilanjana, Ramya, Shrushti, and Nathalie, seasoned engineers, as they lead large-scale engineering tasks utilizing applied sciences like Spark, Merlin, DataHub, and Secor.
• Lastly, hear from Suzanna, Lakshmi, and Beate, feminine engineering leaders, as they make clear the panorama of progress and alternatives for girls in knowledge engineering. 

Hello, I’m Jessica Stewart, and I’m a Senior Software program Engineer on the Knowledge Orchestration staff. I joined in Could 2023.

My staff oversees the inner implementations of information workflows utilizing Apache Airflow and Apache Pinot. Working inside our main cluster, which shops terabytes of information, we keep a system that boasts sub-second question latency and a virtually 99.95% question success charge service-level settlement (SLA). This datastore serves us internally by supporting instruments for workers and externally because the spine for Slack’s user-facing analytics dashboards.

A present technical problem we face revolves round migrating from a digital machine setup to a cloud-native Kubernetes infrastructure. Digital machines imply greater infrastructure prices and upkeep overhead, so we have been keen to extend effectivity with a containerized set-up. We’ve an inside Slack Kubernetes platform that doesn’t absolutely help Helm charts or Kubernetes companies, so we needed to make some customizations to the open-source answer. In an effort to keep efficiency necessities, we configured a buyer networking set-up that integrates Pinot with our inside Kubernetes platform. Moreover, working with Pinot permits us to delve into each software program and infrastructure layers. We’ve developed customized Python tooling that wraps round Pinot to standardize operations, and we’ve streamlined knowledge ingestion by means of Airflow pipelines. On the infrastructure aspect, we’ve automated deployments and upkeep duties utilizing Ansible, Kubernetes, and Jenkins.

When interviewing with Slack, my potential co-workers have been supportive and type, and this continues to be the case as we work on tasks, deal with incidents, and plan future work. I’ve the chance to personal initiatives, work cross-functionally and broaden my talent set — all whereas working with co-workers I belief and revel in.

Hello, I’m Nilanjana Mukherjee, and I’m a Employees Software program Engineer on the Metrics Foundations staff. I joined in October 2021.

My staff is accountable for producing actionable datasets and metrics for data-driven decision-making. We additionally keep excessive knowledge accuracy and desired touchdown time SLAs. 

Becoming a member of the Knowledge Engineering staff two years in the past has been an unbelievable expertise with alternatives and challenges. Amidst my involvement in lots of knowledge modeling tasks, I led a pivotal strategic initiative: a migration of workloads from Hive to Spark 3. Being one of many earliest adopters of Spark at Slack made it tough however rewarding. I grew to become the Spark subject material skilled to over 40 groups throughout Slack.

The journey was difficult however considerably contributed to my private {and professional} improvement. I owe a lot of my success to the sources and unwavering help from my managers and staff members. Just lately, I used to be thrilled to be promoted from Senior Engineer to Employees Engineer — a recognition of my progress, and a testomony to the alternatives for development at Slack in knowledge engineering.

Hello, I’m Ramya Sundaresan, and I’m a Senior Software program Engineer on the Metrics Foundations staff. I joined in Could 2022.

I’ve had the chance to collaborate with two knowledge groups throughout my tenure at Slack: the Knowledge Ingestion staff and the Metrics Foundations staff. On the Knowledge Ingestion staff, we have been entrusted with the important activity of gathering, processing, and ingesting knowledge into our knowledge warehouse from quite a lot of sources. This included extracting knowledge from Slack’s Vitess utility database, managing custom-made log pipelines, and integrating knowledge retrieval pipelines from Google Sheets. Throughout the Metrics Foundations staff, I discovered frequent floor with Nilanjana as we centered on related areas of curiosity and experience.

My contributions to Slack’s knowledge engineering ecosystem embrace implementing Apache Iceberg inside Kafka Join clusters, orchestrating the migration of Airflow to Kubernetes from AWS EC2 (Amazon Net Providers Elastic Compute Cloud) situations, and migrating jobs to Spark3 on AWS EMR6 (Elastic MapReduce) clusters. We closely leverage AWS. Transitioning from Spark 2 on AWS EMR 5 to Spark 3 upgrades on AWS EMR 6 was a strategic endeavor. It was according to a company-wide goal to ascertain a unified tech stack and diminish reliance on legacy techniques like Hive and MapReduce. I led a number of groups by means of automated workflows and complete documentation leading to a assured and profitable migration. We achieved our migration aim in beneath a 12 months utilizing parallel construct pipelines and rigorous testing. This unlocked higher efficiency and fortified our techniques with improved compatibility, interoperability, and long-term help.

My journey as a software program engineer at Slack has given me alternatives to guide, innovate, and contribute meaningfully to my staff’s objectives. Our on-call rotation and incident administration prioritizes staff member well-being with group and automation. My work provides me loads of satisfaction, however I additionally have to acknowledge a pleasant fixed in my life — my toddler son, who usually contains the colourful Slack emblem every time he doodles, and jogs my memory of the fun past code.

Hello, I’m Shrushti Patel, and I’m a Senior Software program Engineer on the Knowledge Infrastructure staff. I joined in August 2020.

My staff owns all of the infrastructure and companies required to ship dependable and well timed knowledge. Being a part of this staff, I’ve labored on companies like AWS EMR, Airflow, Trino, Secor, and Ranger to handle infrastructure.

Once I joined, Slack had an information ingestion setup reliant on Secor for transferring Kafka knowledge to S3 hosted on EC2. This was inflicting points after we tried to include new knowledge matters. My staff decided that Secor fell brief by way of industry-standard recognition, ongoing improvement, and help for rising knowledge codecs. I assumed accountability for this Secor setup, and I spearheaded its migration to Bedrock (an internally-developed Kubernetes framework). This transition simplified the addition of latest matters, harnessed the advantages of Kubernetes, and resulted in value financial savings by means of useful resource optimization.

To additional improve our knowledge infrastructure, we’re migrating from Secor to Kafka Join: a widely-adopted {industry} commonplace for streaming techniques. Kafka Join provides out-of-the-box help for state administration, fault tolerance, and scaling. Not like Secor, Kafka Join operates as an abstraction and leverages connectors (executable JARs) inside the Kafka-Join ecosystem. This strategic shift is shifting us in direction of real-time streaming capabilities. The work aligns with our dedication to {industry} requirements, ongoing improvement, and flexibility to evolving knowledge codecs. 

My time with Slack’s Knowledge Infrastructure staff has been a transformative journey. I’ve delved into mentorship, participating tasks, and a tradition of steady studying. Slack’s dedication to mobility has enriched my progress by offering me the chance to work with all of the groups beneath knowledge and cross-organizational groups like Cloud and Observability. Once I returned from maternity depart, the staff supported me by offering context about ongoing tasks and onboarding me again into the circulate. This expertise showcased Slack’s compassion by creating an atmosphere the place skilled excellence is nurtured alongside empathy for private milestones.

Hello, I’m Nathalie Kaligirwa, a Senior Software program Engineer on the Metrics Platform & Governance staff. My staff builds scalable and standardized instruments to boost the information expertise. I joined in November 2021.

The previous two years have been eventful. I had a wholesome child, took six months of maternity depart, and labored on complicated tasks with a number of stakeholders and distributed groups. I aimed to steadiness private progress with work commitments by aligning duties with my power degree and busy schedule.

Metrics standardization is a significant ongoing initiative at Slack, and it encompasses tasks like Merlin, a framework for abstracting metrics and huge tables creation. Throughout parenthood, I centered on particular elements inside this initiative that may be manageable throughout a time of elevated private calls for. This allowed me to contribute to the bigger imaginative and prescient of the mission whereas balancing new mother or father obligations. For instance, I up to date a service to dynamically add new metrics created with Merlin and eradicated the necessity for handbook migration. I then switched gears to work on enhancing the search expertise. This set the groundwork for a brand new metadata service that gives lineage throughout a number of knowledge instruments and scalable search.

Whether or not I used to be consolidating metrics, studying to alter diapers, integrating knowledge companies, placing collectively a nursery, engaged on metadata companies, or navigating a compromised sleep schedule, I grew professionally and personally. Trying again, I’m happy with my work and grateful for my staff’s help.

As Slack’s Knowledge Engineering management, we’re dedicated to mentorship and offering alternatives to study and develop. The staff can be dedicated to mobility — we encourage inside transfers for skilled progress and diversification of expertise. Slack’s steady studying philosophy not solely enriches particular person careers but additionally creates a dynamic and collaborative office. 

We mannequin Slack’s worth of compassion in our help for staff members taking maternity depart. The staff is organized to make reintegration after a hiatus a seamless expertise. We genuinely care not nearly our teammate’s skilled efficiency but additionally their private milestones. 

As leaders at Slack, we actively nurture an atmosphere the place ladies in knowledge engineering thrive and attain their full potential. We do that by means of:

• Visibility: We amplify ladies’s voices by encouraging them to tackle management roles and communicate at conferences. We rejoice their achievements and showcase them as position fashions by means of participation in {industry} occasions.
• Mentorship: We provide each formal and casual mentorship packages the place we join skilled ladies leaders with mentees for steering and help. This fosters confidence, management expertise and a robust community of friends.
• Advocacy: We actively problem biases and advocate for honest practices in all elements of our work atmosphere.
• Empowerment: We help versatile work preparations, variety and inclusion initiatives, and entry to coaching sources. We create a welcoming and inclusive house for all.

Embracing variety: our dedication to inclusion

Our private experiences at Slack show the affect of those inclusion efforts. That is the Slack we envisioned: a spot the place concepts are heard, contributions are valued, and particular person journeys are fueled by a collaborative and supportive tradition. Our success is clear within the rising variety of ladies selecting to affix our groups and within the legacy of compassion and inclusion we’re proud to domesticate.

All for becoming a member of our Knowledge Engineering staff? Apply now