Nightlife Importer

(Data Engineering Kit V.0)

Github Repository:

Technical Specifications:

Required Packages/Usage:

  • Pandas: I mostly use pandas or numpy for projects as they are dynamic and easy to use solutions.
  • os: Os is used to make sure the file structure is correct and to verify locations within directory
  • Re: Re was used to create sub-strings and replace values within strings for keyword extraction and tag creation.
  • Requests: Requests was used to download the files.

Languages: Python, Batch

Setup:

  1. move the repository into its own file (It is recommended to set up a venv when using this tool.)
  2. Launch Setup.bat in the specified repository (If in CMD use cd to setup in desired repository)
  3. Place links to data sources within categories_list.txt as so [New-dataset-name] = [Link-to-Dataset] If you want to group products use #[Group-Name] [ [New-dataset-name1] = [Link-to-Dataset1] [New-dataset-name2] = [Link-to-Dataset2] ]
  4. Modify prod_name, prod_price, prod_net_weight, prod_stock_level, rrp with desired columns in dataset (Assuming all datasets are normalized.
  5. Run the script and make sure everything runs as expected.

Project Overview

The Nightlife importer was a tool made for Nightlife.com for the purpouse of connecting to the existing feed of WP Imports and allow for more data to be added as well as modifying products to change features as well as create parent/child products more easily. This ended up being a prototype for a full data engineering kit which aims to expand on this to become a dynamic solution to data processing.

Objectives

The project needed multiple steps to achieve what the client wanted including:

  • Data Blending
  • Data Normalisation
  • Data Modification
  • Keyword Extraction
  • Deployement Package Creation
  • Data Cleaning
  • Defining Product Relationships
  • Import data to a wordpress website
  •  
Skip to content