Parsing data contained in a XML file into a pandas DataFrame is a common task in data science. The motivation for this article In May 2019 Roberto Preste wrote a nice article From XML to Pandas dataframes about how to accomplish this using xml.etree.ElementTree in combination with pandas. The article got quite some attention. Roberto…

About why to use mycli, pgcli and equivalents. — My personal motivation At the moment I’m working with a big legacy relational database consisting of over 200 hundred tables, a lot of relationships (quite some are nested)… and containing a lot of data. Interactive, exploratory tools in the Python DataScience area like JupyterLab, JupyterNotebook, etc. are incredible productivity boosters. Especially when one…

… or how to conditionally transform cell and row/column values? Pandas provides a lot of possibilities how to transform values in DataFrames. In many cases cell values within a whole row or column need to be transformed the same way. This is especially true if it’s about common operations w.r.t…

… in a nutshell — Relational databases are everywhere in our data driven world: In web development, data engineering and in general in every system which has to store tabular data. I’ve summarized a few key takeaways from Stephane Maarek’s Ultimate AWS Certified Developer Associate course he stated in the sections “Amazon RDS Overview” and…

… in Data Engineering in a nutshell — SQL is the Structured Query Language and “used in programming and designed for managing data held in a relational database management system (RDBMS)”. Using SQL we can read data from a SQL database, mutate (change) the data contained in a SQL database and write data into a SQL database. This…

A guide on parsing XML in Python using BeautifulSoup. — As a software developer, you’ll be surprised by how often you’ll be working with XML files. There are 2 basic use cases when it comes to working with XML: parsing XML to extract data from it modifying XML to mutate XML in-place In most cases, you’ll need parsing only so…

GitHub Actions for adding CI/CD to Vue 3 projects. — Test GitHub Actions workflow locally To enable a test-driven development for GitHub Action workflows, you need to be able to test GitHub Actions locally. act enables you to do that. Prevent from committing and pushing into the main branch In GitHub, it’s possible to set up branch protection rules which provide an option “Restrict who can push to matching branches” which allows preventing people…

A detailed look at cookiecutter-pypackage, pyscaffold, python-package-template, and wemake-pythonpackage. — Evaluation criteria Let’s think about what we’ve to consider during the evaluation of Python package generators. Project templating engine used Python projects contain metadata that is different for every project. This means a project needs to be generated dynamically. To achieve this most package template projects use cookiecutter.

The power of type annotation-based parsing and validation. — Pydantic is a library for type-safe parsing of data into Python objects with optional data validation… and more. Adoption A lot of well known Python projects (FastAPI, Project Jupyter), over 40000 open source projects and companies (Microsoft, Amazon Web Services, NSA, Uber, etc.) use pydantic. …